
## Table of Contents

1. [About Terraform with GCP](#about-terraform-with-gcp)
2. [Preparation](#preparation)
3. [Create Terraform Files](#create-terraform-files)
4. [Start Deployment](#start-deployment)
5. [Conclusion](#conclusion)

## About Terraform with GCP

Terraform provides infrastructure automation to provision resources to many cloud providers. GCP is one of them. This article shows how to use Terraform to deploy a simple BigQuery Schedule Job to GCP.

## Preparation

Before getting started, ensure you have Terraform installed and a GCP account with the necessary permissions.

## Create Terraform Files

### Create `provider.tf` File

Start with the Terraform provider, Google credentials, and the GCP project.

```
terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "5.15.0"
    }
  }
}

provider "google" {
  credentials = file("{this is option but your can create GCP service account with BigQuery.job.create role}")
  project = "{your gcp project id}"
}
```

### Create Service Account

For the above credentials, below are a few steps to create a service account with the minimal role to perform the GCP operation.

- Go to the GCP Roles page and create a new custom role with `bigquery.job.create` permission.
- Go to the IAM Service Account page, create a new service account, and then assign the custom role you created.
- Once the service account is created, go to the KEYS tab and click `ADD KEY` to create a new JSON key. Save it locally so the above credentials can reference it.

### Create `backend.tf` File

Here, we're using a GCP Cloud Storage bucket to store the Terraform state file. This is similar to sharing the code within the team so that the same infrastructure is deployed no matter who runs it.

```
terraform {
 backend "gcs" {
   bucket  = "{create a GCP cloud bucket}"
   prefix = "test"
 }
}
```

### Create `schedule-job.tf`

Finally, here is the main code. We're writing a simple query and deploying it to a GCP BigQuery Schedule Job that runs every Sunday.

```
resource "google_bigquery_data_transfer_config" "job" {
    display_name = "test-query"
    project = "{your gcp project}"
    location = "us"
    data_source_id = "scheduled_query"
    schedule = "first sunday of quarter 00:00"
    params = {
      query = "select 1 as ID;"
    }
}
```

## Start Deployment

First, go to the folder containing the above Terraform files and run this command to initialize Terraform and download state files from the GCP Cloud Storage bucket:

```
terraform init
```

This command will create a new workspace called `test`. If it does not exist, the state will be saved to the cloud bucket:

```
terraform workspace new test
```

Run the plan command to check what resources will be created in GCP:

```
terraform plan
```

If you have confirmed the resources, run this command to deploy the changes to GCP:

```
terraform apply
```

## Conclusion

Using Terraform is a great way to deploy infrastructure resources to GCP for production environments. It provides reproducible, version-controlled infrastructure management.


This article demonstrates how to use Terraform to deploy a GCP Schedule Job.

How to Create GCP Schedule Job with Terraform


## Table of Contents

- [Introduction](#introduction)
- [Create Service Account](#create-service-account)
- [Load Service Account](#load-service-account)
- [Run Node.js Locally](#run-nodejs-locally)
- [Test The Node.js with BigQuery Access](#test-the-nodejs-with-bigquery-access)
- [How to Debug via Visual Studio Code](#how-to-debug-via-visual-studio-code)
- [Conclusion](#conclusion)

## Introduction

This blog shows how to test GCP Cloud Function locally with Node.js. We will
set up a service account, load it in our Node.js code, run the function
locally, and debug it using Visual Studio Code.

## Create Service Account

Create a GCP service account and add the corresponding role. Create a new key
and save the file (JSON or P12) locally.

## Load Service Account

Next, load the file in the Node.js file.

```Javascript
const options = {
    keyFileName: 'location_of_the_file.json',
    projectId: 'your_gcp_project_id'
};

const bigquery = new BigQuery(options);

const dataset = bigquery.dataset('my_dataset');

const table = dataset.table('my_table');
```

## Run Node.js Locally

Add the code below to the `package.json`'s script section. When you run this
with the command `npm run test`, the function will start locally. I have
tested this on macOS with Node version 18; anything below might not work.

```CMD
 "test": "npx functions-framework --target=main [--signature-type=http]",
```

## Test The Node.js with BigQuery Access

Running the command below should start an HTTP server locally. Now you can
verify, debug the code, and make sure everything works before running the
deploy.

```
npm run test
```

## How to Debug via Visual Studio Code

Visual Studio Code is a great tool for software development and has improved a
lot since the beginning. One of the features for debugging Node.js in Visual
Studio Code is that you can switch to the `JavaScript Debug Terminal` and run
`npm run test`. This will use the built-in debugger attached to the process.

So it is pretty simple to debug Node.js with Visual Studio Code.

## Conclusion

Testing GCP Cloud Functions locally before deploying saves time and makes
debugging much easier. By setting up a service account and using the functions
framework, you can run and test your functions on your local machine. Combined
with the Visual Studio Code debugger, you have a smooth local development
workflow.


This blog shows how to test GCP Cloud Function locally with Node.js.

How to Test GCP Function Locally with Node.js


#### Table of Contents

1. Create BQ Table for test
2. PubSub Topic
3. Create a Dataflow
4. Manually Publish PubSub Message
5. Send Pub/Sub Message from C#
6. Authenticating as a service account
7. Conclusion

GCP Pub/Sub is managed messaging service provide by GCP, this blog demo how to
create Pub/Sub and use dataflow to subscribe data into BigQuery.

## Create BQ Table for test

First, need to create a BQ table, so we can store our message from Pub/Sub
we send, below is the JSON and BQ Table assume we want to store the
information.

`{&ldquo;id&rdquo;: 100, &ldquo;name&rdquo;:&rdquo;Smith James",
url:&rdquo;https://testsmithjames.com&rdquo;,&rdquo;age&rdquo;:30,&rdquo;price&rdquo;:300,&rdquo;desc&rdquo;:&rdquo;I
like sell my arts.&rdquo;}

## PubSub Topic

Here, we try to create a topic via gcloud command line,

To create a PubSub Topic

gcloud pubsub topics create test_topic_myname

In case, need to delete the topic, below command is how to delete a PubSub
Topic

gcloud pubsub topics delete test_topic_myname

## Create a Dataflow

Now, the following command example will create a dataflow job by using the
template PubSub to BigQuery and will insert the new record to the Big Query
table defined.

This command also expects the cloud bucket already exists.

```
gcloud dataflow jobs run ps-to-bq-test-topic-myname --gcs-location gs://dataflow-templates-us-central1/latest/PubSub_to_BigQuery --region us-central1 --staging-location gs://test-pubsub-name-topic/temp --parameters inputTopic=projects/test-dataflow-sql/topics/test_topic_myname,outputTableSpec=test-dataflow-sql:dataflow_sql_tutorial.test_pubsub_to_bq
```

## Manually Publish PubSub Message

Let's test the PubSub Dataflow to BigQuery, go to the GCP Pub/Sub page, and
paste the below Json, once click the publish button should see the new
record in BigQuery table.

```
{
  "id": 100,
  "name": "smith james",
  "url": "https://testsmithjames.com",
  "age": 30,
  "price": 300,
  "desc": "I like sell my arts"
}
```

## Send Pub/Sub Message from C#

Following is a sample code by using C# to send a message to the projecId and
topicId.

```
TopicName topicName = TopicName.FromProjectTopic(ProjectId, TopicId);

PublisherClient publisher = await PublisherClient.CreateAsync(topicName);

SamplePubSubModel sampleModel = new()
{
                id = 304,
                name = "my name",
                url = "https://test-url",
                age = 40,
                price = 300,
                desc = "some desc"
};

await publisher.PublishAsync(JsonConvert.SerializeObject(sampleModel));
```

Above C# code require the Google.Cloud.PubSub.V1, but also need to
authenticate then the code will successfully run.

## Authenticating as a service account

Now, just run the C# code above, and you'll see the exception which requires
providing the google credentials. There're a few ways to achieve this, what
we're doing here is trying to create a service account that has permission to
pub/sub message.

Once create a service account, make sure has the Role
Pub/Sub Publisher`.

Create a token and save the credential JSON file to the local, if you're using
windows, run the below command. Now, after following the below command, might
still see the credential error but, it might be fixed after restarting the
windows.

`set GOOGLE_APPLICATION_CREDENTIALS=key_path`

## Conclusion

Should be very easy to set up all processes, but might take some time on how
the google application credentials especially if you're using windows.


GCP Pub/Sub is managed messaging service provide by GCP, this blog demo how to create Pub/Sub and use dataflow to subscribe datainto BigQuery.

BigQuery

How to Create GCP Schedule Job with Terraform

How to Test GCP Function Locally with Node.js

PubSub to BigQuery via Dataflow with GCP