
## About Terraform with GCP

Terraform provide infrastructure automation to provision resources to many cloud provider, GCP is one of them, this article shows how to use Terraform to deploy a simple Big Query Schedule Job to the GCP Cloud.

## Preparation

## Create terraform files

### Create `provider.tf` file

Here start with the terraform provider, google credentials and the GCP project.

```
terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "5.15.0"
    }
  }
}

provider "google" {
  credentials = file("{this is option but your can create GCP service account with BigQuery.job.create role}")
  project = "{your gcp project id}"
}
```

### Create service account

For the above credentials, below are few steps to create a service account has minimal role to perform below GCP operation.

- go to the GCP Roles page, create a new custom role with `bigquery.job.create` permission.
- go to the IAM service account page, create new service account and then assign the previous custom role.
- once the service account created, go to the KEYS tab and click `ADD KEY` to create new JSON key and save locally so above credentials can lookup

### Create `backend.tf` file

Here, we're using the GCP cloud bucket to store the terraform state file, this is similar to share the code within the team so we can deploy same infrastructure no matter who runs it.

```
terraform {
 backend "gcs" {
   bucket  = "{create a GCP cloud bucket}"
   prefix = "test"
 }
}
```

### Create `schedule-job.tf`

Now, finally here come to the main code we're trying to write a simple query and deploy to GCP BQ Schedule job and runs every sunday.

```
resource "google_bigquery_data_transfer_config" "job" {
    display_name = "test-query"
    project = "{your gcp project}"
    location = "us"
    data_source_id = "scheduled_query"
    schedule = "first sunday of quarter 00:00"
    params = {
      query = "select 1 as ID;"
    }
}
```

### Start Deployment

First, go to the folder has above terraform files, run this command to initialize terraform from gcp cloud bucket to download state files.

```
terraform init
```

This command will start a new workspace with `test`, if not exist then will save the state to the cloud bucket.

```
terraform workspace new test
```

Terraform command to run the plan, to check what resources will generate to the GCP.

```
terraform plan
```

If you confirmed the resource, run this command to deploy the changes to the GCP.

```
terraform apply
```

## Conclusion

Use the terraform is great to deploy infrastructure resource to the GCP for productions.


This article demonstrate how to terraform to deploy GCP Schedule JOB

How to Create GCP Schedule Job with Terraform


<p>
  GCP Pub/Sub is managed messaging service provide by GCP, this blog demo how to
  create Pub/Sub and use dataflow to subscribe data into BigQuery.
</p>
<h2>Create BQ Table for test</h2>
<p>
  <span style="font-weight: 400"
    >First, need to create a BQ table, so we can store our message from Pub/Sub
    we send, below is the JSON and BQ Table assume we want to store the
    information.</span
  >
</p>
<p>
  <code
    ><span style="font-weight: 400"
      >{&ldquo;id&rdquo;: 100, &ldquo;name&rdquo;:&rdquo;Smith James",
      url:&rdquo;</span
    ><a href="about:blank"
      ><span style="font-weight: 400"
        >https://testsmithjames.com&rdquo;,&rdquo;age&rdquo;:30,&rdquo;price&rdquo;:300,&rdquo;desc&rdquo;:&rdquo;I
        like sell my arts.</span
      ></a
    ><span style="font-weight: 400">&rdquo;}</span></code
  >
</p>
<h2><span style="font-weight: 400">PubSub Topic</span></h2>
<p>
  <span style="font-weight: 400"
    >Here, we try to create a topic via gcloud command line,&nbsp;</span
  >
</p>
<p class="p1"><span class="s1">To create a PubSub Topic</span></p>
<p class="p1">
  <code
    ><span class="s1">gcloud pubsub topics create test_topic_myname</span></code
  >
</p>
<p class="p1">
  <span class="s1"
    >In case, need to delete the topic, below command is how to delete a PubSub
    Topic</span
  >
</p>
<p class="p1">
  <code
    ><span class="s1">gcloud pubsub topics delete test_topic_myname</span></code
  >
</p>
<h2>Create a Dataflow</h2>
<p>
  Now, the following command example will create a dataflow job by using the
  template PubSub to BigQuery and will insert the new record to the Big Query
  table defined.
</p>
<p>This command also expects the cloud bucket already exists.</p>
<pre
  class="language-bash"
><code>gcloud dataflow jobs run ps-to-bq-test-topic-myname --gcs-location gs://dataflow-templates-us-central1/latest/PubSub_to_BigQuery --region us-central1 --staging-location gs://test-pubsub-name-topic/temp --parameters inputTopic=projects/test-dataflow-sql/topics/test_topic_myname,outputTableSpec=test-dataflow-sql:dataflow_sql_tutorial.test_pubsub_to_bq</code></pre>
<h2 class="p1"><span class="s1">Manually Publish PubSub Message</span></h2>
<p>
  <span class="s1"
    >Let's test the PubSub Dataflow to BigQuery, go to the GCP Pub/Sub page, and
    paste the below Json, once click the publish button should see the new
    record in BigQuery table.</span
  ><span class="s1"></span>
</p>
<pre class="language-markup"><code>{
  "id": 100,
  "name": "smith james",
  "url": "https://testsmithjames.com",
  "age": 30,
  "price": 300,
  "desc": "I like sell my arts"
}</code></pre>
<h2>Send Pub/Sub Message from C#</h2>
<p>
  Following is a sample code by using C# to send a message to the projecId and
  topicId.
</p>
<pre
  class="language-csharp"
><code>TopicName topicName = TopicName.FromProjectTopic(ProjectId, TopicId);

PublisherClient publisher = await PublisherClient.CreateAsync(topicName);

SamplePubSubModel sampleModel = new()
{
                id = 304,
                name = "my name",
                url = "https://test-url",
                age = 40,
                price = 300,
                desc = "some desc"
};

await publisher.PublishAsync(JsonConvert.SerializeObject(sampleModel));</code></pre>
<p>
  Above C# code require the Google.Cloud.PubSub.V1, but also need to
  authenticate then the code will successfully run.
</p>
<h2>Authenticating as a service account</h2>
<p>
  Now, just run the C# code above, and you'll see the exception which requires
  providing the google credentials. There're a few ways to achieve this, what
  we're doing here is trying to create a service account that has permission to
  pub/sub message.
</p>
<p>
  Once create a service account, make sure has the Role
  <code>Pub/Sub Publisher</code>.
</p>
<p>
  Create a token and save the credential JSON file to the local, if you're using
  windows, run the below command. Now, after following the below command, might
  still see the credential error but, it might be fixed after restarting the
  windows.
</p>
<p><code>set GOOGLE_APPLICATION_CREDENTIALS=key_path</code></p>
<h2>Conclusion</h2>
<p>
  Should be very easy to set up all processes, but might take some time on how
  the google application credentials especially if you're using windows.
</p>
<p></p>


GCP Pub/Sub is managed messaging service provide by GCP, this blog demo how to create Pub/Sub and use dataflow to subscribe datainto BigQuery.

Category BigQuery

How to Create GCP Schedule Job with Terraform

PubSub to BigQuery via Dataflow with GCP