How to Create GCP Schedule Job with Terraform

About Terraform with GCP

Terraform provide infrastructure automation to provision resources to many cloud provider, GCP is one of them, this article shows how to use Terraform to deploy a simple Big Query Schedule Job to the GCP Cloud.

Preparation

Create terraform files

Create provider.tf file

Here start with the terraform provider, google credentials and the GCP project.

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "5.15.0"
    }
  }
}

provider "google" {
  credentials = file("{this is option but your can create GCP service account with BigQuery.job.create role}")
  project = "{your gcp project id}"
}

Create service account

For the above credentials, below are few steps to create a service account has minimal role to perform below GCP operation.

• go to the GCP Roles page, create a new custom role with bigquery.job.create permission.
• go to the IAM service account page, create new service account and then assign the previous custom role.
• once the service account created, go to the KEYS tab and click ADD KEY to create new JSON key and save locally so above credentials can lookup

Create backend.tf file

Here, we're using the GCP cloud bucket to store the terraform state file, this is similar to share the code within the team so we can deploy same infrastructure no matter who runs it.

terraform {
 backend "gcs" {
   bucket  = "{create a GCP cloud bucket}"
   prefix = "test"
 }
}

Create schedule-job.tf

Now, finally here come to the main code we're trying to write a simple query and deploy to GCP BQ Schedule job and runs every sunday.

resource "google_bigquery_data_transfer_config" "job" {
    display_name = "test-query"
    project = "{your gcp project}"
    location = "us"
    data_source_id = "scheduled_query"
    schedule = "first sunday of quarter 00:00"
    params = {
      query = "select 1 as ID;"
    }
}

Start Deployment

First, go to the folder has above terraform files, run this command to initialize terraform from gcp cloud bucket to download state files.

terraform init

This command will start a new workspace with test, if not exist then will save the state to the cloud bucket.

terraform workspace new test

Terraform command to run the plan, to check what resources will generate to the GCP.

terraform plan

If you confirmed the resource, run this command to deploy the changes to the GCP.

terraform apply

Conclusion

Use the terraform is great to deploy infrastructure resource to the GCP for productions.