How to Create GCP Schedule Job with Terraform

Table of Contents

1. About Terraform with GCP
2. Preparation
3. Create Terraform Files
4. Start Deployment
5. Conclusion

About Terraform with GCP

Terraform provides infrastructure automation to provision resources to many cloud providers. GCP is one of them. This article shows how to use Terraform to deploy a simple BigQuery Schedule Job to GCP.

Preparation

Before getting started, ensure you have Terraform installed and a GCP account with the necessary permissions.

Create Terraform Files

Create provider.tf File

Start with the Terraform provider, Google credentials, and the GCP project.

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      version = "5.15.0"
    }
  }
}

provider "google" {
  credentials = file("{this is option but your can create GCP service account with BigQuery.job.create role}")
  project = "{your gcp project id}"
}

Create Service Account

For the above credentials, below are a few steps to create a service account with the minimal role to perform the GCP operation.

• Go to the GCP Roles page and create a new custom role with bigquery.job.create permission.
• Go to the IAM Service Account page, create a new service account, and then assign the custom role you created.
• Once the service account is created, go to the KEYS tab and click ADD KEY to create a new JSON key. Save it locally so the above credentials can reference it.

Create backend.tf File

Here, we're using a GCP Cloud Storage bucket to store the Terraform state file. This is similar to sharing the code within the team so that the same infrastructure is deployed no matter who runs it.

terraform {
 backend "gcs" {
   bucket  = "{create a GCP cloud bucket}"
   prefix = "test"
 }
}

Create schedule-job.tf

Finally, here is the main code. We're writing a simple query and deploying it to a GCP BigQuery Schedule Job that runs every Sunday.

resource "google_bigquery_data_transfer_config" "job" {
    display_name = "test-query"
    project = "{your gcp project}"
    location = "us"
    data_source_id = "scheduled_query"
    schedule = "first sunday of quarter 00:00"
    params = {
      query = "select 1 as ID;"
    }
}

Start Deployment

First, go to the folder containing the above Terraform files and run this command to initialize Terraform and download state files from the GCP Cloud Storage bucket:

terraform init

This command will create a new workspace called test. If it does not exist, the state will be saved to the cloud bucket:

terraform workspace new test

Run the plan command to check what resources will be created in GCP:

terraform plan

If you have confirmed the resources, run this command to deploy the changes to GCP:

terraform apply

Conclusion

Using Terraform is a great way to deploy infrastructure resources to GCP for production environments. It provides reproducible, version-controlled infrastructure management.