Automating Google Cloud Resource Management with GitHub Actions and Workload Identity Federation

In this blog post, we’ll demonstrate how to leverage GitHub Actions (GHA) and Google Workload Identity Federation (WIF) to securely authenticate and create resources on Google Cloud Platform (GCP) using Terraform.

We’ll use two GitHub repositories and two GCloud projects:

Main GitHub Repository:
Contains the Terraform code defining the infrastructure to be provisioned in the Target Project.
Triggers the GitHub Actions workflow upon code changes.

Workflow Repository:
Hosts the GitHub Actions workflow that automates the deployment process.
Authenticates to the Host Project using Workload Identity Federation (WIF).
Executes Terraform commands to apply the infrastructure changes to the Target Project.

Host Google Cloud Project:
Serves as the landing zone for the GitHub Actions workflow.
Configured with WIF, utilises a service account that’s been granted the necessary permissions to access the Target Project.
Does not directly host any infrastructure resources.

Target Google Cloud Project:
Receives the infrastructure changes from the workflow.
Hosts the provisioned resources defined in the Terraform code.

Let’s dive into the setup!

Step 1: Setting Up Google Workload Identity Federation
1.1 - On your host project create a Google Cloud Service Account
Navigate to the Google Cloud Console.
Go to IAM & Admin > Service Accounts.
Create a new service account with the necessary roles for managing your GCP resources.
1.2 - Configure Workload Identity Federation
Go to IAM & Admin > Workload Identity Federation.
Create a Workload Identity Pool and a Provider linked to your GitHub repository.
Follow Google’s official WIF setup guide for detailed instructions. More in a separate Blog Post soon.

Step 2: Permissions
2.1 - Ensure that the target gcp project has a terraform state bucket (ex: project_id-tfstate)
2.2 - Assign relevant roles(roles/editor &
roles/storage.admin) to the Service Account of the Host Project to be able to create/edit resources on the Target Google Project

  gcloud projects add-iam-policy-binding gcp_project_name \
    --member="serviceAccount:service-acct-name@host_gcp_project.iam.gserviceaccount.com" \
    --role="roles/editor"

  gcloud projects add-iam-policy-binding gcp_project_name \
    --member="serviceAccount:service-acct-name@host_gcp_project.iam.gserviceaccount.com" \
    --role="roles/storage.admin"

Step 3: Configure the Main Repo
This repository (e.g., org_name/google_cloud) will contain your Terraform code to manage GCP resources.

Example main.tf:

provider "google" {
  project = var.project
  region  = var.region
}

resource "google_storage_bucket" "bucket" {
  name     = "${var.project}-bucket"
  location = var.region
}

Step 4: Add a Workflow to the Main Repo

Create a GitHub Actions workflow (e.g., .github/workflows/dispatch.yml) in the main repository. This workflow is triggered by events such as push, pull request, or tag creation. When triggered, it makes a GitHub API call to initiate a workflow in another repository (referred to as the Workflow Repo).

Step 5: Configure the Workflow Repo

In the Workflow Repository (Workflow Repo), set up a GitHub Actions workflow dedicated to running Terraform-specific tasks, such as generating a Terraform plan and applying it to the infrastructure. For example, you can name this file .github/workflows/terraform_plan_apply_gcp.yml. Let’s refer to it as the Terraform Workflow.

How It Works

1.Triggering Event:
This Terraform Workflow is designed to be triggered by repository_dispatch events. These events are custom webhook events sent by the main repository via a GitHub API call. When a repository_dispatch event occurs with a specific type, it starts this workflow.
2.Event Types:
You can define specific event types that the workflow should listen to. In this example, the workflow listens for two custom event types:
• terraform_plan: Used to trigger the workflow for generating a Terraform plan.
• terraform_apply: Used to trigger the workflow for applying the Terraform changes.
3.Workflow Configuration:
Add the following YAML configuration to define the trigger mechanism in the Terraform Workflow:

on:
  repository_dispatch:
    types: [terraform_plan, terraform_apply]

• The on: repository_dispatch block specifies that this workflow listens for repository_dispatch events.
• The types field restricts the workflow to respond only to the specified event types (terraform_plan and terraform_apply).

4.Integration with Main Repository:
In the main repository, another workflow (e.g., .github/workflows/dispatch.yml) sends these repository_dispatch events using a GitHub API call. The event payload includes the event type (e.g., terraform_plan) and any additional data required by the Terraform Workflow.
5.Example Workflow Behavior:
• If the main repository triggers a repository_dispatch event with the type terraform_plan, the Terraform Workflow starts and runs tasks related to generating a Terraform plan.
• Similarly, if the event type is terraform_apply, the workflow executes the Terraform apply process to update the infrastructure.

This mechanism decouples the triggering mechanism in the main repository from the Terraform-specific operations in the Workflow Repo, enabling modular and maintainable workflows.

Scenarios to demonstrate how the Main Repo Workflow Operates
Scenario 1: Push with a Tag
A developer pushes Terraform code to a feature branch and adds a tag (e.g., gcp_project_TFPLAN_01).
The dispatch.yml workflow triggers when a tag matching a specific pattern is pushed (e.g., '[a-z]+-[a-z]+TFPLAN[0-9]+').
The workflow determines the Terraform action (e.g., plan) and triggers the Workflow Repo's Terraform workflow via a GitHub API repository_dispatch event.

Scenario 2: Pull Request Creation
When a pull request is opened from a feature branch to develop, the workflow sends a repository_dispatch event with details such as:
event_type: terraform_plan
GCP project name & PR metadata (e.g., PR number, status=opened, merged=false).

Scenario 3: Pull Request Merge
When a pull request is merged, the workflow sends a repository_dispatch event with:
event_type: terraform_apply
GCP project name & PR metadata (e.g., PR number, status=closed, merged=true).

Full workflow

# This GitHub Actions workflow is designed to trigger a Terraform workflow based on specific events.

name: Trigger Terraform Workflow

on:
  push:
    tags:
      - '[a-z]+-[a-z]+_PLAN_[0-9]+'
    # branches:
    #   - 'feature/*'
  pull_request:
    types: [opened, synchronize, closed]
    branches:
      - develop

jobs:
  trigger:
    runs-on: ubuntu-latest
    if: >
      github.event_name == 'push' || 
      (github.event_name == 'pull_request' && 
        (github.event.action == 'opened' || 
          github.event.action == 'synchronize' || 
          (github.event.action == 'closed' && github.event.pull_request.merged == true)))

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      # Debug step to print the GITHUB_REF
      - name: Retrieve GitHub Data
        if: github.event_name == 'push' || github.event_name == 'pull_request' 
        run: |
          echo "GITHUB_REF=${GITHUB_REF}"
          TAG_NAME=${GITHUB_REF#refs/tags/}
          echo "TAG_NAME=$TAG_NAME" >> $GITHUB_ENV

      - name: Print PR Information
        if: github.event_name == 'pull_request'
        run: |
          # Get the latest TAG Name from the source branch
          git fetch --tags

          # Get latest tag
          TAG_NAME=$(git tag --sort=-creatordate | head -n 1)

          # Output the tag
          if [ -z "$TAG_NAME" ]; then
            echo "No tags found on the source branch: $SOURCE_BRANCH"
          else
            echo "The latest tag on the source branch ($SOURCE_BRANCH) is: $TAG_NAME"
            echo "TAG_NAME=$TAG_NAME" >> $GITHUB_ENV
          fi
          echo "PR Number: ${{ github.event.pull_request.number }}"
          echo "PR Action: ${{ github.event.action }}"
          echo "PR Merged: ${{ github.event.pull_request.merged }}"

      - name: Extract Information from Tag or PR
        id: extract_info
        run: |
          GCP_PROJECT=$(echo $TAG_NAME | cut -d'_' -f1)
          echo "GCP_PROJECT=$GCP_PROJECT" >> $GITHUB_ENV
          echo "The Tag Name is: $TAG_NAME"
          echo "The Target GCP Project is: $GCP_PROJECT"
          if [[ "${{ github.event_name }}" == "push" ]]; then
            ACTION="plan"
          elif [[ "${{ github.event_name }}" == "pull_request" ]]; then
            if [[ "${{ github.event.action }}" == "closed" && "${{ github.event.pull_request.merged }}" == "true" ]]; then
              ACTION="apply"
            else
              ACTION="plan"
            fi
          fi
          echo "ACTION=$ACTION" >> $GITHUB_ENV
          echo "The Terraform Action is: $ACTION"

      - name: Trigger Terraform Workflow for Push Commits
        if: github.event_name == 'push'
        run: |
          echo "This was triggered as a result of the Event: ${{ github.event_name }} commits"
          curl -X POST \
            -H "Accept: application/vnd.github.v3+json" \
            -H "Authorization: Bearer ${{ secrets.GH_DISPATCH_PAT }}" \
            https://api.github.com/repos/${{ github.repository }}/dispatches \
            -d "{\"event_type\":\"terraform_${{ env.ACTION }}\", \"client_payload\": {\"repository\": \"${{ github.repository }}\", \"project_name\": \"${{ env.GCP_PROJECT }}\", \"tag_name\": \"${{ env.TAG_NAME }}\"}}"

      - name: Trigger Terraform Workflow for PR
        if: github.event_name == 'pull_request'
        run: |
          echo "This was triggered as a result of PR Number: ${{ github.event.pull_request.number }} being ${{ github.event.action }}"
          curl -X POST \
            -H "Accept: application/vnd.github.v3+json" \
            -H "Authorization: Bearer ${{ secrets.GH_DISPATCH_PAT }}" \
            https://api.github.com/repos/${{ github.repository }}/dispatches \
            -d "{\"event_type\":\"terraform_${{ env.ACTION }}\", \"client_payload\": {\"repository\": \"${{ github.repository }}\", \"pr_number\": \"${{ github.event.pull_request.number }}\", \"pr_event\": \"${{ github.event.action }}\", \"pr_merged\": \"${{ github.event.pull_request.merged  }}\", \"project_name\": \"${{ env.GCP_PROJECT }}\", \"action\": \"${{ env.ACTION }}\"}}"

How the Terraform Workflow Operates
This workflow is hosted in the Workflow Repo and is triggered by repository dispatch events. Below, we break down each step with detailed explanations and corresponding code.

Step 1: Define Workflow Triggers
The workflow listens for certain event types:

repository_dispatch: Triggered by the Main Repo for Terraform plan and apply actions. More on repository dispatch can be found on the Official GitHub Documentation.

name: Terraform CI/CD

on:
  workflow_dispatch:
  repository_dispatch:
    types: [terraform_plan, terraform_apply]

Step 2: Set Up the Terraform Job
Define a Terraform job that runs on ubuntu-latest with relevant permissions for GitHub Actions to interact with the repository and pull requests.

jobs:
  terraform:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
      pull-requests: write
      repository-projects: write

Step 3: Set Environment Variables
Capture the payload data from the triggering event(the main workflow) and set them as environment variables for subsequent steps.

- name: Set ENV variables
  run: |
    echo "TARGET_GCP_PROJECT=${{ github.event.client_payload.project_name }}" >> $GITHUB_ENV
    echo "CLOUDSDK_CORE_PROJECT=${{ github.event.client_payload.project_name }}" >> $GITHUB_ENV
    echo "GH_REPOSITORY=${{ github.event.client_payload.repository }}" >> $GITHUB_ENV
    GH_REPO_NAME=$(echo "${{ github.event.client_payload.repository }}" | cut -d'/' -f2)
    echo "GH_REPO_NAME=${GH_REPO_NAME}" >> $GITHUB_ENV       
    echo "GH_PR_NUMBER=${{ github.event.client_payload.pr_number }}" >> $GITHUB_ENV
    echo "GH_PR_EVENT=${{ github.event.client_payload.pr_event }}" >> $GITHUB_ENV
    echo "GH_PR_MERGED=${{ github.event.client_payload.pr_merged }}" >> $GITHUB_ENV

Step 4: Checkout the Code
Clone the Main Repo containing the Terraform code.

- name: Checkout code
  uses: actions/checkout@v2
  with:
    repository: ${{ env.GH_REPOSITORY }}
    token: ${{ secrets.GITHUB_TOKEN }}

Step 5: Set Up Terraform
Set up Terraform to run commands such as init, plan, and apply.

- name: Set up Terraform
  uses: hashicorp/setup-terraform@v3

Step 6: Authenticate to Google Cloud
Use Workload Identity Federation to securely authenticate with Google Cloud.

- name: Authenticate to Google Cloud
  id: authenticate
  uses: google-github-actions/auth@v2
  with:
    create_credentials_file: true
    workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
    service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}

Step 7: Initialize Terraform
Set the GCP project and initialize Terraform with the remote backend configuration.

- name: Terraform Init
  id: init
  run: terraform init -backend-config="bucket=$TARGET_GCP_PROJECT-tfstate"
  working-directory: ${{ env.TF_WORKING_DIR }}

Step 8: Validate Terraform Code
Ensure the Terraform configuration is correctly formatted and syntactically valid.

- name: Terraform Format
  id: fmt
  run: terraform fmt
  working-directory: ${{ env.TF_WORKING_DIR }}

- name: Terraform Validate
  id: validate
  run: terraform validate
  working-directory: ${{ env.TF_WORKING_DIR }}

Step 9: Generate Terraform Plan
Generate a plan and output the details for review.

- name: Terraform Plan
  id: plan
  run: terraform plan -var-file="$TARGET_GCP_PROJECT.tfvars" -out=tfplan
  working-directory: ${{ env.TF_WORKING_DIR }}

- run: terraform show -no-color tfplan
  id: show
  working-directory: ${{ env.TF_WORKING_DIR }}
## We will use the output of terraform show to write the plan as a comment to the pull request

Step 10: Comment on Pull Requests
If triggered by a pull request, post the plan as a comment for review.

- name: PR Comment
  uses: actions/github-script@v7
  if: github.event.action == 'terraform_plan' && ( env.GH_PR_EVENT == 'opened' || env.GH_PR_EVENT == 'synchronize' )
  env:
    PLAN: "terraform\n${{ steps.show.outputs.stdout }}"
  with:
    github-token: ${{ secrets.GH_PAT }}
    script: |
      const { data: comments } = await github.rest.issues.listComments({
        owner: context.repo.owner,
        repo: process.env.GH_REPO_NAME,
        issue_number: process.env.GH_PR_NUMBER,
      })
      const botComment = comments.find(comment => comment.body.includes('Terraform Format and Style'))
      const output = `#### Terraform Plan\n\`\`\`\n${process.env.PLAN}\n\`\`\``

      if (botComment) {
        github.rest.issues.updateComment({
          owner: context.repo.owner,
          repo: process.env.GH_REPO_NAME,
          comment_id: botComment.id,
          body: output
        })
      } else {
        github.rest.issues.createComment({
          owner: context.repo.owner,
          repo: process.env.GH_REPO_NAME,
          issue_number: process.env.GH_PR_NUMBER,
          body: output
        })
      }

Step 11: Apply the Terraform Plan
If the event is terraform_apply, apply the plan to create resources.

- name: Terraform Apply
  if: github.event.action == 'terraform_apply'
  id: apply
  run: terraform apply -auto-approve tfplan
  working-directory: ${{ env.TF_WORKING_DIR }}

By integrating GitHub Actions and Google Workload Identity Federation, you can establish a secure, automated CI/CD pipeline for managing GCP resources using Terraform. This approach ensures that Terraform plans are reviewed, validated, and applied only after thorough approval, enhancing both security and operational efficiency.

Blog

Automating Google Cloud Resource Management with GitHub Actions and Workload Identity Federation

sam-nash

Join Our Newsletter. No Spam, Only the good stuff.

Related