I've been on the CloudFormation side of the IaC Wars since 2014, when I started working in AWS. I've dabbled in terraform but never made it my primary IaC choice. For the Fooli Meme Factory, I needed to mess things up and then quickly revert the state to what it was at deployment time. So for SECCDC 2023, I ported Meme Factory almost entirely over to Terraform.

This led me to two problems. The first was the perennial issue I've had with Terraform from day one: "How do I manage state?". The second issue was how do I leverage some form of CI/CD tooling to allow me to leverage one of Terraform's biggest strengths - the terraform plan capability. Since Fooli is an AWS product, I figured that I should be able to use AWS native tools for this. I've used CodePipeline in the past to preview change-sets with aws-org-formation, so I thought it would be easy to find a well-worn pattern from AWS on doing it.

Apparently, there is no canonical way to use Terraform in CodeBuild, with CodePipeline as the method to review plans before applying them!!! Seriously, WTF?

This made me sad. So I decided to do it my damn self. And now I'm documenting it here for everyone else.

This solution will provide the following:

1. CloudFormation Template (CFT) to deploy a CodePipeline, CodeBuild Projects, and an S3 Bucket for state and artifact handling.
2. Buildspec files to perform the terraform plan and terraform apply steps.
3. Makefiles because I'm old school like that.

Why a CloudFormation template for step 1? To get around the chicken-and-egg problem with state. The CFT will deploy and do the needful to get the account setup for the terraform pipeline without needing a terraform pipeline already in place

How it works.

When a push is made to a monitored GitHub repo, the CodePipeline will trigger. AWS's CodeStar Connections are used to manage the integration between GitHub and CodePipeline. (As an aside: CodeStar connections are so under-the-radar I can't even find a product page to link to. Just some API docs and the above blog post.) Anyway - CodeStar is a much better solution than previous methods that required overly-privileged GitHub Personal Access Tokens to be uploaded into shared AWS Accounts.

So CodeStar Connections will monitor GitHub and fire the pipeline on a push to the specified branch. After that, the pipeline will execute the following stages:

1. Source Stage - where it downloads the code package from GitHub and stores it in the S3 Bucket.
2. Terraform Plan Stage - where CodeBuild will execute the terraform plan and copy the tfplan into S3
3. Review Stage - sends a message to SNS, which (if configured) will email someone to review the output of the Terraform plan in CodeBuild.
4. Apply Stage - If approved, this stage will fire up CodeBuild to do the terraform apply on the preexisting tfplan file.

CodePipeline & CodeBuild

The CodePipeline is defined entirely in the CloudFormation Template. The CodeBuild projects are created by the CloudFormation Template, but the commands to be executed are defined in the build spec files.

Terraform Plan

The definition of the Plan stage in CodePipeline is:

- Name: terraform-plan
  Actions:
    - Name: terraform_plan
      RunOrder: 1
      Namespace: TfPlan
      InputArtifacts:
        - Name: GitHubCode
      OutputArtifacts:
        - Name: TerraformPlan
      ActionTypeId:
        Category: Build
        Provider: CodeBuild
        Owner: AWS
        Version: '1'
      Configuration:
        ProjectName: !Ref TerraformPlanProject
        EnvironmentVariables: !Sub |
          [
            {"name": "EXECUTION_ID",    "value": "#{codepipeline.PipelineExecutionId}"},
            {"name": "BRANCH",          "value": "#{GitHubSource.BranchName}"},
            {"name": "REPO",            "value": "#{GitHubSource.FullRepositoryName}"},
            {"name": "COMMIT_ID",       "value": "#{GitHubSource.CommitId}"},
            {"name": "env",             "value": "${pEnvironment}"}
          ]

And the Project is:

TerraformPlanProject:
  Type: AWS::CodeBuild::Project
  Properties:
    Name: !Sub ${AWS::StackName}-tf-plan
    Artifacts:
      Type: CODEPIPELINE
    Source:
      Type: CODEPIPELINE
      BuildSpec: buildspec-tf-plan.yaml
    Environment:
      ComputeType: BUILD_GENERAL1_SMALL
      Type: LINUX_CONTAINER
      Image: !Ref BuildImageName
    ServiceRole: !GetAtt ProjectServiceRole.Arn

The EnvironmentVariables in the pipeline are passed into the CodeBuild Project. CodePipeline substitutes environment variables that begin with a # at execution, and the ones beginning with $ are substituted by CloudFormation at deployment. The BuildSpec exports some environment variables, and they're stored in the pipeline under the TfPlan Namespace (Line 5). The BuildSpec part of the CodeBuild Project (Line 9) defines the commands that CodeBuild will execute. That file looks like:

version: 0.2

env:
  exported-variables:
    - BuildID
    - BuildTag

phases:
  install:
    commands:
      - "curl -s https://releases.hashicorp.com/terraform/1.3.6/terraform_1.3.6_linux_amd64.zip -o terraform.zip"
      - "unzip terraform.zip -d /usr/local/bin"
      - "chmod 755 /usr/local/bin/terraform"
  pre_build:
    commands:
      - "make tf-init"
  build:
    commands:
      - "make tf-plan"
      - "export BuildID=`echo $CODEBUILD_BUILD_ID | cut -d: -f1`"
      - "export BuildTag=`echo $CODEBUILD_BUILD_ID | cut -d: -f2`"

artifacts:
  name: TerraformPlan
  files:
    - terraform/$env-terraform.tfplan

Lines 4-6 indicate that we will export two environment variables we want to pass back to CodePipeline, BuildID and BuildTag. These are needed to build the URL for reviewing the plan. The artifacts section on line 23 defines the files created that CodeBuild/CodePipeline should store in S3 and pass between the pipeline stages.

Review Stage

The review stage consists of a manual step and looks like this:

- Name: Review-Plan
  Actions:
    - Name: review-plan
      RunOrder: 1
      ActionTypeId:
        Category: Approval
        Provider: Manual
        Owner: AWS
        Version: '1'
      Configuration:
        NotificationArn: !Ref PipelineNotificationsTopic
        CustomData: "Review the Terraform Plan"
        ExternalEntityLink: !Sub "https://${AWS::Region}.console.aws.amazon.com/codesuite/codebuild/${AWS::AccountId}/projects/#{TfPlan.BuildID}/build/#{TfPlan.BuildID}%3A#{TfPlan.BuildTag}/?region=${AWS::Region}"

Here we construct the ExternalEntityLink from the BuildID and BuildTag from the plan stage. Again variables that begin with a # are substituted by CodePipeline at execution and the ones beginning with $ are substituted by CloudFormation at deployment. We send a message to the PipelineNotificationsTopic which triggers an email to the user:

Apply Stage.

The apply stage is similar to the plan. In CodePipeline it looks like this:

- Name: ExecuteTerraform
  Actions:
    - Name: terraform-apply
      RunOrder: 1
      InputArtifacts:
        - Name: GitHubCode
        - Name: TerraformPlan
      ActionTypeId:
        Category: Build
        Provider: CodeBuild
        Owner: AWS
        Version: '1'
      Configuration:
        ProjectName: !Ref ExecuteTerraformProject
        PrimarySource: GitHubCode
        EnvironmentVariables: !Sub |
          [
            {"name": "EXECUTION_ID",    "value": "#{codepipeline.PipelineExecutionId}"},
            {"name": "BRANCH",          "value": "#{GitHubSource.BranchName}"},
            {"name": "REPO",            "value": "#{GitHubSource.FullRepositoryName}"},
            {"name": "COMMIT_ID",       "value": "#{GitHubSource.CommitId}"},
            {"name": "env",             "value": "${pEnvironment}"}
          ]

In this case, we're inputting the input artifacts from both GitHub and the plan (lines 5-7). We set the GitHubCode as the PrimarySource on line 15, and that becomes the working directory. The other files are written to a different directory, and we have to move them in the BuildSpec file (line 9 below).

The BuildSpec for the apply looks like this:

version: 0.2

phases:
  install:
    commands:
      - "curl -s https://releases.hashicorp.com/terraform/1.3.6/terraform_1.3.6_linux_amd64.zip -o terraform.zip"
      - "unzip terraform.zip -d /usr/local/bin"
      - "chmod 755 /usr/local/bin/terraform"
      - "mv $CODEBUILD_SRC_DIR_TerraformPlan/terraform/$env-terraform.tfplan terraform"
  pre_build:
    commands:
      - "make tf-init"
  build:
    commands:
      - "make tf-apply"

The buildspec file installs terraform, moves the tfplan file back to where it's expected, runs make tf-init (because this is a new container), and then terraform apply

Makefiles

I use Makefiles to simplify the process of deploying both in CodeBuild and when deploying the terraform locally.

The root Makefile for the repo looks like this:

# Copyright 2022 - Chris Farris (chrisf@primeharbor.com) - All Rights Reserved
#
ifndef env
$(error env is not set)
endif

include config.$(env)
export

#
# Terraform
#
tf-init:
    cd terraform && $(MAKE) tf-init

tf-plan:
    cd terraform && $(MAKE) tf-plan

tf-apply:
    cd terraform && $(MAKE) tf-apply

And the Makefile in the terraform directory is:

# Copyright 2022 - Chris Farris (chrisf@primeharbor.com) - All Rights Reserved
#
tf-init:
    terraform init -backend-config=../$(env).tfbackend -reconfigure

tf-plan:
    terraform plan -out=$(env)-terraform.tfplan -no-color

tf-apply:
    terraform apply $(env)-terraform.tfplan

The config.env file contains all the TF_VAR exports to feed variables to terraform similar to:

export TF_VAR_mail_relay_ami=ami-0e03dcd66f...

The $(env).tfbackend contains the line to define the bucket:

bucket="fooli-tf-state-test"

Done!

There you have it, a complete solution to deploy Terraform in CodePipeline with CodeBuild and a manual review of the changes to be made. You can tweak the makefiles and buildspec files as you see fit. Here is the entire CloudFormation Template.

I'm surprised that there isn't a CodeBuild container from Hashicorp or AWS with Terraform pre-installed. The effort of making one would be less than the expense of everyone curling the terraform binary and providers twice on every build.