TheLazy.Dev
Portfolio GitHub

Blog

aws |cicd |codecatalyst |terraform

CI/CD Pipeline for Terraform Workflow Using Amazon CodeCatalyst

lanandra

Luthfi Anandra

Posted on November 22, 2023

CI/CD Pipeline for Terraform Workflow Using Amazon CodeCatalyst

Terraform workflow can be ran using several methods. One of them is running Terraform workflow inside CI/CD pipeline. Running Terraform workflow inside CI/CD pipeline can have several benefits, such as: automate creation or provision resources, simplify collaboration between engineers/developers, etc.

Table of Contents

  1. Scenario
  2. Prerequisites
  3. Configuration Steps
  4. Summary

Scenario

In this blog, I will explain how to run Terraform workflow inside CI/CD pipeline which in this blog is Amazon CodeCatalyst. GitHub is used as source code repository and has been connected with CodeCatalyst. Next in CodeCatalyst pipeline/workflow, We will provision resources via Terraform. In this blog, Terraform is using S3 backend. Here is the topology used in this blog:

topology

This blog is using reference from tutorial “Bootstrapping your Terraform automation with Amazon CodeCatalyst” written by @Cobus Bernard with some adjustment in configuration.

If You want to know more details about CodeCatalyst, please check CodeCatalyst documentation.

Prerequisites

  • Before starting configuration, if you have not got AWS Builder ID account, please sign up first
  • After sign up, go to Amazon CodeCatalyst web console to start project and configuration. Apply initial configuration that needed by CodeCatalyst such as:
  • Prepare S3 bucket and DynamoDB lock table (optional) that will be used by Terraform backend

Configuration Steps

Directory Structure

Below is directory structure used in this blog:

.
├── .codecatalyst/
│   └── workflows/
│       └── tf-sbx2-vpc-apse1.yml
└── sandbox2/
    ├── vpc/
    │   └── ap-southeast-1/
    │       ├── main.tf
    │       ├── resources.tf
    │       └── variables.tf
    └── tf-backend/
        ├── main.tf
        ├── resources.tf
        └── variables.tf
Enter fullscreen mode Exit fullscreen mode

Terraform Backend Configurations

Configuration start with provisioning resources that will be needed by Terraform backend such as S3 bucket and DynamoDB table. Beside that, We also need to prepare IAM role that will be needed to provision resources from CodeCatalyst workflow/pipeline.

  1. At first, We haven’t had any resource for storing Terraform state in S3, so that we need to run and store terraform state on our local, for example: laptop. Later, We will move that state from local to S3. We start by defining configuration in sandbox2/tf-backend/main.tf

    terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.11.0"
    }
  }

  required_version = ">= 1.3.0"
}

provider "aws" {
  region = var.aws_region
}

  2. Define variable region in file sandbox2/tf-backend/variables.tf

    variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}

  3. Define resources S3, DynamoDB and IAM that needed by CodeCatalyst in file sandbox2/tf-backend/resources.tf. In this blog, IAM role CodeCatalyst has administrator access just for demo purposes. Please bear in mind to always use least privilege method on your production to prevent any security breaches

    ######
# S3 #
######

resource "aws_s3_bucket" "my_sandbox2_tfbucket" {
  bucket = "my-sandbox2-tfbucket"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "my_sandbox2_tfbucket" {
  bucket = aws_s3_bucket.my_sandbox2_tfbucket.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "my_sandbox2_tfbucket" {
  bucket = aws_s3_bucket.my_sandbox2_tfbucket.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "my_sandbox2_tfbucket" {
  bucket                  = aws_s3_bucket.my_sandbox2_tfbucket.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

############
# DynamoDB #
############

resource "aws_dynamodb_table" "my_sandbox2_tflocks" {
  name         = "my-sandbox2-tflocks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

#######
# IAM #
#######

data "aws_iam_policy_document" "codecatalyst_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type = "Service"
      identifiers = [
        "codecatalyst.amazonaws.com",
        "codecatalyst-runner.amazonaws.com"
      ]
    }
  }
}

resource "aws_iam_role" "codecatalyst_admin" {
  name               = "codecatalyst-admin"
  assume_role_policy = data.aws_iam_policy_document.codecatalyst_assume_role_policy.json
}

resource "aws_iam_role_policy_attachment" "codecatalyst_admin" {
  role       = aws_iam_role.codecatalyst_admin.name
  policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}

  4. Refer AWS keys that needed for running terraform workflow commands, for example by exporting environment variables as explained in this document

  5. Run terraform cli commands that needed for provisioning resources

    terraform init
terraform plan
terraform apply

  6. Verify terraform apply is successful and resources successfully provisioned

  7. Back to file sandbox2/tf-backend/main.tf, add additional configurations related to S3 backend

    ###########################
# Terraform Configuration #
###########################

terraform {
  backend "s3" {
    bucket         = "my-sandbox2-tfbucket"
    key            = "path/to/terraform.tfstate"
    region         = "ap-southeast-1"
    dynamodb_table = "my-sandbox2-tflocks"
    encrypt        = true
  }
}

  8. Run terraform init command again. Terraform will detect changes related to S3 backend and will move terraform state to S3 bucket that has been defined. Choose yes if asked for input

    terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Releasing state lock. This may take a few moments...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

  9. Go to Amazon Web Services (AWS) web console, go to S3 service menu. Verify terraform.tfstate file has been copied to defined S3 bucket/path

CodeCatalyst Workflow File Configurations

After configurations for terraform backend has been finished as described above, next we continue configure CodeCatalyst workflow/pipeline.

Before starting workflow configuration, please make sure to configure all configurations that needed by CodeCatalyst as mentioned in Prerequisites.

As mentioned in document Build, test, and deploy with workflows in CodeCatalyst, CodeCatalyst use workflow definition file as reference workflow/pipeline configuration and this file is saved in directory ~/.codecatalyst/workflows/.

In this blog, I will create basic resources that needed by VPC on AWS. As mentioned on Directory Structure above, pipeline will be ran when there is changes inside directory sandbox2/vpc/ap-southeast-1.

Below is full content of workflow file codecatalyst/workflows/tf-sbx2-vpc-apse1.yml that used as example in this blog.

Name: tf-sbx2-vpc-apse1
SchemaVersion: "1.0"

Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox2\/vpc\/ap-southeast-1\/.*

Actions:
  terraform-plan:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2
  wait-period:
    DependsOn:
      - terraform-plan
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: echo "Please wait for a while before terraform apply"
      - Run: echo "If you wish to cancel terraform apply, please stop this run"
      - Run: sleep 60
    Compute:
      Type: EC2
  terraform-apply:
    DependsOn:
      - wait-period
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2
Enter fullscreen mode Exit fullscreen mode

Next, I will explain more detail each sections that defined on workflow file above.

  1. Define workflow name. Then define how workflow will be ran. In this blog, workflow will be ran when there is a push to master branch but only ran when changes is under directory sandbox2/vpc/ap-southeast-1/ (regular expression or regex is used to detect changes)

    Triggers:
  - Type: PUSH
    Branches:
      - master
    FilesChanged:
      - sandbox2\/vpc\/ap-southeast-1\/.*

  2. Define action. Action will be divided into 3 sections. The first one is used to run workflow terraform plan. In this section, action uses identifier aws/build@v1. Then action will be ran on sandbox environment that has been connected with AWS account and also has IAM role associated with that account. In Configuration section, has been defined that action will be ran on top of container using specific version of terraform public image that pulled from DockerHub. Also has defined working directory and several terraform commands which include terraform plan. Lastly, define compute type which is EC2

    Actions:
  terraform-plan:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform fmt -check -no-color
      - Run: terraform init -no-color
      - Run: terraform validate -no-color
      - Run: terraform plan -no-color -input=false
    Compute:
      Type: EC2

  3. Next action that defined is called wait-period. This action is defined so that there is an interlude between terraform plan and terraform apply. In case there is something that unwanted in terraform plan, so that we have a time to not continue to next action or process which is terraform apply. This action is workaround because by the time this blog is released, there is no out of the box solution provided by CodeCatalyst, for example manual approval. So in case there is something unwanted, we can abort the workflow. In this action, I created a logic to pause the workflow for 60 seconds and this action is depends on previous action which is terraform plan

    wait-period:
    DependsOn:
      - terraform-plan
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Steps:
      - Run: echo "Please wait for a while before terraform apply"
      - Run: echo "If you wish to cancel terraform apply, please stop this run"
      - Run: sleep 60
    Compute:
      Type: EC2

  4. Next action that defined is action to run terraform apply. This action more or less similar with action terraform-plan, the difference is just terraform command that declared on configuration. In this section, action will run commands that needed for terraform apply. This action is depends on previous action which is wait-period

    terraform-apply:
    DependsOn:
      - wait-period
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Name: sandbox
      Connections:
        - Name: lanandra-sandbox
          Role: codecatalyst-admin
    Configuration:
      Container:
        Registry: DockerHub
        Image: hashicorp/terraform:1.5.7
      Steps:
      - Run: cd sandbox2/vpc/ap-southeast-1
      - Run: terraform init -no-color
      - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

Example of Terraform Workflow Process in CodeCatalyst CI/CD

After configure CodeCatalyst workflow file, next I will create example of Terraform codes. In this blog, I will create VPC resources using public module terraform-aws-modules/vpc/aws as reference. Here is the initial configuration/codes that defined:

sandbox2/vpc/ap-souhteast-1/main.tf

###########################
# Terraform Configuration #
###########################

terraform {
  backend "s3" {
    bucket         = "my-sandbox-tfbucket"
    key            = "path/to/terraform.tfstate"
    region         = "ap-southeast-1"
    dynamodb_table = "my-sandbox-tflocks"
    encrypt        = true
  }
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.11.0"
    }
  }

  required_version = ">= 1.3.0"
}

provider "aws" {
  region = var.aws_region
}
Enter fullscreen mode Exit fullscreen mode

sandbox2/vpc/ap-souhteast-1/variables.tf

variable "aws_region" {
  type        = string
  description = "AWS Region"
  default     = "ap-southeast-1"
}

variable "env_name" {
  type    = string
  default = "sbx2"
}
Enter fullscreen mode Exit fullscreen mode

sandbox2/vpc/ap-souhteast-1/resources.tf

#######
# VPC #
#######


data "aws_availability_zones" "available" {}

locals {
  azs = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    "myTag:environment" = "sbx2"
    "myTag:managedBy"   = "terraform"
  }
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.1.2"

  name = var.env_name
  cidr = "10.255.8.0/21"

  azs             = local.azs
  private_subnets = ["10.255.8.0/24", "10.255.9.0/24", "10.255.10.0/24"]
  public_subnets  = ["10.255.15.0/24", "10.255.14.0/24", "10.255.13.0/24"]

  enable_nat_gateway = false
  single_nat_gateway = true

  tags = local.tags
}
Enter fullscreen mode Exit fullscreen mode

  1. To conduct testing of Terraform workflow in CI/CD, I will change 1 line of code inside file sandbox2/vpc/ap-souhteast-1/resources.tf. I will activate NAT gateway with changing this line of code that current value is false

    enable_nat_gateway = true

  2. As defined in workflow file, all changes inside directory sandbox/vpc/ap-southeast-1 will trigger workflow run. To verify, go to CodeCatalyst web console, go to space and project where codes resided. Then go to CI/CD menu, choose Workflows. Go to tab Runs, verify there is new run that currently running.

    new-run

  3. Go to run details by clicking name/id of run. Then We will be redirected to overview page of run. Click action name terraform-plan, on the right will be showed tray box that display details configuration/steps of action. For example We want to see details of terraform plan, go to tab log, then click step terraform plan -no-color -input-false. Verify action succeeded

    tf-plan-overview

  4. Verify terraform plan show expected result. Terraform will create resources releated to NAT gateway as mentioned in point number 1

    tf-plan-details

  5. As defined in workflow file, after terraform-plan action ran successfully, next workflow will proceed next action which is wait-period to give some interlude whether terraform plan will be executed to next action which is terraform-apply or apply will be aborted. If plan has met expectation and We want to proceed to terraform apply, no action needed, just wait 60 seconds and then verify action succeeded

    wait-period-successful

  6. But, if want to abort workflow and don’t want continue to terraform apply, click stop button when action still in progress

    stop-wait-period

  7. Let’s say We want to continue to terraform apply process, then terraform-apply action will be ran. We can see the details of the action inside tray box similar with previous actions. To verify resources has been provisioned via terraform apply, click step terraform apply -auto-approve -no-color -input=false

    tf-apply-overview

  8. Verify terraform apply has been expected and action is succeeded

    tf-apply-details

  9. Go to VPC service page in AWS web console to verify NAT gateway has been provisioned successfully

    verify-nat-gw

Summary

We have reached the last section of this blog. Here are some key takeaways that can be summarized:

  • Amazon CodeCatalyst can act as alternative for CI/CD engine/tools that can be used to run Terraform workflow
  • Amazon CodeCatalyst use IAM role to interact with AWS services. By using this method, Terraform doesn’t need to inject static credentials such as AWS Access Key and AWS Secret Key into the pipeline. This can’t help prevent security breaches
  • Amazon CodeCatalyst can give seamless experience if We want to deploy application to AWS environments
  • But some workflow or logic haven’t been provided by CodeCatalyst by the time this blog is released. Maybe there will be some improvement in the future

Please check out my GitHub repository to see source code example that used in this blog:

Please comment if you have any suggestions, critiques, or thoughts.

Hope this article will benefit you. Thank you!

💖 💪 🙅 🚩
lanandra
Luthfi Anandra

Posted on November 22, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

CI/CD Pipeline for Terraform Workflow Using Amazon CodeCatalyst
aws CI/CD Pipeline for Terraform Workflow Using Amazon CodeCatalyst

November 22, 2023