TheLazy.Dev
Portfolio GitHub

Blog

terraform |azure |iac |azuredevops

Dynamic terraform deployments using DevOps replace tokens

pwd9000

Marcel.L

Posted on August 26, 2021

Dynamic terraform deployments using DevOps replace tokens

Replace tokens

Replace tokens is a DevOps extension that can be installed into your DevOps Organisation from the Azure DevOps marketplace, simply put it is an Azure Pipelines extension that replace tokens in files with variable values. Today we will look at how we can use this Devops extension working with a terraform HCL code base, to dynamically deploy infrastructure hosted on Azure based on environments defined as variables in DevOps using terraform.

Installing Replace Tokens

Before we can use replace tokens we have to install it into our Devops Organisation from the marketplace.

Go to DevOps Organisation Settings and select the Extensions tab followed by Browse marketplace and search for Replace tokens. In addition also install the terraform extension called Terraform by Microsoft DevLabs as we will use this later on to use terraform tasks in our DevOps pipeline.

ado_task

Project layout and objective

For this tutorial we will write a simple terraform configuration that will deploy a resource group, but we will use the replace tokens task to manipulate our configuration file to deploy 3 different resource groups based on environment. For example Infra-dev-Rg, Infra-uat-Rg and Infra-prod-Rg. I have set up a new project in my organisation called DynamicTerraform, I also created a repository called Infrastructure. Inside of my repository I have created the following paths:

  • /terraform-azurerm-resourcegroup This location will be my root path used to store the main terraform configuration files which will be used to deploy a simple resource group.
  • /terraform-azurerm-resourcegroup/pipelines This location will be used to store and configure yaml deployment pipelines for the resources in my root path.
  • /terraform-azurerm-resourcegroup/pipelines/variables This location will be used to store and configure yaml variable template files used for my pipelines.
  • /terraform-azurerm-resourcegroup/pipelines/task_groups This location will be used to store and configure yaml tasks/steps used in my pipelines.

Any additional future resources can be created in new root paths e.g.: /terraform-azurerm-resourceX, /terraform-azurerm-resourceY, /terraform-azurerm-resourceZ etc... For this tutorial we will just be using /terraform-azurerm-resourcegroup to deploy multiple resource groups dynamically based on an environment e.g. dev, uat and prod. This is what the DevOps project layout looks like:

repo_layout

Terraform Configuration

As a pre-requisite I have also pre-created an Azure DevOps service connection that will be used to allow my pipelines to access Azure via the terraform task we installed earlier, and I also pre-created an Azure storage account which will act as my terraform backend to safely store my terraform state files in.

Under my repo path: /terraform-azurerm-resourcegroup/, I have created the following three terraform files:

  1. main.tf (Main terraform configuration file)


   # code/terraform-azurerm-resourcegroup/main.tf

   ##################################################
   # Terraform Config                               #
   ##################################################
   terraform {
     required_version = ">= ~{terraformVersion}~"

     backend "azurerm" {
       resource_group_name  = "~{terraformBackendRG}~"
       storage_account_name = "~{terraformBackendSA}~"
       container_name       = "tfstate"
       key                  = "infra_~{environment}~_rg.tfstate"
     }

     required_providers {
       azurerm = {
         source  = "hashicorp/azurerm"
         version = "~> 2.73"
       }
     }
   }

   provider "azurerm" {
     features {}
     skip_provider_registration = true
   }

   ##################################################
   # RESOURCES                                      #
   ##################################################
   resource "azurerm_resource_group" "resource_group" {
     name     = var.resource_group_name
     location = var.location
     tags     = var.tags
   }
Enter fullscreen mode Exit fullscreen mode

NOTE: In the terraform configuration you will notice the following values: ~{terraformVersion}~, ~{terraformBackendRG}~, ~{terraformBackendSA}~ and ~{environment}~, we will be dynamically changing the values inside of ~{ }~ with values from our pipeline variable file later on in this tutorial using replace tokens.

  1. variables.tf (Terraform variable definition file)


   # code/terraform-azurerm-resourcegroup/variables.tf

   variable "resource_group_name" {
     type        = string
     description = "Specifies the name of the resource group that will be created."
   }

   variable "location" {
     type        = string
     description = "The location/region where Azure resource will be created."
   }

   variable "tags" {
     type        = map(any)
     description = "Specifies a map of tags to be applied to the resources created."
   }
Enter fullscreen mode Exit fullscreen mode
  1. resourcegroup.auto.tfvars (Terraform variables which will be dynamically changed by replace tokens task)


   # code/terraform-azurerm-resourcegroup/resourcegroup.auto.tfvars

   resource_group_name = "Infra-~{environment}~-Rg"
   location            = "~{location}~"
   tags = {
     terraformDeployment = "true"
     Environment         = "~{environment}~"
   }
Enter fullscreen mode Exit fullscreen mode

NOTE: In the TFVARS configuration file you will notice the following values: ~{environment}~ and ~{location}~, we will be dynamically changing the values inside of ~{ }~ with values from our pipeline variable file later on in this tutorial using replace tokens.

DevOps Pipeline Variable files

Under my repo path: /terraform-azurerm-resourcegroup/pipelines/variables/, I have created the following four yaml variable template files:

  1. common_vars.yml (Declares variables that will be used in all pipelines).


   # code/terraform-azurerm-resourcegroup/pipelines/variables/common_vars.yml

   variables:
     #Terraform Config + backend
     - name: terraformVersion
       value: '1.0.5'

     - name: terraformBackendRG
       value: 'TF-Core-Rg'

     - name: terraformBackendSA
       value: 'tfcorebackendsa'

     #Variables used for service connection
     - name: AzureServiceConnection
       value: 'TF-Terraform-SP'

     - name: rootDirName
       value: 'terraform-azurerm-resourcegroup'
Enter fullscreen mode Exit fullscreen mode
  1. dev_vars.yml (Declares variables that will be used in DEV specific pipeline).


   # code/terraform-azurerm-resourcegroup/pipelines/variables/dev_vars.yml

   variables:
     #Development Variables
     - name: environment
       value: 'dev'

     - name: location
       value: 'uksouth'
Enter fullscreen mode Exit fullscreen mode
  1. uat_vars.yml (Declares variables that will be used in UAT specific pipeline).


   # code/terraform-azurerm-resourcegroup/pipelines/variables/uat_vars.yml

   variables:
     #UAT Variables
     - name: environment
       value: 'uat'

     - name: location
       value: 'uksouth'
Enter fullscreen mode Exit fullscreen mode
  1. prod_vars.yml (Declares variables that will be used in PROD specific pipeline).


   # code/terraform-azurerm-resourcegroup/pipelines/variables/prod_vars.yml

   variables:
     #Production Variables
     - name: environment
       value: 'prod'

     - name: location
       value: 'ukwest'
Enter fullscreen mode Exit fullscreen mode

NOTE: You will notice that the variable names in each yaml template are aligned with the values used on the terraform configuration files earlier: ~{environment}~, ~{location}~, ~{terraformBackendRG}~, ~{terraformBackendSA}~. Also note that our production variable file has a different location specified: ukwest.

DevOps Pipelines

Under my repo path: /terraform-azurerm-resourcegroup/pipelines/, I have created the following three yaml pipelines (one for each environment):

  1. dev_deployment.yml (Deploy dev RG - Pipeline)


   # code/terraform-azurerm-resourcegroup/pipelines/dev_deployment.yml

   name: Deployment-Dev-RG-$(Rev:rr)
   trigger: none

   variables:
     - template: variables/common_vars.yml
     - template: variables/dev_vars.yml

   stages:
     - stage: TF_DEPLOY_DEV_RG
       displayName: Deploy Dev ResourceGroup
       dependsOn: []
       jobs:
         - deployment: TF_Deploy_Dev_Rg
           displayName: Terraform - Dev - RG
           pool:
             name: Azure Pipelines
             vmImage: windows-latest
           workspace:
             clean: all
           environment: Infra-Dev
           strategy:
             runOnce:
               deploy:
                 steps:
                   - checkout: self
                   ### Run common terraform deploy steps
                   - template: task_groups/tf_deploy_tasks.yml
                     parameters:
                       terraformVersion: ${{ variables.terraformVersion }}
                       rootDirName: ${{ variables.rootDirName }}
                       AzureServiceConnection: ${{ variables.AzureServiceConnection }}
                       terraformBackendRG: ${{ variables.terraformBackendRG }}
                       terraformBackendSA: ${{ variables.terraformBackendSA }}
                       environment: ${{ variables.environment }}
Enter fullscreen mode Exit fullscreen mode
  1. uat_deployment.yml (Deploy uat RG - Pipeline)


   # code/terraform-azurerm-resourcegroup/pipelines/uat_deployment.yml

   name: Deployment-UAT-RG-$(Rev:rr)
   trigger: none

   variables:
     - template: variables/common_vars.yml
     - template: variables/uat_vars.yml

   stages:
     - stage: TF_DEPLOY_UAT_RG
       displayName: Deploy Uat ResourceGroup
       dependsOn: []
       jobs:
         - deployment: TF_Deploy_Uat_Rg
           displayName: Terraform - Uat - RG
           pool:
             name: Azure Pipelines
             vmImage: windows-latest
           workspace:
             clean: all
           environment: Infra-Uat
           strategy:
             runOnce:
               deploy:
                 steps:
                   - checkout: self
                   ### Run common terraform deploy steps
                   - template: task_groups/tf_deploy_tasks.yml
                     parameters:
                       terraformVersion: ${{ variables.terraformVersion }}
                       rootDirName: ${{ variables.rootDirName }}
                       AzureServiceConnection: ${{ variables.AzureServiceConnection }}
                       terraformBackendRG: ${{ variables.terraformBackendRG }}
                       terraformBackendSA: ${{ variables.terraformBackendSA }}
Enter fullscreen mode Exit fullscreen mode
  1. prod_deployment.yml (Deploy prod RG - Pipeline)


   # code/terraform-azurerm-resourcegroup/pipelines/prod_deployment.yml

   name: Deployment-Prod-RG-$(Rev:rr)
   trigger: none

   variables:
     - template: variables/common_vars.yml
     - template: variables/prod_vars.yml

   stages:
     - stage: TF_DEPLOY_PROD_RG
       displayName: Deploy Prod ResourceGroup
       dependsOn: []
       jobs:
         - deployment: TF_Deploy_Prod_Rg
           displayName: Terraform - Prod - RG
           pool:
             name: Azure Pipelines
             vmImage: windows-latest
           workspace:
             clean: all
           environment: Infra-Prod
           strategy:
             runOnce:
               deploy:
                 steps:
                   - checkout: self
                   ### Run common terraform deploy steps
                   - template: task_groups/tf_deploy_tasks.yml
                     parameters:
                       terraformVersion: ${{ variables.terraformVersion }}
                       rootDirName: ${{ variables.rootDirName }}
                       AzureServiceConnection: ${{ variables.AzureServiceConnection }}
                       terraformBackendRG: ${{ variables.terraformBackendRG }}
                       terraformBackendSA: ${{ variables.terraformBackendSA }}
Enter fullscreen mode Exit fullscreen mode

DevOps Pipelines - Task group

Under my repo path: /terraform-azurerm-resourcegroup/pipelines/task_groups/, I have created the following common yaml tasks/steps template which defines common steps that will be used in each pipeline:



# code/terraform-azurerm-resourcegroup/pipelines/task_groups/tf_deploy_tasks.yml

parameters:
  terraformVersion:
  rootDirName:
  AzureServiceConnection:
  terraformBackendRG:
  terraformBackendSA:
  environment:

steps:
  ### Install Terraform Version from commom_vars
  - task: TerraformInstaller@0
    inputs:
      terraformVersion: ${{ parameters.terraformVersion }}

  ### replace tokens in tf and tfvars.
  - task: qetza.replacetokens.replacetokens-task.replacetokens@3
    displayName: 'Replace tokens in tfvars and tf'
    inputs:
      rootDirectory: '$(System.DefaultWorkingDirectory)'
      targetFiles: |
        ${{ parameters.rootDirName }}\*.tf
        ${{ parameters.rootDirName }}\*.tfvars
            encoding: 'utf-8'
      actionOnMissing: 'warn'
      keepToken: false
      tokenPrefix: '~{'
      tokenSuffix: '}~'

  ### Terraform Init
  - task: TerraformTaskV2@2
    displayName: Terraform Init
    inputs:
      provider: 'azurerm'
      command: 'init'
      workingDirectory: '$(System.DefaultWorkingDirectory)/${{ parameters.rootDirName }}'
      backendServiceArm: '${{ parameters.AzureServiceConnection }}'
      backendAzureRmResourceGroupName: '${{ parameters.terraformBackendRG }}'
      backendAzureRmStorageAccountName: '${{ parameters.terraformBackendSA }}'
      backendAzureRmContainerName: 'tfstate'
      backendAzureRmKey: 'Infra_${{ parameters.environment }}_rg.tfstate'

  ### Terraform Plan
  - task: TerraformTaskV2@2
    displayName: Terraform Plan
    inputs:
      provider: 'azurerm'
      command: 'plan'
      workingDirectory: '$(System.DefaultWorkingDirectory)/${{ parameters.rootDirName }}'
      commandOptions: '--out=$(System.DefaultWorkingDirectory)/${{ parameters.rootDirName }}/plan.tfplan'
      environmentServiceNameAzureRM: '${{ parameters.AzureServiceConnection }}'

  ### Terraform Apply
  - task: TerraformTaskV2@2
    displayName: Terraform Apply
    inputs:
      provider: 'azurerm'
      command: 'apply'
      workingDirectory: '$(System.DefaultWorkingDirectory)/${{ parameters.rootDirName }}'
      environmentServiceNameAzureRM: '${{ parameters.AzureServiceConnection }}'
Enter fullscreen mode Exit fullscreen mode

Note that the replace tokens task is defined and configured to replace the variables we defined within the tokenPrefix: ~{ and tokenSuffix: }~ as you can see below:



### replace tokens in tf and tfvars.
- task: qetza.replacetokens.replacetokens-task.replacetokens@3
  displayName: 'Replace tokens in tfvars and tf'
  inputs:
    rootDirectory: '$(System.DefaultWorkingDirectory)'
    targetFiles: |
      ${{ parameters.rootDirName }}\*.tf
      ${{ parameters.rootDirName }}\*.tfvars
          encoding: 'utf-8'
    actionOnMissing: 'warn'
    keepToken: false
    tokenPrefix: '~{'
    tokenSuffix: '}~'
Enter fullscreen mode Exit fullscreen mode

Now we can configure each pipeline, which will consume its own corresponding variable template file as well as a common variable template file, but use the same terraform configuration code to dynamically deploy the same resource group but each having its own state file, name and tags dynamically.

pipelines

Also remember to set the environments in Azure DevOps as shown on each of our yaml pipelines e.g.:



# code/terraform-azurerm-resourcegroup/pipelines/dev_deployment.yml#L21-L21

environment: Infra-Dev
Enter fullscreen mode Exit fullscreen mode

environments

After each pipeline has been run, you will notice that our terraform configuration was dynamically changed each time with the replace tokens task, replacing the values on our TF and TFVARS files.

replace_token

You'll also see the each resource group have been dynamically created.

rg_dep

NOTE: Remember we changed location to be in the UK West region on our variable template file for prod.

Also note that each of the deployments have their own unique state file based on the environment as depicted on each of the yaml pipelines and declared in the variable files e.g.:



# code/terraform-azurerm-resourcegroup/pipelines/dev_deployment.yml#L58-L58

backendAzureRmKey: 'Infra_${{ variables.environment }}_rg.tfstate'
Enter fullscreen mode Exit fullscreen mode

state

I hope you have enjoyed this post and have learned something new. You can also find the code samples used in this blog post on my GitHub page. ❤️

Author

Like, share, follow me on: 🐙 GitHub | 🐧 X/Twitter | 👾 LinkedIn

pwd9000 image

Marcel.L

Dual Microsoft MVP DevOps + Azure | DevOps Architect | Technical speaker focused on Microsoft technologies, IaC and automation in Azure. Find me on GitHub: https://github.com/Pwd9000-ML

💖 💪 🙅 🚩
pwd9000
Marcel.L

Posted on August 26, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Terraform - Variable Validation
terraform Terraform - Variable Validation

August 28, 2023

Terraform - Understanding the Lifecycle Block
terraform Terraform - Understanding the Lifecycle Block

June 1, 2023

Terraform - Selective configuration with 'lookup()'
terraform Terraform - Selective configuration with 'lookup()'

April 14, 2022

Terraform - Filter results using 'for' loops
terraform Terraform - Filter results using 'for' loops

April 4, 2022

Terraform - Creating dynamic variables using locals
terraform Terraform - Creating dynamic variables using locals

March 23, 2022