Enhancing Terraform with Terragrunt

Over the last few months, I've become very familiar with using Terraform to deploy infrastructure across multiple cloud vendors. I'm a big fan of it, but some of its idiosyncrasies can be frustrating, particularly as the size of your project begins to increase.

Fortunately, there's a tool that helps to mitigate many of these issues; Terragrunt.

Below is a short post on some of the features of Terragrunt that I've found useful in my projects. I've also created an example that can hopefully better illustrate the benefits I'm writing about. Provided you have the following, you should be able to spin up the example resources yourself:

What is Terragrunt?

Terragrunt is a thin wrapper that overlays Terraform that allows you to avoid having large amounts of repetition in your infrastructure as code, as well as helping with Terraform modules and remote state.

DRY Providers

You can use the generate block in your root terragrunt.hcl file to inject common configuration files. One of the most common uses for it is to use it to generate your provider configuration. By adding the generate block in your root, you can delete the provider.tf files in your other directories. Whenever any Terragrunt command is called, the providers will be copied to the appropriate directories.

The change in my tutorial repo can be found here

Executing on multiple modules

When you use Terraform, you can't run commands against all of your modules at the same time. With Terragrunt, you can. If you add a terragrunt.hcl file to each module, you can run commands using run-all from root to run every module. Terragrunt recursively looks through each of your directories for the presence of that file, and if it finds it, will run the command you've included with run-all.

For example, if you wanted to apply all of the modules in this solution, you can run the following commands from the root of your repo:

terragrunt run-all init
terragrunt run-all apply

If you subsequently wanted to tear down everything you had, you can run:

terragrunt run-all destroy

Parallelism

Another nice feature of Terragrunt is when you run commands such as run-all, it will run as many of the modules that have terragrunt.hcl files as possible in parallel. Terragrunt doesn't put a constraint on this, which is a blessing and a curse. Your infrastructure deployment times will be faster, but if you're not careful you can get rate limited by your provider, so use it with caution.

If you do start to experience rate limits, you can specify the --terragrunt-parallelism flag:

terragrunt run-all apply --terragrunt-parallelism 1

The change in my tutorial repo can be found here

DRY CLI Flags

It's quite likely that when you run Terraform you're running commands with extra arguments. For example, you may have some common variables that you pass for each module, or you may want to restrict parallelism to avoid rate limiting.

With Terragrunt, you can specify an extra_arguments block that makes sure that commands that you repeat each time are always included.

terraform {
  # Ensures paralellism never exceed two modules at any time
  extra_arguments "reduced_parallelism" {
    commands  = get_terraform_commands_that_need_parallelism()
    arguments = ["-parallelism=2"]
  }

  extra_arguments "common_tfvars" {
    commands = get_terraform_commands_that_need_vars()

    required_var_files = [
      "${get_parent_terragrunt_dir()}/tfvars/common.tfvars"
    ]
  }
}

The change in my tutorial repo can be found here

Before and After Hooks

Terragrunt gives you the ability to run commands before and after the execution of a Terraform command. For example, you may run a script to bootstrap your environment or clean up after an apply has been run. You may want to copy files to certain locations, or even just add extra information that can be outputted as part of the terragrunt commands.

  before_hook "before_hook" {
    commands     = ["apply"]
    execute      = ["echo", "Applying my terraform"]
  }

  after_hook "after_hook" {
    commands     = ["apply"]
    execute      = ["echo", "Finished applying Terraform successfully!"]
    run_on_error = false
  }

Commit here

Auto-init

Something that people will have had to do countless times when running Terraform is running terraform init before running any other Terraform command. Running this command initialises your working directory by installing provider plugins and initialising your backend configuration. A feature of Terragrunt is that you don't need to explicitly call init, it will do this automatically before any other commands being run.

Generally auto-init works fine, but I have found occasions where it hasn't worked as expected. If you want to be totally sure that your modules are initialised before you apply/destroy/plan, you can add in your own before_hook to cover off the scenario.

terraform {
  before_hook "auto_init" {
    commands = ["validate", "plan", "apply", "destroy", "workspace", "output", "import"]
    execute  = ["terraform", "init"]
  }
}

The change in my tutorial repo can be found here

CLI options

Outside of the commands we've been using with Terragrunt, much of which mirrors the commands you would see with Terraform, there are a few particularly useful CLI options to use with Terragrunt as well. This list isn't exhaustive (you can find that here), it's just a list of commands I have found useful in the past:

terragrunt run-all apply --terragrunt-non-interactive

This will stop interactive prompts from being displayed, and will prompt all answers with a "yes". Particularly useful when running in CI/CD environments.

terragrunt run-all apply --terragrunt-working-dir registry

You can pass directories to Terragrunt to indicate where the Terraform command should run. Note that for any *-all commands, it will start from that directory but run any subsequent sub-folders with a terragrunt.hcl file.

terragrunt run-all apply --terragrunt-exclude-dir registry
terragrunt run-all apply --terragrunt-include-dir registry

The commands allow you to explicitly exclude and include certain modules (and their dependencies).

terragrunt run-all apply --terragrunt-parallelism 1

When passed in, limit the number of modules that are run concurrently to this number during *-all commands. Note that this parallelism is markedly different from the one you specify using Terraform directly. With Terraform, parallelism indicates running resource creation concurrently. With Terragrunt, it indicates running entire modules concurrently.

terragrunt run-all apply --terragrunt-log-level trace

If you want some more detail in your output, you can modify the log levels to do so.

Summary

Terragrunt is a fantastic foil for Terraform, taking you the extra yards to make your experience of using Terraform that much better. I've worked on a few projects using Terraform, and my advice would be to leave Terragrunt to the side, at least initially. It's important to understand the problems you're trying to solve, and when they start to manifest for you, that's the time to pull the trigger.

I've found that the value of Terragrunt increases exponentially with the size of your project. Smaller ones can generally be managed easily with "raw" Terraform, whereas larger, multi-module projects start to really benefit from Terragrunts features.

Resources

Credit

Goran Ivos for the photo 📷

Blog