Terraform Certification (Part 4): Providers

One could say that the core of Terraform is just glue between the various providers that we use to create different resources.

In this lesson we go through what providers are, how we can find what providers are available, and how we use providers in our Terraform configuration.

At a high-level this lesson covers the following parts of the exam curriculum:

Part	Content
3	Understand Terraform basics
(a)	Install and version Terraform providers
(b)	Describe plugin-based architecture
(c)	Write Terraform configuration using multiple providers
(d)	Describe how Terraform finds and fetches providers

What is a provider?

A provider is an abstraction on top of an API. This API could be the API of a public cloud provider, such as AWS or Azure. The abstraction I am talking about here is that through the provider we can declaratively configure a resource, such as a virtual machine, that we want to create. We do not need to know what API-calls to make to create the resource, or in what order we need to call each API-endpoint. We just declare the resource in Terraform, and the provider does the job of calling the correct APIs behind the scenes for us.

How do we find available providers?

To find available providers we use the Terraform registry at registry.terraform.io.

Through the Terraform registry we can discover available providers, and read the documentation about a specific provider we would like to use.

Apart from providers we can also use the Terraform registry to find:

Modules: a module is a Terraform abstraction consisting of a collection of resources packaged in a reusable way. We will take a closer look at modules in a future lesson.
Policies: a policy is a rule that must be fulfilled in order for a Terraform configuration to be valid. An example of this is "you can't use an AWS virtual machine instance-size larger than m5". Policies are written in a tool called Hashicorp Sentinel and it also uses the Hashicorp Configuration Language (HCL).
Run Tasks: a run task is an integration with a third-party tool that can be configured to run at certain points in the Terraform lifecycle, for instance right before a terraform apply. This could be to configure tasks that run various security scans.

Policies and run tasks are not part of the Terraform Associate Certification, so we won't discuss them further. However, it is definitely a good thing to know that they exist!

Using a provider

Time to get hands-on! We will use the Terraform registry to find a provider and add it a to a new Terraform project.

Find a provider

I begin by navigating to registry.terraform.io and I click on Browse Providers:

I see the most common providers highlighted for me:

I select the Azure provider and arrive at the azurerm provider landing page:

Add a provider to our Terraform configuration

When you have found a provider you want to use you can click on the USE PROVIDER button to see the required HCL to add this provider to your Terraform configuration:

I create a new file called main.tf and I copy the code from the documentation:

// main.tf
terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.37.0"
    }
  }
}

provider "azurerm" {}

Here we see our first example of HCL. In HCL there are two fundamental concepts called arguments and blocks. A block consists of a type, zero or more labels, and the block content. It has the following general form:

type "label 1" "label 2" ... "label n" {
  # block content
}

In the example above we saw three blocks: terraform, required_providers, and provider. Blocks can be nested, like the required_providers is nested inside of the terraform block. Only the provider block has a label "azurerm".

Note that the provider block in the HCL above is currently empty. We will not discuss this block further in this lesson but we will come back to it in future lessons.

The other fundamental concept is an argument. Arguments assign a value to a name and it has the general form:

argument_name = "argument value"

The value can be any type. In the example above we saw three arguments:

azurerm = { ... },
source = "hashicorp/azurerm", and
version = "3.37.0"

Arguments can be nested inside of other arguments. The azurerm = { ... } argument has a complex type that includes the other two arguments.

The terraform block is where we specify all the providers that we want to use in our Terraform configuration. It is a required block as long as you want to use one or more providers.

Apart from required_providers another common thing to specify inside of the terraform block is required_version. This is an argument that you can use to enforce that a certain version of Terraform is used for this configuration. Let us include it in our terraform block, so that we now have:

// main.tf
terraform {
  required_version = "> 1.3"

  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.37.0"
    }
  }
}

provider "azurerm" {}

Here I have added required_version = "> 1.3" inside of the terraform block. This argument states that this Terraform configuration requires a Terraform version that is greater than 1.3. You would only include this argument if you are using a specific feature of Terraform that became available in a certain version. The "> 1.3" part is called a version constraint. You can use version constraints for both the Terraform version and for provider versions. There are a few different constraints you can specify:

= allow exact one specific version, e.g. =1.3.1
!= exclude a specific version, but allow the rest, e.g. !=1.3.2
> greater than (e.g. > 1.3.1), >= greater than or equal (e.g. >= 1.3.1), < less than (e.g. < 1.3.2), <= less than or equal (e.g. <= 1.3.3)
~> allow the right-most part of the semantic version number to increase, e.g. ~> 1.3.1 means that any version 1.3.X where X is greater than or equal to 1 will be accepted

In the HCL we have seen so far you might have noticed that I use two types of comments. Comments are just like comments in any other programming language, a text block used to clarify something. In HCL there are three types of comments, illustrated in the following code block:

// this is a comment

# this is a comment

/*
  This is a multi-line
  comment
*/

Initializing Terraform

Once you have declared the required providers that your Terraform configuration needs, you can initialize your Terraform project. This is done with the terraform init command:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/azurerm versions matching "3.37.0"...
- Installing hashicorp/azurerm v3.37.0...
- Installed hashicorp/azurerm v3.37.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

The output shows a few things going on. First of all it says Initializing the backend.... We have not covered the concept of backend yet in this course so we will skip that part for now. The next thing that happens is Initializing provider plugins..., where Terraform reads the required_providers block and downloads the specified versions for each provider. In this case it installs version 3.37.0 of the Azure provider.

After running terraform init we might observe that we have additional files in our current directory:

$ tree -a .

.
├── .terraform
│   └── providers
│       └── registry.terraform.io
│           └── hashicorp
│               └── azurerm
│                   └── 3.37.0
│                       └── darwin_arm64
│                           └── terraform-provider-azurerm_v3.37.0_x5
├── .terraform.lock.hcl
└── main.tf

In the .terraform directory Terraform has stored the binary for each provider it downloaded. In this case it is a single provider binary located in .terraform/providers/registry.terraform.io/hashicorp/azurerm/3.37.0/darwin_arm64/.

We can also see that we have a .terraform.lock.hcl file:

$ cat .terraform.lock.hcl

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/azurerm" {
  version     = "3.37.0"
  constraints = "3.37.0"
  hashes = [
    "h1:tD9TmGFgYV/oxZQu0pXuA46H+ML9nALCDwFqoaETjGg=",
    "zh:2a7bda0b7679d1c791c762103a22f333b544b6e6776c4177f33bafc9cc28c919",
    "zh:49ff49670c349f918017315838a43ece09bf6f1bf7721b992f1cadbceb273c62",
    "zh:55c9346d03380585e17616b79c4233b726d6fb9efa1921848834fc881e5d7d54",
    "zh:5ab117b56a4236ea29926e9d95c27d7bf8ae6706d0fffb76c0b1bfe67bf3a78e",
    "zh:5cfc086d5d56308edb3e68aac5f8a448ddc6e56541be7b152ae886399e9b2c69",
    "zh:7a8929ed38152aac6652711f32193c8582bc996f8fa73879a3ac7a9bf88d2460",
    "zh:895294e90a37f719975fcd2269b95e973147e48ec0ebb9c2fe472bc93531b49c",
    "zh:8baa5e2b6e5b02df5b45d253a3aea93f22619920cf9577290d682b59a6d5664b",
    "zh:b146a732c7909238c10d216b92a35092be4f72a0509a4c6742cc3245bf3b3bf3",
    "zh:cedef898ccd512a6519eae3dff7eb0d581d2c3dad8e0001992da16ad1d7fded8",
    "zh:f016d9ba94ea88476883b4d63cff88a0225974e0a8b8c3e8555f73c5de6f7119",
    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
  ]
}

This file lists what provider versions have been installed. The .terraform.lock.hcl is similar to a package-lock.json file in a Node.js project. It should be included in your source repository.

One thing we have glossed over so far is the source argument we used when we specified a provider in the required_providers block:

azurerm = {
  source = "hashicorp/azurerm"
  version = "3.37.0"
}

We say that the azurerm provider comes from source = "hashicorp/azurerm". This is technically a short-hand for saying source = "registry.terraform.io/hashicorp/azurerm". The general format of the source is <HOSTNAME>/<NAMESPACE>/<TYPE>, but if the <HOSTNAME> part is left out it will default to the Terraform registry address registry.terraform.io. With that said, it is perhaps clear that we could use other sources for our providers, it does not have to be the official Terraform registry.

Upgrading a provider

When you run terraform init for a given Terraform configuration, Terraform will look at your terraform block as well as your .terraform.lock.hcl file if it exists to determine what version of a provider to install. If the .terraform.lock.hcl file exists it will take the version that is specified there, if not it will download the version that fulfills the version constraint in the terraform block.

If you at some point need to upgrade the provider you can edit the version constraint in the terraform block, and then run terraform init -upgrade. If I had version 3.36.0 of the Azure provider installed and I edit the version constraint to =3.37.0 and run terraform init -upgrade this is what I would see:

$ terraform init -upgrade

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/azurerm versions matching "3.37.0"...
- Installing hashicorp/azurerm v3.37.0...
- Installed hashicorp/azurerm v3.37.0 (signed by HashiCorp)

Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

$ tree -a .

.
├── .terraform
│   └── providers
│       └── registry.terraform.io
│           └── hashicorp
│               └── azurerm
│                   ├── 3.36.0
│                   │   └── darwin_arm64
│                   │       └── terraform-provider-azurerm_v3.36.0_x5
│                   └── 3.37.0
│                       └── darwin_arm64
│                           └── terraform-provider-azurerm_v3.37.0_x5
├── .terraform.lock.hcl
└── main.tf

9 directories, 4 files

I can see that now I have two versions of the Azure provider in my .terraform directory.

A few common providers

There are a few common providers that could come up in the certification.

Local provider

The local provider is used to read and write local files. It exposes two resources: local_file and local_sensitive_file, as well as two data sources with the same names. We have not covered data sources yet, but what they allow us to do with this provider is to read existing files.

An example of how to include the local provider in your terraform block is shown below:

terraform {
  required_providers {
    local = {
      source = "hashicorp/local"
      version = "2.2.3"
    }
  }
}

Why would you use the local provider? If you need to read a local file containing content you wish to provide to another resource. This could be configuration data, an image, an HTML-file, or something else. You could also use it to generate a file containing data from another resource you created with Terraform, and then pass that file along somewhere else.

Random provider

The random provider is used to introduce randomness into your Terraform configuration. This could be to generate a random name or a random number. It would not be very useful if it generated pure random values every time you ran Terraform, so it only generates random values once for a given input, and then holds on to those values until the inputs change.

The random provider is an example of a logical provider. A logical provider is a provider that does not interact with an external API, instead it does all of its work inside of Terraform itself.

An example of how to include the random provider in your terraform block is shown below:

terraform {
  required_providers {
    random = {
      source = "hashicorp/random"
      version = "3.4.3"
    }
  }
}

The random provider only includes the following resources: random_id, random_integer, random_password, random_pet, random_shuffle, random_string, and random_uuid. Most resource names are self-explanatory, but you can read more about this provider and all the resources it exposes at registry.terraform.io/providers/hashicorp/random/.

Why do you want to use the random provider? Usually it is used to generate random prefixes or suffixes that are part of other resource names.

Null provider

The null provider exposes a single resource called null_resource and a single data source called null_data_source.

An example of how to include the null provider in your terraform block is shown below:

terraform {
  required_providers {
    null = {
      source = "hashicorp/null"
      version = "3.2.1"
    }
  }
}

The resource and data source in the null provider intentionally do nothing. To cite the documentation for what they are used for:

... they can be useful in various situations to help orchestrate tricky behavior or work around limitations.

I will not provide any example of when to use the null provider here, but we might see examples of it in a future lesson.

Using multiple providers

How do you go about to use multiple providers in the same Terraform configuration? Do we need several terraform blocks, each with its own required_providers block? No, you only use a single terraform block. This is what it would look like if we wanted to use the local, random, and null providers in the same Terraform configuration:

terraform {
  required_providers {
    local = {
      source = "hashicorp/local"
      version = "2.2.3"
    }

    random = {
      source = "hashicorp/random"
      version = "3.4.3"
    }

    null = {
      source = "hashicorp/null"
      version = "3.2.1"
    }
  }
}

Do we need to provide a version for our providers?

What happens if we don't provide a specific version of a provider? The latest available version will be used. Let's say we have the following main.tf:

// main.tf
terraform {
  required_providers {
    local = {
      source = "hashicorp/local"
    }
  }
}

If I run terraform init with this configuration I get the following output:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/local...
- Installing hashicorp/local v2.2.3...
- Installed hashicorp/local v2.2.3 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

Version v2.2.3 was selected for me, which at the time of writing is the latest available version.

Summary

A summary of the concepts we learned in this lesson:

Providers:
- A provider is an abstraction over an API.
- A provider allows Terraform to use an API through HCL syntax.
- You can browse available providers and read documentation on the Terraform registry.
- We saw how to install a provider with terraform init
- We saw how to upgrade a provider with terraform init -upgrade
- We learned about version constraints for providers and for the Terraform version itself
- We saw examples of three common but unusual providers: local, random, and null. These are used in special cases, and could come up in the certification exam.
- We learned how to use multiple providers in the same Terraform configuration
Hashicorp Configuration Language
- We saw that there are two fundamental HCL concepts: arguments and blocks.
- We saw three examples of blocks: terraform, required_providers, and provider.
- We saw how to specify a minimum required Terraform version with the required_version argument inside of the terraform block.
- We saw how to create comments with // comment, # comment, and /* comment */.

What we have seen here includes 95% of everything I have ever used when it comes to providers. So I am fairly certain you won't need to know more than this about providers to pass the certification. What is included in the last 5% you ask? It is provider aliases and how to pass providers to modules. We will see these concepts in a future lesson.