TheLazy.Dev
Portfolio GitHub

Blog

terraform |gcp |tutorial

Exploring GCP With Terraform: Setting Up The Environment And Project

madmaxx

Robert Nemet

Posted on August 19, 2023

Exploring GCP With Terraform: Setting Up The Environment And Project

This topic is nothing new. There are many articles and tutorials on the internet about this. I'll be setting up a Terraform project to explore GCP. I cover the basics of Terraform and GCP.

Read Before Proceed..
I replaced the real project ID with project-id in the following examples. You need to replace it with your project ID.

Prerequisites

Create GCP Project

First, you must create a GCP account, then create a project and set up billing on that project. If you already have a Gmail account, you can use it to make a GCP account. Head to the Google Cloud Console and create a new project.

My plan is to pay for the services I use. I need to feel the pain and learn more about cost optimization. You can be more intelligent than me and use the free tier for the first year. After that, you will be charged for the services you use. You can find more information about the free tier here.

DevCube | Robert Nemet | Substack

Weekly rant about software design, devops, kubernetes, sre... Click to read DevCube, by Robert Nemet, a Substack publication. Launched 6 months ago.

favicon rnemet.substack.com

Install And Set Up The gcloud CLI Tool

The gcloud CLI is a part of the Google Cloud SDK. You can find the installation instructions for gcloud CLI here. Next, you need to set up the gcloud CLI tool. Docs are here.

For future reference, you can find the list of all gcloud commands here.

After initialization, you can always check what you set with the following command:

$ gcloud config list [SECTION/PROPERTY] [--all]
Enter fullscreen mode Exit fullscreen mode

Install Terraform CLI Tool

You can find the Terraform installation instructions here. But I'm not following the instructions.

I'm using the tool tfenv to manage Terraform versions. Other tools can do that. You can use asdf, too. I saw that asdf can do more than manage Terraform versions.

Let me explain how I do it since I use tfenv. I assume you already installed it. I use tfenv to install the desired version of Terraform. And to set it as default:

$ tfenv install 1.5.5
$ tfenv use 1.5.5
Enter fullscreen mode Exit fullscreen mode

You can check the version with the following command:

$ terraform --version
Enter fullscreen mode Exit fullscreen mode

I'll explain later why I use tfenv.

Other Tools I Use

Other tools I'll be using are:

  • task - a task runner and a replacement for make
  • git - version control system
  • direnv - unconsciously set and unset environment variables

Setting Up The Project

I'm using this layout for now:

.                                       # project root                 
├── README.md                           # project README
├── LICENSE                             # project LICENSE
├── .gitignore                          # project .gitignore
├── Taskfile.yml                        # project taskfile
└── gcp                                 # GCP Terraform code
    └── base                            # base GCP Terraform code
        ├── README.md                   # base README
        ├── .terraform-version          # base terraform version
        ├── main.tf                     # base main.tf
        ├── outputs.tf                  # base outputs.tf
        ├── variables.tf                # base variables.tf
        ├── terraform.tfvars            # base terraform.tfvars
        └── provider.tf                 # base provider.tf
Enter fullscreen mode Exit fullscreen mode

Let me quickly explain. The gcp directory is the root directory for all GCP Terraform code. The base directory is the root directory for all base GCP Terraform code. That base code would cover the following, for now:

  • Creating a bucket for storing Terraform state
  • Creating Key and KeyRing for encrypting Terraform state file
  • Setting up Terraform's backend
  • Setting up Terraform provider
  • Setting up Terraform base variables

Just create the directories and files, and we are ready to go.

How I use tfenv?
Oh, one small thing. The file .terraform-version is used by tfenv to set and pin the Terraform version. Even if I set with tfenv the default Terraform version, I use this file to say which Terraform version should be used for the base. If you are using tfenv do:
$ cd gcp/base
$ tfenv pin
Pinned version by writing "1.5.5" to /somepath/gcp/base/.terraform-version

With this, it does not matter what is the default Terraform version. The tfenv will use the version from the .terraform-version file. If, for some reason, you
do not have the desired version, tfenv will install it for you.

Setting Up Provider

The first thing we need to do is to set up the provider. I'm using the latest version of the provider. You can find the latest version here:

provider "google" {
  project = var.project_id
  region  = var.region
  zone    = var.zone
}

terraform {
  required_version = ">=1.5.5"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.77.0"
    }
  }

  backend "local" {
      path = "local-state.tfstate"
  }
}
Enter fullscreen mode Exit fullscreen mode

I'm using the local backend for now. I plan to switch to the gcs backend later. The local backend is used for storing the Terraform state locally. The state file is called local-state.tfstate. And it will contain the Terraform state for the base. It is not encrypted. My plan is to keep it safe and encrypted.

Block terraform sets the required Terraform provider version, provider, and backend. You can find more here.

The provider block configures the provider by setting the project ID, region, and zone. You probably noticed that I'm using variables.

Setting Up Variables

I'm using variables for defining variables I'll be using in the base. I'm using the following variables:

variable "project_id" {
  description = "project id"
  type        = string
}

variable "region" {
  description = "default region"
  type        = string
}

variable "zone" {
  description = "default zone"
  type        = string
}
Enter fullscreen mode Exit fullscreen mode

Once you defined the variables, you need to set them. You can do that in the terraform.tfvars file:

project_id = "my-project"
region     = "europe-west1"
zone       = "europe-west1-b"
Enter fullscreen mode Exit fullscreen mode

You should avoid committing the terraform.tfvars file. It may contain sensitive information. I'll show you how to handle that later. But for now, you must create the terraform.tfvars file and set the variables. For the project_id variable, you need to set the project ID you created earlier. Choose the region and zone that is closest to you. You can find the list of regions and zones here.

Regions&Zones Important?
Is this important? Yes, it is.

You need to choose the region and zone carefully, considering reliability, availability, and cost into account. For your users to have the best experience, your servers must be close to them and deliver the content quickly. It would be best if you considered the cost, too.

To learn more about choosing the correct region and zone, you can find more here in Google documentation.

Init, Plan, and Apply

At this moment, I can initialize the Terraform workflow:

$ terraform init


Initializing the backend...

Successfully configured the backend "local"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding hashicorp/google versions matching "4.77.0"...
- Installing hashicorp/google v4.77.0...
- Installed hashicorp/google v4.77.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Enter fullscreen mode Exit fullscreen mode

This will create a directory .terraform and file .terraform.lock.hcl file. You should include .terraform.lock.hcl in your commit. It is used to lock the provider version. If you run terraform init again, it will use the same provider version. The folder .terraform stores the provider plugins and other files required for Terraform to work.

Next, I can run terraform plan:

$ terraform plan

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
Enter fullscreen mode Exit fullscreen mode

This will show me what Terraform will do. Since I did not create any resources, it will not do anything. But it is good to run terraform plan before terraform apply.

Next, I can run terraform apply:

$ terraform apply

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Enter fullscreen mode Exit fullscreen mode

Well, this is expected. I did not create any resources. If there are any changes, Terraform will show you what it will do and ask you to confirm them. If you want to apply the changes, type yes
and hit enter.

Setting Up The Bucket

From the terraform init output, I can see that the backend is configured. It is configured to use the local backend. My goal is to use the gcs backend. That means I want to store the Terraform state file in the GCP bucket. Let's look at how to do it:

data "google_storage_project_service_account" "gcs_account" {
}

resource "google_kms_key_ring" "tf_states" {
  name     = "terraform-state-key-ring"
  location = "us"

  lifecycle {
    prevent_destroy = true
  }
}

resource "google_kms_crypto_key" "tf_states" {
  name            = "terraform-state-key"
  key_ring        = google_kms_key_ring.tf_states.id
  rotation_period = "100000s"

  lifecycle {
    prevent_destroy = true
  }
}

resource "google_kms_crypto_key_iam_binding" "binding" {
  crypto_key_id = google_kms_crypto_key.tf_states.id
  role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"

  members = ["serviceAccount:${data.google_storage_project_service_account.gcs_account.email_address}"]
}

resource "google_storage_bucket" "tf_states_bucket" {
  name          = "terraform-states-${var.project_id}"
  force_destroy = false
  location      = "us"
  storage_class = "STANDARD"
  versioning {
    enabled = true
  }
  encryption {
    default_kms_key_name = "your-crypto-key-id"
  }

  depends_on = [google_kms_crypto_key_iam_binding.binding]
  lifecycle {
    prevent_destroy = true
  }
}
Enter fullscreen mode Exit fullscreen mode

When working with Terraform, you are calling the API of the provider. With each call, you are pushing configuration to the provider. The provider will then create, update, or delete your resources.

That is a lot of code. Let me explain.

First, I'm creating a Key(google_kms_crypto_key) and Key Ring(google_kms_key_ring) for encrypting the Terraform state file. These resources are coming from Google KMS. The service allows you to work with cryptographic keys and execute cryptographic operations.

Next is adding the Role roles/cloudkms.cryptoKeyEncrypterDecrypter to the Service Account. Which is needed to encrypt/decrypt the state file. In the end, I'm creating a bucket for storing the Terraform state file, and I'm using the Key for encrypting the Terraform state file. There is a small gotcha. I need a Service Account for encrypting the Terraform state file. But it does not exist
yet. So, I'll use an automatic Google Cloud Storage service account to get it when the resource is created. I'm using a data resource for that.

I'm using lifecycle.prevent_destroy to ensure the resources are not accidentally deleted.

If I now run the terraform plan I'll see the following:

$ terraform plan
data.google_storage_project_service_account.gcs_account: Reading...
data.google_storage_project_service_account.gcs_account: Read complete after 1s [id=service-964026605440@gs-project-accounts.iam.gserviceaccount.com]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_kms_crypto_key.tf_states will be created
  + resource "google_kms_crypto_key" "tf_states" {
      + destroy_scheduled_duration = (known after apply)
      + id                         = (known after apply)
      + import_only                = (known after apply)
      + key_ring                   = (known after apply)
      + name                       = "terraform-state-key"
      + purpose                    = "ENCRYPT_DECRYPT"
      + rotation_period            = "100000s"
    }

  # google_kms_crypto_key_iam_binding.binding will be created
  + resource "google_kms_crypto_key_iam_binding" "binding" {
      + crypto_key_id = (known after apply)
      + etag          = (known after apply)
      + id            = (known after apply)
      + members       = [
          + "serviceAccount:service-964026605440@gs-project-accounts.iam.gserviceaccount.com",
        ]
      + role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
    }

  # google_kms_key_ring.tf_states will be created
  + resource "google_kms_key_ring" "tf_states" {
      + id       = (known after apply)
      + location = "us"
      + name     = "terraform-state-key-ring"
      + project  = (known after apply)
    }

  # google_storage_bucket.tf_states_bucket will be created
  + resource "google_storage_bucket" "tf_states_bucket" {
      + force_destroy               = false
      + id                          = (known after apply)
      + labels                      = (known after apply)
      + location                    = "US"
      + name                        = "terraform-states-project-id"
      + project                     = (known after apply)
      + public_access_prevention    = (known after apply)
      + self_link                   = (known after apply)
      + storage_class               = "STANDARD"
      + uniform_bucket_level_access = (known after apply)
      + url                         = (known after apply)

      + encryption {
          + default_kms_key_name = "your-crypto-key-id"
        }

      + versioning {
          + enabled = true
        }
    }

Plan: 4 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
Enter fullscreen mode Exit fullscreen mode

Ok, let's try to apply the changes:

$ terraform apply

...

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_kms_key_ring.tf_states: Creating...

Error: Error creating KeyRing: googleapi: Error 403: Google Cloud KMS API has not been used in project 3534534535 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=3534534535 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

  with google_kms_key_ring.tf_states,
  on main.tf line 4, in resource "google_kms_key_ring" "tf_states":
   4: resource "google_kms_key_ring" "tf_states" {
Enter fullscreen mode Exit fullscreen mode

I had to cut output. But you can see the error. I need to enable the Cloud KMS API. This will be a standard message if you try to use the API, which is not enabled. You can enable it by following the link in the error message. Or you can enable it with the following command:

$ gcloud services enable cloudkms.googleapis.com
Enter fullscreen mode Exit fullscreen mode

gcloud CLI useful commands
You can see all the APIs you enabled with the following command:
$ gcloud services list --enabled

And list all available APIs with the following command:

$ gcloud services list --available

I already have enabled Google Storage API. But if you did not, you need to enable it, too. Try to figure out how. Ok, let's try again:

$ terraform apply
...

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_kms_key_ring.tf_states: Creating...
╷
│ Error: Error creating KeyRing: googleapi: Error 409: KeyRing projects/project-id/locations/us/keyRings/terraform-state-key-ring already exists.
│
│   with google_kms_key_ring.tf_states,
│   on main.tf line 4, in resource "google_kms_key_ring" "tf_states":
│    4: resource "google_kms_key_ring" "tf_states" {
Enter fullscreen mode Exit fullscreen mode

I was impatient and ran terraform apply again. Google API responded that it was not enabled, but actually, it was. And not only that, it created the Key and KeyRing. This situation is rare. But it can happen. We can fix it by importing created resources. Let's do this:

$ terraform import google_kms_key_ring.tf_states project-id/us/terraform-state-key-ring
Enter fullscreen mode Exit fullscreen mode

How do I know that? Well, I read the error message and the provider's documentation. You'll have a section Import in Terraform documentation for each resource.
There, you'll be able to see how to import it.

Now, let's try again:

$ terraform apply

...
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

data.google_storage_project_service_account.gcs_account: Reading...
google_kms_key_ring.tf_states: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring]
data.google_storage_project_service_account.gcs_account: Read complete after 0s [id=service-964026605440@gs-project-accounts.iam.gserviceaccount.com]
google_kms_crypto_key.tf_states_key: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring/cryptoKeys/terraform-states-key]
google_kms_crypto_key_iam_binding.binding: Refreshing state... [id=projects/project-id/locations/us/keyRings/terraform-state-key-ring/cryptoKeys/terraform-states-key/roles/cloudkms.cryptoKeyEncrypterDecrypter]
google_storage_bucket.tf_states_bucket: Creating...
google_storage_bucket.tf_states_bucket: Creation complete after 2s [id=terraform-states-project-id]
Enter fullscreen mode Exit fullscreen mode

The bucket is created. You'll see it if you go to GCP console. You can check bucket properties and configuration there in the console or with the following command:

$ gsutil ls -L -b gs://terraform-states-project-id
Enter fullscreen mode Exit fullscreen mode

You can see that the bucket is encrypted with the Key we created. And to check the Key properties and configuration in the console or with the following command:

$ gcloud kms keys describe terraform-states-key --location us --keyring terraform-state-key-ring
Enter fullscreen mode Exit fullscreen mode

You can list the content of the bucket with the command:

$ gsutil ls gs://terraform-states-project-id
Enter fullscreen mode Exit fullscreen mode

But you'll get nothing because it is empty. I still need to move Terraform's state file to the bucket.

Setting Up The Backend

If you remember, I'm using the local backend. If you check the local directory, you'll see a file called local-state.tfstate. That is the Terraform state file. And that file I want to move to
the bucket and encrypt it there. I can do that with the following code:

  backend "gcs" {
    bucket  = "terraform-states-project-id"
    prefix  = "terraform/state"
  }
Enter fullscreen mode Exit fullscreen mode

And removing this one in provider.tf:

  backend "local" {
    path = "local-state.tfstate"
  }
Enter fullscreen mode Exit fullscreen mode

Now, if I run terraform init, I'll see the following:

terraform init

Initializing the backend...
╷
│ Error: Backend configuration changed
│
│ A change in the backend configuration has been detected, which may require migrating existing state.
│
│ If you wish to attempt automatic migration of the state, use "terraform init -migrate-state".
│ If you wish to store the current configuration with no changes to the state, use "terraform init -reconfigure".
╵
Enter fullscreen mode Exit fullscreen mode

Well, let's do as it says:

$ terraform init -migrate-state
Enter fullscreen mode Exit fullscreen mode

When asked, type yes and wait to finish. Let's check if we have anything in the bucket:

$ gsutil ls gs://terraform-states-project-id/terraform/state
Enter fullscreen mode Exit fullscreen mode

You'll see that the bucket has a file called default.state. That is the Terraform state file. It is encrypted. You can check it with the following command:

$ gsutil cat gs://terraform-states-project-id/terraform/state/default.tfstate | jq .
Enter fullscreen mode Exit fullscreen mode

DevCube | Robert Nemet | Substack

Weekly rant about software design, devops, kubernetes, sre... Click to read DevCube, by Robert Nemet, a Substack publication. Launched 6 months ago.

favicon rnemet.substack.com

Conclusion

With I just set up a project. With it, I can start exploring GCP. We touched basic Terraform CLI commands(init, plan, apply, and import), Terraform variables, provider, backend, and resources. I hope you learned something new. I know I did. I'll continue exploring GCP with Terraform in the next article.

References

💖 💪 🙅 🚩
madmaxx
Robert Nemet

Posted on August 19, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Exploring GCP With Terraform: Setting Up The Environment And Project
terraform Exploring GCP With Terraform: Setting Up The Environment And Project

August 19, 2023