Terraform is currently the go-to tool for managing infrastructure through version control. There is somewhat of a learning curve but then it is fairly straightforward to provision new infrastructure. Being able to create a dependency graph and provide details about various components involved is a great way for explaining the nuances of an existing infrastructure to new engineers. Being able to express the infrastructure via code also helps with dissemination of information across multiple engineering teams and avoids having to over document things.

One of the challenges that I have come across when working with Google Cloud Platform (GCP) is managing service accounts. Creation of service accounts is straightforward but managing keys is a different matter altogether especially for cases where you use the keys in different services.

When you create a new JSON key for service accounts, you can download the key directly from the UI and you can also manage it via Terraform (TF). If you go with the former approach, you will have to manage the keys yourself especially around who has access. With TF, the keys are re-generated every time you run terraform apply and you would not have access to them to share with services.

resource "google_service_account" "service_account" {
  account_id   = "service-account-id"
  display_name = "Service Account"
}

To deal with this problem of re-generation and to have access, I went with a hybrid approach of using TF to manage service accounts and then manage the keys myself. After the accounts are created, I use the Google IAM section to generate JSON key files for the service accounts that were just generated. These keys are then stored in the same TF state bucket which is private (by default) but at a location that is not mapped in the TF files. The location would be at a path something like /keys/sa/svc-microservice1.json and the hierarchy can be of any classification that makes sense for the team.

A potential classification can take the form of service names and then each folder with have all of the service account keys used by that service. There could also be a separate folder for shared keys. The folder hierarchy does not actually matter as the storage bucket does not have a concept of folder. The path is actually the name of the file.

+/
+--keys
   +--sa
      +--microservice1
         +--svc-db.json
         +--svc-storage.json
      +--microservice2
         +--svc-tasks.json
         +--svc-storage.json
      +--shared
         +--svc-build.json

As the access to the TF state bucket is limited (private) and an automatic audit log is maintained by GCP about who accessed the files, it is relatively safe to maintain the service account key files in the bucket. It also makes it easier for anyone else apart from you to find the keys when needed especially when you are not around.