Managing AWS SSM Parameters With Terraform And Parzival

Modern applications often use external systems to manage configuration values and secrets. One of those external systems is AWS Parameter Store, which is part of the AWS Systems Manager service.

There's a great blog about the top 10 Best Secret Management Software for Application Security that covers this subject in a significant way.

This blog post will demonstrate a practical way of managing secrets/parameters in the AWS SSM Parameter Store with Terraform. Wondering what Parzival is? Keep on reading to find out.

Plot

Managing application values is quite easy for small applications, where only one (1) to three (3) values are fetched from an external system. Let's say my application is small and requires 3 values per live environment (development, staging and production) let's map them and their types:

Name	Type
GOOGLE_CLIENT_ID	Secure String
GOOGLE_CLIENT_SECRET	Secure String
LOG_LEVEL	String

Challenge #1: Live Environments Alignment

I logged in to AWS Console and created the three (3) parameters in AWS Parameter Store per environment (9 total). Now, I need to add another parameter, SLACK_WEBHOOK_URL of String type. Do you realize that each parameter that needs to be added requires doing so per live environment? That's 3x times, and it gets crazier as the application grows.

There's also a chance that I've added a parameter in development and didn't add it in staging, following that I rolled out a deployment of my application to staging and boom 💥 - staging is dead baby, staging's dead.

Luckily it's only staging, and production didn't suffer any downtime, but it still consumes precious engineering time investigating why the application crashed in staging. Also, if the QA engineers are running tests on staging, they're now stuck until I figure out what went wrong. Of course, that applies to all cloud resources, such as AWS EC2, S3 Bucket, etc.

Many sad stories are being told by engineers who maintain their cloud resources for multiple live environments with the AWS Console (GUI), let's not be the ones who tell those stories.

Live Environments Alignment - Solution

Using Infrastructure as Code mitigates the risk of having misaligned live environments. And as mentioned in the title, I'll implement IaC for managing SSM Parameters with Terraform.

Challenge #2: Storing Secrets Safely

For obvious reasons, I don't want to commit sensitive data to my source code (git). But, on the other hand, I do want version control for parameters values.

This part is tricky; how do I create an SSM Parameter with Terraform without committing its value to the source code? Terraform's files are part of the source code, so it's kind of a paradox.

Storing Secrets Safely - Solution

Firstly, I'll create SSM parameters with Terraform; the initial value for each parameter should be a dummy value like "empty". Later on, the Dev team can validate parameter values by checking if it's still "empty", which means humans didn't set it. That makes it easier to track and monitor which parameters weren't set by humans.

Secondly, I'll manage the parameter values with AWS Console (GUI). Another paradox is that I'm about to use AWS Console, but in the previous section, I wrote:

"...Many sad stories are being told by engineers who maintain their cloud resources ... with AWS Console (GUI)..."

The bolded text is the answer; I'm about to manage my values in the GUI, not cloud resources.

Lastly, to address the requirement "...versioning for parameters values..." I'll use the version management feature of SSM Parameters, see Working with SSM Parameters Versioning. I will not cover SSM Parameters Versioning in this blog post.

Okay, sounds great, but I don't see a real solution here; it sounds like a nice theory, but how do I do it in practice?

There's A Terraform Module For That

There are many terraform modules out there that help with managing SSM Parameters, here are some examples:

I needed something "simpler"; A way to provide a list of parameters and their types, a prefix to be added to each parameter, and ... that's it ¯_(ツ)_/¯

Meet unfor19/terraform-aws-ssm-parameters, a Terraform module that I've created. Here's what it does:

"Create AWS SSM Parameter Store parameters with a Terraform module. The creation/deletion (schema) is managed with Terraform, and the values should be maintained via AWS Console."

Getting back to our example, here's how you would create SSM Parameters with my module. To keep it simple, I'm leaving out the previously mentioned parameter SLACK_WEBHOOK_URL.

module "app_params" {
    source  = "unfor19/ssm-parameters/aws"
    version = "0.0.2"

    prefix = "/myapp/dev/"

    string_parameters = [
        "LOG_LEVEL",
    ]
    securestring_parameters = [
        "GOOGLE_CLIENT_ID",
        "GOOGLE_CLIENT_SECRET"
    ]
}

Well? Isn't it incredible? The Terraform module will create all the provided parameters and add prefix to their names. For me, having the parameters schema (name+type) in a single place makes it easier to realize which parameters are available per live environment.

Reminding you that the initial value for each parameter is "empty", which can be changed by setting some of the inputs; check Terraform Registry Inputs Page to see which inputs are available.

Finally, I can log in to AWS Console and manage my SSM Parameter values without worrying that the next terraform apply will change their values. If you wonder how I did it, I used Terraform lifecycle Meta-Argument; To see a practical example of how to use lifecycle with SSM Parameters values, check my module's main.tf.

It's also important to mention that the module is tested against various Terraform versions.

0.12.31
0.13.6
0.14.8
0.15.0-beta2
1.0.6 (latest)

For more details about the test suite, see the CI/CD logs and Terraform Module Local Development.

Recap

I've created three (3) SSM Parameters with Terraform, managing their values through the AWS Console. The values are detached from Terraform's scope, so using terraform apply will not override the changes that I've made to values via AWS Console.

Challenge #3: Managing SSM Parameter Values Programmatically

Setting the values for three (3) parameters via AWS Console per live environment is okayish. However, as the application grows, the number of parameters might grow.

Bear with me on this one; The application grew more prominent and required thirty (30) SSM Parameters. So I've created those thirty (30) parameters in development, and set their values by using AWS Console (GUI). Let's assume that twenty-five (25) out of thirty (30) are identical in development and staging, leaving five (5) unique values per environment.

I wish there were a way to copy the twenty-five (25) parameters' values from development to staging. I hope you realize how painful it can be to do it manually and go back and forth to copy-paste parameter values.

Usually, it's not a pain in development because parameter values are set one at a time and not at once. However, the scenario I'm referring to happens when creating a new live environment, like staging, where all the initial values are "empty".

Another possible scenario for needing a "bulk copy of SSM Parameters" is when I need to create a dynamic environment and set the initial values of the dynamic environment to be the same as those in the development environment.

And again, it sounds nice, but how would you copy SSM Parameters values in bulk? Is there such a way?

Managing SSM Parameter Values Programmatically - Solution

Meet unfor19/parzival. The reason for the name is ridiculous 😁 - I thought about naming it paraval which stands for parameter-values, and it sounded like Parzival which is way cooler, especially if you watched Ready Player One.

CHALLENGE: I urge you to check if there's an existing CLI that can get/set more than 10 SSM Parameters by path in a single command. The official AWS CLI supports pagination, but it means I need to write code that wraps aws ssm get-parameters-by-path .... The only similar project that I found is called binxio/aws-ssm-copy, which didn't have the capabilities that I need.

Moving on with the plot, Parzival is beneficial for the following scenarios:

Copy values from one environment to another - for example, from development to staging, or development to my-dynamic-env-1, and so on. Copying values across accounts is also possible because the get and set commands are separated.
Troubleshooting SSM Parameters values - for example, get all parameters by the path and find the ones that a human didn't initialize (contains the value "empty").

To conclude, Parzival is a simple wrapper for get-parameters-by-path and put-parameter.

Let's see it in action!

Get Parameters By Path

parzival get --region "us-east-1" \
     --output-file-path ".dev_parameters.json" \
     --parameters-path "/myapp/dev/"

Set Parameters Values By Path

parzival set --region "us-east-1" \
     --input-file-path ".dev_parameters.json" \
     --prefix-to-replace "/myapp/dev/" \
     --parameters-path "/myapp/stg/"

To view the output of the above commands, check the CI/CD Logs and look for the Test job.

Final Words

What are you waiting for? Go ahead and play with the Terraform module unfor19/terraform-aws-ssm-parameters, and then manage the values of SSM Parameters with unfor19/parzival.

Both projects take advantage of localstack, which provides a fully functional local cloud environment, like "local AWS". Using localstack as part of the development process helped me a lot to battle-test my application rapidly; for that reason, you don't even need an AWS account to check the Terraform module and Parzival. So follow the instructions in the README.md files and do your magic.

Happy SSMing!