Fixing CloudFormation SSM parameter drift

Edit: I've created a small shell script that downloads the original template and creates a new local template file that you can upload directly to the CloudFormation console.

Introduction

AWS CloudFormation is a tool that helps provisioning infrastructure and deploying resources to Amazon Web Services. It's common to use AWS SAM (Serverless Application Model), a tool that builds upon CloudFormation to ease development of serverless applications and describe their resources.

SAM uses a template file, encoded in either YAML or JSON. It contains the definitions of the Lambda functions, what triggers them, DynamoDB tables, API gateways, and any other AWS resource needed by the application stack. When deploying the app, SAM compares what is already present in the cloud with the content of the template: if there are any differences, it creates a changeset and apply the required changes to the resources.

SSM Parameter drift

Introduction to SSM Parameters

AWS Systems Manager provides features to visualize and manage applications and other AWS resources. One of these features is the Parameter Store: a way to define custom application parameters that can be referenced from the SAM template (and then consumed by the Lambda functions as operating system environment variables).

It's easy to define and modify SSM parameters. When accessing the AWS Console, open Systems Manager and then Parameter Store under Application Manager.

From this page you can add, modify, and delete parameters freely.

After defining the parameters, one can refer to them in the SAM template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  ChangeClientPassword:
    Type: AWS::Serverless::Function 
    Properties:
      Handler: src/ChangeClientPassword.handler
      Environment:
        Variables:
          SENDGRID_API_KEY: '{{resolve:ssm:/app2/sendgrid/api-key}}'
          SENDGRID_API_SECRET: '{{resolve:ssm:/app2/sendgrid/api-secret}}'
      ...

During stack deployment, SAM gets the current SSM parameter values and injects them in each Lambda that uses them.

How to break the stack

There's a very innocent operation that can "break" an application stack: deleting and recreating a SSM parameter that is being used by a deployed application.

If you do that and try to sam deploy again, it will fail to create the changeset:

Error: Failed to create changeset for the stack: 
app2, ex: Waiter ChangeSetCreateComplete failed: 
Waiter encountered a terminal failure state: For 
expression "Status" we matched expected path: 
"FAILED" Status: FAILED. Reason: Parameters: 
[ssm:/app2/sendgrid/api-key:1:1638185826080] 
last modified date does not match with the last 
modified date of the retrieved parameters.

This happens because, after a successful deployment, CloudFormation saves the processed template so it can be compared to new templates by tools like SAM. The processed template is just like the original one, but it has some external references replaced. One of these references are SSM parameters.

Remember how was the original reference to the /app2/sendgrid/api-key SSM parameter?

SENDGRID_API_KEY: '{{resolve:ssm:/app2/sendgrid/api-key}}'

All SSM parameters are versioned and the last modification time is managed too. So, after deploying, CloudFormation stores this in the processed template:

SENDGRID_API_KEY: '{{resolve:ssm:/app2/sendgrid/api-key:1:1638185826080}}'

To create the changeset, SAM gets the version and timestamp of the parameter as saved in the processed template and searches for it in the parameter's change history. If it can't find it there, the changeset cannot be created.

Solution

1. Get the processed template

To workaround this, you'll have to update the timestamps of the affected SSM parameters in the template.

First, open the stack in CloudFormation console and view the Template tab. Click View in Designer.

Wait until the Designer loads fully and shows the template in JSON format in the bottom of the page. Copy the processed template entirely and save it to a file (you'll have to edit and upload it later).

2. Get the new parameter timestamp

Get the new timestamp for the parameter using AWS CLI, as you'll need a timestamp with milliseconds.

$ aws ssm get-parameter --name '/app2/sendgrid/api-key'
{
    "Parameter": {
        "Name": "/app2/sendgrid/api-key",
        "Type": "String",
        "Value": "****",
        "Version": 2,
        "LastModifiedDate": "2022-08-19T10:21:25.249000-03:00",
        "ARN": "arn:aws:ssm:us-east-1:523681489511:parameter/app2/sendgrid/api-key",
        "DataType": "text"
    }
}

The timestamp is LastModifiedDate. You'll need to convert it to a Unix timestamp with milliseconds. You can do it by pasting the date in https://time.lol/ and copying the appropriate formatted value.

3. Fix the timestamp in the template

Open the template and replace, in the affected parameter references, any outdated timestamps with the new one. Remember to also correct the version if needed.

 "Environment": {
   "Variables": {
-      "SENDGRID_API_KEY": "{{resolve:ssm:/app2/sendgrid/api-key:1:1638185826080}}"
+      "SENDGRID_API_KEY": "{{resolve:ssm:/app2/sendgrid/api-key:2:1660915285249}}"
   }
 }

4. Update the stack

Back to the CloudFormation console, close the Designer without saving any changes.

In your stack, click Update, choose "Replace current template", and "Upload a template file". Pick the new template file you've just corrected.

Finish the wizard by clicking Next until the last step, and finally click Update stack.

The stack will be updated (even though the changeset is empty).

You can check that now the processed template contains the latest version and timestamp of the parameter. The next time you deploy the application with SAM, everything should work fine.

Blog