Andrew May
Posted on October 29, 2020
I recently started building stackmanager, an Open Source project for managing CloudFormation stacks.
There are already a number of other tools for doing this, and I've used or tried several of them, but none quite fit how I wanted them to work.
Functional Requirements
Managing all the configuration used to create/update a CloudFormation stack in a single file
The CloudFormation create-stack command has lots of separate values you need supply, and with the CLI it's a pain to supply Stack Name, Template File, Parameters, Targets, Capabilities, etc.
Support for deploying the same stack to different environments (e.g. dev, prod, or different regions) while reducing duplication of configuration
Almost all the CloudFormation I write is used in multiple different environments (often different accounts), and sometimes is deployed to multiple regions for failover.
Uses ChangeSets, with control over whether to immediately apply or apply later
ChangeSets allow you to preview (to a limited degree) the changes you're making, and especially if you're using the AWS Serverless Application Model transformation, or a CloudFormation macro, you don't necessarily know exactly what resources are going to be created until the CloudFormation service process the template.
In some CI/CD workflows, there may be an approval requirement before changes go into production, and the ChangeSet allows you to preview changes.
Log progress
Print a summary of the ChangeSet and show the CloudFormation events whether the stack update succeeds or fails. I really like how SAM shows this information.
Support Lambda Functions
This is somewhat outside the core functionality, but as I started writing this while working on a project that deployed Lambda functions using CloudFormation (where the S3 Key is changed when there is new code, triggering an update), I wanted to be able to use a single tool to build, upload and deploy Lambda functions, similar to the SAM CLI, but better suited to more general CloudFormation management and CI/CD.
Non-functional requirements
This is not the first utility of this sort I've written, but previous versions have belonged to the client who I wrote them for. I wanted to create an Open Source utility that I could use at multiple clients and for Leading EDJE infrastructure.
Writing stackmanager from scratch allowed me to improve on what I'd written previously, and include things I'd previously neglected like unit tests.
Even if I end up the only one contributing to the code base I also wanted to run it like a regular open source project with a list of issues, pull requests and releases when I reached a suitable milestone.
Getting to 1.0
A lot of Open Source projects never reach a 1.0 release, but I just wanted to release something relatively complete and well tested and not necessarily include everything that stackmanager could possibly do. The hardest thing to finish in the list of issues I'd included in the 1.0 milestone was to get acceptable code coverage, as my experience writing Python unit tests is somewhat limited, especially when it comes to mocking.
Using stackmanager
I don't really want to rewrite the README here, but stackmanager allows you to do things like:
$ stackmanager --profile dev --region us-east-1 \
build-lambda --source-dir integration/functions/python/hello_world/ --output-dir /c/dev/temp/ --runtime python3.7 \
upload --bucket stackmanager-lambda-files-us-east-1 --key python/hello_world.zip \
deploy --environment dev --config-file integration/functions/python/config.yaml --auto-apply
Building python3.7 Lambda function from integration/functions/python/hello_world/
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
Built Lambda Archive C:\dev\temp\hello_world.zip
2020-10-28 20:33:21 Found credentials in shared credentials file: ~/.aws/credentials
Uploaded C:\dev\temp\hello_world.zip to s3://stackmanager-lambda-files-us-east-1/python/hello_world.zip
Stack: d-StackManager-PythonFunction, Status: does not exist
Creating ChangeSet c8a1e3704d0da48f69903bafcaf239c86
Action LogicalResourceId ResourceType Replacement
-------- ------------------- --------------------- -------------
Add FunctionRole AWS::IAM::Role -
Add Function AWS::Lambda::Function -
Executing ChangeSet c8a1e3704d0da48f69903bafcaf239c86 for d-StackManager-PythonFunction
ChangeSet c8a1e3704d0da48f69903bafcaf239c86 for d-StackManager-PythonFunction successfully completed:
Timestamp LogicalResourceId ResourceType ResourceStatus Reason
------------------------- ----------------------------- -------------------------- ------------------ ---------------------------
2020-10-28 20:33:24-04:00 d-StackManager-PythonFunction AWS::CloudFormation::Stack REVIEW_IN_PROGRESS User Initiated
2020-10-28 20:33:35-04:00 d-StackManager-PythonFunction AWS::CloudFormation::Stack CREATE_IN_PROGRESS User Initiated
2020-10-28 20:33:39-04:00 FunctionRole AWS::IAM::Role CREATE_IN_PROGRESS -
2020-10-28 20:33:39-04:00 FunctionRole AWS::IAM::Role CREATE_IN_PROGRESS Resource creation Initiated
2020-10-28 20:33:53-04:00 FunctionRole AWS::IAM::Role CREATE_COMPLETE -
2020-10-28 20:33:56-04:00 Function AWS::Lambda::Function CREATE_IN_PROGRESS -
2020-10-28 20:33:57-04:00 Function AWS::Lambda::Function CREATE_IN_PROGRESS Resource creation Initiated
2020-10-28 20:33:57-04:00 Function AWS::Lambda::Function CREATE_COMPLETE -
2020-10-28 20:33:59-04:00 d-StackManager-PythonFunction AWS::CloudFormation::Stack CREATE_COMPLETE -
This built a Python Lambda function, uploaded the zip file to S3 and then deployed a CloudFormation stack that used the file in S3, previewing the change-set and then running it.
In this case the functionality is similar to the SAM CLI for Lambda functions, but the focus of stackmanager is really the deploy command that is for any CloudFormation stack and not just Lambda functions.
At the heart of stackmanager is the configuration file that can encapsulate the configuration for multiple environments. For the example I ran above this is the configuration file:
---
Environment: all
StackName: "{{ EnvironmentCode }}-StackManager-PythonFunction"
Parameters:
Environment: "{{ Environment }}"
Tags:
Application: StackManager
Environment: "{{ Environment }}"
Template: integration/functions/python/template.yaml
Capabilities:
- CAPABILITY_IAM
---
Environment: dev
Region: us-east-1
Variables:
EnvironmentCode: d
In this simple example the inheritance between the dev environment and the all environment isn't particular useful, but once I have multiple environments and a mix of common and unique parameters this really cuts down on the total amount of configuration (and the number of separate files to edit).
Is stackmanager right for you?
If you're already using CloudFormation, but you have a mess of parameter files and other arguments you're using to run your CloudFormation then I'd like to think this will help you clean up the mess.
Moving forward there's a possibility that CDK will obsolete writing CloudFormation directly, but there's a lot of CloudFormation already out there so it's going to be around for a while.
Posted on October 29, 2020
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.
Related
January 29, 2023
