How to sync CloudFormation templates from CodeCommit to S3
Todor Todorov
Posted on June 21, 2022
If it happens to you to work on a project where you have selected for one or another reason CloudFormation, you most probably have fallen into a situation to ask yourself the same question as me "OK! But how could I reduce the manual effort and risk from human error while maintaining my CloudFormation Templates between CodeCommit and S3?".
You can easily replace CodeCommit with any other SCM platform, however, this topic will be still valid.
CloudFormation is the managed service offered by AWS for managing your infrastructure as a code. Whether you prefer JSON or YAML files you will always need a location to centrally store your templates.
Use case
Keep track of the changes made to the files when in the meantime you do not need to sync templates to the S3 bucket artifact location manually which will increase the risk of human error during the manual synchronization.
Question
How can we automate this and always have near-real-time synchronization between our repo and S3 bucket from which we point URLs inside the CloudFormation stacks?
Solution
As with every topic we need to cover, the first thing to do is what?
Diagram of course ...
Here is a sample of a CloudFormation template that we can utilize to achieve the goal.
A prerequisite is to have a CodeCommit repository created in advance.
AWSTemplateFormatVersion: 2010-09-09
Parameters:
pCodeCommitRepo:
Description: This parameter serves to point which is our repository to use as a single point of truth
pSolutionName:
Description: Name which we can use to spin up more than one Pipeline for different repositories
pBucketName:
Type: String
Default: my-cf-templates
pMyAWSKeyAccountId:
Type: String
Default: 123456789012
pMyAWSKeyRegion:
Type: String
Default: eu-west-1
pMyAWSKeyArn:
Type: String
Default: arn:aws:kms:eu-west-1:123456789012:key/abcdefgh-ijkl-1234-5678-zxcvbnmasdfg
Resources:
rS3Bucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
Properties:
BucketName: !Ref pBucketName
VersioningConfiguration:
Status: Enabled
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: 'aws:kms'
KMSMasterKeyID: !GetAtt rKMSS3Encryption.Arn
# What we can do here, if we have multiple AWS accounts where we are going to use the CloudFormation Templates is to replicate
# the content in a dedicated S3 bucket inside every AWS account where we need.
ReplicationConfiguration:
Role: !GetAtt rS3ReplicationRole.Arn
Rules:
- Destination:
Bucket: !Sub "arn:aws:s3:::my-cf-templates-${pMyAWSKeyAccountId}.${pMyAWSKeyRegion}"
AccessControlTranslation:
Owner: Destination
Account: !Ref pMyAWSKeyAccountId
EncryptionConfiguration:
ReplicaKmsKeyID: !Ref pMyAWSKeyArn
DeleteMarkerReplication:
Status: Enabled
Priority: 1
Filter:
Prefix: ''
Status: Enabled
SourceSelectionCriteria:
SseKmsEncryptedObjects:
Status: Enabled
rKMSS3Encryption:
Type: AWS::KMS::Key
DeletionPolicy: Retain
Properties:
Enabled: true
KeyPolicy:
Version: '2012-10-17'
Id: key-default-1
Statement:
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS:
Fn::Join:
- ''
- - 'arn:aws:iam::'
- Ref: AWS::AccountId
- :root
Action: kms:*
Resource: '*'
- Sid: Allow use of the CMK
Effect: Allow
Principal:
AWS:
- !Sub arn:aws:iam::${AWS::AccountId}:role/service-role/AWSCodePipelineServiceRole-*
Action:
- kms:Encrypt
- kms:Decrypt
- kms:ReEncrypt*
- kms:GenerateDataKey*
- kms:DescribeKey
Resource: "*"
EnableKeyRotation: true
rKMSS3EncryptionAlias:
Type: AWS::KMS::Alias
DeletionPolicy: Retain
Properties:
AliasName: alias/myS3BucketKey
TargetKeyId: !Ref rKMSS3Encryption
rS3ReplicationPolicy:
Type: AWS::IAM::ManagedPolicy
Properties:
ManagedPolicyName: !Sub myReplicationPolicy-${pBucketName}-${AWS::Region}
Description: S3 replication policy
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Action:
- 's3:GetReplicationConfiguration'
- 's3:ListBucket'
Resource: !Sub arn:aws:s3:::${pBucketName}
-
Effect: Allow
Action:
- 's3:GetObject*'
Resource: !Sub arn:aws:s3:::${pBucketName}/*
-
Effect: Allow
Action:
- 's3:ReplicateObject'
- 's3:ReplicateDelete'
- 's3:ReplicateTags'
- 's3:ObjectOwnerOverrideToBucketOwner'
Resource: !Sub arn:aws:s3:::my-cf-templates-${pMyAWSKeyAccountId}.${pMyAWSKeyRegion}/*
-
Effect: Allow
Action:
- 'kms:*'
Resource: !GetAtt rKMSS3Encryption.Arn
-
Effect: Allow
Action:
- 'kms:*'
Resource: !Ref pMyAWSKeyArn
rS3ReplicationRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ReplicationRole-${pBucketName}-${AWS::Region}
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: 'Allow'
Principal:
Service:
- 's3.amazonaws.com'
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- !Ref rS3ReplicationPolicy
rCodePipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
Name: !Sub "${pSolutionName}-pipeline"
RestartExecutionOnUpdate: false
RoleArn: !GetAtt rCodePipelineServiceRole.Arn
Stages:
- Name: Source
Actions:
- Name: SourceAction
ActionTypeId:
Category: Source
Owner: AWS
Provider: CodeCommit
Version: 1
Configuration:
RepositoryName: !Ref pCodeCommitRepo
BranchName: master
RunOrder: 1
- Name: SyncToS3
Actions:
- Name: SyncToS3
ActionTypeId:
Category: Deploy
Owner: AWS
Version: 1
Provider: CodeDeploy
Configuration:
Extract: true
BucketName: !Ref rS3Bucket
ObjectKey: "/"
RunOrder: 1
rCodePipelineServiceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action: 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: !Sub "AWS-CodePipeline-Service-${pCodeCommitRepo}"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'codecommit:CancelUploadArchive'
- 'codecommit:GetBranch'
- 'codecommit:GetCommit'
- 'codecommit:GetUploadArchiveStatus'
- 'codecommit:UploadArchive'
Resource: '*'
- Effect: Allow
Action:
- 'codedeploy:CreateDeployment'
- 'codedeploy:GetApplicationRevision'
- 'codedeploy:GetDeployment'
- 'codedeploy:GetDeploymentConfig'
- 'codedeploy:RegisterApplicationRevision'
Resource: '*'
- Effect: Allow
Action:
- 'codebuild:BatchGetBuilds'
- 'codebuild:StartBuild'
Resource: '*'
- Effect: Allow
Action:
- 'devicefarm:ListProjects'
- 'devicefarm:ListDevicePools'
- 'devicefarm:GetRun'
- 'devicefarm:GetUpload'
- 'devicefarm:CreateUpload'
- 'devicefarm:ScheduleRun'
Resource: '*'
- Effect: Allow
Action:
- 'lambda:InvokeFunction'
- 'lambda:ListFunctions'
Resource: '*'
- Effect: Allow
Action:
- 'iam:PassRole'
Resource: '*'
- Effect: Allow
Action:
- 'elasticbeanstalk:*'
- 'ec2:*'
- 'elasticloadbalancing:*'
- 'autoscaling:*'
- 'cloudwatch:*'
- 's3:*'
- 'sns:*'
- 'cloudformation:*'
- 'rds:*'
- 'sqs:*'
- 'ecs:*'
Resource: '*'
rAmazonCloudWatchEventRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- events.amazonaws.com
Action: 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: !Sub "cw-event-pipeline-execution-${pCodeCommitRepo}"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: 'codepipeline:StartPipelineExecution'
Resource: !Join
- ''
- - 'arn:aws:codepipeline:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- ':'
- !Ref rCodePipeline
rAmazonCloudWatchEventRule:
Type: AWS::Events::Rule
Properties:
EventPattern:
source:
- aws.codecommit
detail-type:
- CodeCommit Repository State Change
resources:
- !Join
- ''
- - 'arn:aws:codecommit:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- ':'
- !Ref pCodeCommitRepo
detail:
event:
- referenceCreated
- referenceUpdated
referenceType:
- branch
referenceName:
- master
Targets:
- Arn: !Join
- ''
- - 'arn:aws:codepipeline:'
- !Ref 'AWS::Region'
- ':'
- !Ref 'AWS::AccountId'
- ':'
- !Ref rCodePipeline
RoleArn: !GetAtt rAmazonCloudWatchEventRole.Arn
Id: codepipeline
In a conclusion, I could say that our target should always be to remove any repetitive effort through automation for the sake of reducing the risk of making errors and at the same time focus on more important and interesting tasks.
💖 💪 🙅 🚩
Todor Todorov
Posted on June 21, 2022
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.