TheLazy.Dev
Portfolio GitHub

Blog

cfnguard |testing |aws

This is how you can test your cfn-guard rules

nr18

Joris Conijn

Posted on December 16, 2022

This is how you can test your cfn-guard rules

In my previous blog, How do you prove that your infrastructure is compliant.

I explained how you can prove your infrastructure is compliant using CloudFormation Guard.
But, how do you write those rules? And even more important, how do you test your rules? If you look at the repository CloudFormation Guard.

You will notice that the project itself offers a testing framework.
Alright! Let’s build a ruleset and write some tests for it!

Let’s build a ruleset for S3

We will start with a sample for S3, create a file called: rules/s3.guard.

let buckets = Resources.*[ Type == 'AWS::S3::Bucket' ]

rule BucketEncryption when %buckets !empty {
  %buckets.Properties {
    BucketEncryption.ServerSideEncryptionConfiguration[*] {
      ServerSideEncryptionByDefault.SSEAlgorithm IN ["AES-256", "aws:kms"] <<Ensure all S3 buckets use encryption-at-rest>>
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Now we will write the test, called: rules/s3_tests.yaml.

---
- name: "Skip if not present"
  input:
     Resources: {}
  expectations:
    rules:
      BucketEncryption: SKIP
- name: "No encryption used"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
  expectations:
    rules:
      BucketEncryption: FAIL

- name: "Bucket with KMS Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: aws:kms
  expectations:
    rules:
      BucketEncryption: PASS

- name: "Bucket with AES-256 Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: AES-256
  expectations:
    rules:
      BucketEncryption: PASS

- name: "Bucket with UNKNOWN Encryption"
  input:
     Resources:
       MyBucket:
         Type: AWS::S3::Bucket
         Properties:
           BucketEncryption:
             ServerSideEncryptionConfiguration:
               - ServerSideEncryptionByDefault:
                   SSEAlgorithm: UNKNOWN
  expectations:
    rules:
      BucketEncryption: FAIL
Enter fullscreen mode Exit fullscreen mode

Now it’s time to run the tests:

cfn-guard test --rules-file rules/s3.guard --test-data rules/s3_tests.yaml
Enter fullscreen mode Exit fullscreen mode

This will give the following output:

Test Case #1
Name: "Skip if not present"
  PASS Rules:
    BucketEncryption: Expected = SKIP, Evaluated = SKIP

Test Case #2
Name: "No encryption used"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

Test Case #3
Name: "Bucket with KMS Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #4
Name: "Bucket with AES-256 Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #5
Name: "Bucket with UNKNOWN Encryption"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL
Enter fullscreen mode Exit fullscreen mode

You might be thinking, that looks nice! But the output is quite verbose, and you need to execute a command per rule/test set.

It gets even worse when a test does fail. For this example I altered the rules/s3.guard file to use aws:kmz instead of aws:kms.

Then it looks like this:

Test Case #1
Name: "Skip if not present"
  PASS Rules:
    BucketEncryption: Expected = SKIP, Evaluated = SKIP

Test Case #2
Name: "No encryption used"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL

Test Case #3
Name: "Bucket with KMS Encryption"
  FAILED Rules:
    BucketEncryption: Expected = PASS, Evaluated = FAIL

Test Case #4
Name: "Bucket with AES-256 Encryption"
  PASS Rules:
    BucketEncryption: Expected = PASS, Evaluated = PASS

Test Case #5
Name: "Bucket with UNKNOWN Encryption"
  PASS Rules:
    BucketEncryption: Expected = FAIL, Evaluated = FAIL
Enter fullscreen mode Exit fullscreen mode

If you look at Test Case #3 it says: BucketEncryption: Expected = PASS, Evaluated = FAIL. That makes it hard to spot what test failed.

cfn-guard-test

For this reason I created a python package called cfn-guard-test. This package

makes it easier to run many rule/test sets. This is especially useful in the CI/CD pipelines.

Please read the repository for the installation instructions.

Let’s run the tests again!

cfn-guard-test --verbose
Enter fullscreen mode Exit fullscreen mode

The output now looks something like:

===== Analyzing Results =====

Running rules/s3_tests.yaml

Passed 5
Failed 0
Enter fullscreen mode Exit fullscreen mode

That is a nice and clean overview. It lists the number of passed and failed tests. For the next example I altered the

rules/s3.guard file to use aws:kmz instead of aws:kms. This will again trigger a failure:

===== Analyzing Results =====

Running rules/s3_tests.yaml

Passed 4
Failed 1

Rule BucketEncryption failed on #3 "Bucket with KMS Encryption" in rules/s3.guard
Enter fullscreen mode Exit fullscreen mode

Well you have to agree. That immediately sums up how many passed and failed. And more it shows that the BucketEncryption

rule fails. And it failed on the test case "Bucket with KMS Encryption".
You don’t want to spend time figuring out what is wrong. This is why we use tooling, to make our life easier.
This tool also has the ability to generate JUnit report.

cfn-guard-test --verbose \
  --junit-path "reports/cfn-guard.xml"
Enter fullscreen mode Exit fullscreen mode

Conclusion

cfn-guard-test can help you run many tests. And get an aggregated report of

the results. This tool will help you detect bugs in your rules easier and quicker.

💖 💪 🙅 🚩
nr18
Joris Conijn

Posted on December 16, 2022

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

This is how you can test your cfn-guard rules
cfnguard This is how you can test your cfn-guard rules

December 16, 2022