And more specifically do you write Snapshot tests for your Infrastructure as Code ( like AWS CDK stacks)?

The context

Snapshot testing is a type of “output comparison” or “golden master” testing.

Basically, you freeze a specific state/view/characteristic of your application and compare it overtime to make sure it did not change. Snapshot tests do not test the behavior, the fact that with a given input the output is always the same.
They just make sure that the current state/representation is the same as the one you have stored. If it is not, you are asked if you want to overwrite the saved snapshot and replace it with the current result.

Over the years I can count on the fingers of one hand, the times when I actually used Snapshot tests:

• when we were refactoring the basic engine of one of our games and we wanted to be really really sure we weren't changing anything of how that complex UI was looking. Once the refactoring was done, we dropped the snapshot testing because changes to the UI were part of every sprint, so it did not make sense.
• when we just started with React, and we were really excited to take advantage of the Snapshot testing functionality for our React components.
• when we started using AWS CDK to build our Serverless Infrastructure. In this case, the snapshot is not that "Visual", like a picture of the UI, rather a comparison of the JSON output of Cloudformation.

The problem:

As soon as you change something, not even in the stack itself but in the lambda code (since the hash of the uploaded asset will be different), the test would fail. Therefore, you have to rerun the test, forcing/confirming and you have to run the test locally, confirm the update of the snapshot, and push.

The difference with Unit ( or integration tests etc) is that with those, usually, you have to update the tests only if something changes either in the inputs or in the expected outputs.

That is why we have tests in the first place, to make sure that refactorings of the code itself won't affect behavior from the outside.

I understand that if you add a new resource to your API, or change a policy to write to your bucket, you are modifying the architecture of your application, I understand that it requires an additional action from the developer to intentionally edit the snapshots. and I know that the snapshots will be version controlled, so you can quickly compare all the final deployed stacks. But I still don't see much benefit in it.

It could be handy for some small things like (very) small react components returning confirmation or error messages ( instead of using regex to check for the match, match the snapshot)
https://kentcdodds.com/blog/effective-snapshot-testing

But for the entire stack or the entire screenshot of an application, snapshot updates would be required for ANY single change, so it will simply become part of the routine:

• change code
• run test
• get the warning snapshot test failed
• update snapshot
• commit without much thinking. Therefore making the test useless.

The Question

I do not have yet a strong opinion about such tests, and I think it is nice to have them especially for some specific critical happy paths in your UI (like the login form, or shop page or for small CDK Constructs (that maybe you share with other teams), but it really adds a burden of work, and often gives a false sense of security, and when it comes to test the CDK I much prefer to have a smaller test checking for the existence of a specific Resource / Role rather than Snapshot test the entire thing)

Do you use them, how far do you go with them?

How extensively do you snapshot test your application?

How long have you been doing it (in some project I worked on, it was introduced, but then dropped just after a few weeks, for the reasons I mentioned above)

Reference

• Improving Communication and Confidence with Snapshot tests
• Recognize the need for visual testing
• Jest Image Snapshot Docs
• How to test AWS CDK
• Benefits and Drawbacks of Snapshot testing

Photo by britt gaiser on Unsplash