The Joke: Testing in Production

"Testing in production" has become a common joke in the development world - mostly because so many of us have been there and it kind of stings to think about it. You may have heard about these development pipelines, but after looking into them, thought it was too big an investment and the reality is that they're not cheap! Continuous delivery is not something you should try and spit out overnight.

Everyone wants to have confidence that when they deploy code to production and to know that their code is ready. Before having a pipeline of some sort, every release is like that scene from Jurassic Park, when Samuel L Jackson starts bringing power back on.

So how do we get to a point where we're confident our code is ready for production? Pipelines!

What is a pipeline?

Let's start with some definitions for those newer to this idea.

• Pipeline - (from Gitlab https://docs.gitlab.com/ee/ci/pipelines.html)

  A pipeline is a group of jobs that get executed in stages. All of the jobs in a stage are executed in parallel (if there are enough concurrent Runners), and if they all succeed, the pipeline moves on to the next stage. If one of the jobs fails, the next stage is not (usually) executed. You can access the pipelines page in your project’s Pipelines tab.

• Environment - somewhere where your code runs for a some specific purpose

• Staging - a set of computing infrastructure that runs your code for the purposes of integration testing, user experience testing, penetration testing, and other testing before deploying to your customers

• Production - the set of computing infrastructure that runs the code your customers see and where customer data is stored

In other words, once we've put some code together, we first want to deploy it to some kind of staging environment that looks like production, test some stuff out, and then go to production. Your staging environment ideally looks exactly like production, but if you're cash strapped a local VM, Docker container, or their equivalent in the cloud will work just fine. The point is that you test your build to make sure it actually runs before you get to production.

Pipelines can unfortunately get more complex than that. Here are some components that may or may not be in your ultimate pipeline:

• Merge Request
• Commit
• Changelog
• Build Application
• Code Analysis
• Unit Tests
• Integration/API Tests
• Performance Tests
• Multiple environments (Staging (often broken down into multiple environments), Production)
• Smoke Tests
• Regression Tests
• Visual regression Tests
• Release Candidate
• Versioning
• Promoted to Production
• Dark launch
• Percentage-based deployments (day 1 to 10% of customers)
• Automated deploy
• Metrics
• Rollback

We could easily keep keep adding to that list, but it might already seem excessive if not impossible to organizations with limited resources or those unfamiliar with pipelines. Honestly a lot of these components will depend on what your organization needs and you may not need everything there.

The Gitlab link I mentioned earlier keeps it pretty simple https://docs.gitlab.com/ee/ci/pipelines.html, but it also has a feature set that supports building out your pipelines so if you don't use Gitlab you'll have to use other software. Here's a pretty tech agnostic diagram that they use on that page that shows what your pipeline could look like at a high level

So Where Do You Start?

Where you start may depend on resources available. If you are cash strapped or running a project solo, you can still have a pipeline! If you have the resources to start building out a pipeline, but are only just now getting started, no problem because I would recommend starting in the same place.

1. Create a script that can build your software
2. Create a script that can deploy your software once it's built. This can also be reused to deploy to a staging environment if you have one available.
3. Create a script that can roll back to your previous release

You are already doing these 3 things anyway. You might as well automate those steps with scripts. You can always replace these with something more sophisticated later.

Next Steps

Once you have those 3 pieces you can start introducing other components with minimal investment.

• Ensure code reviews are part of your process if you're not running a solo project.
• Start adding unit tests every time you touch the code. When you run your build, add a few lines to execute your unit tests. Toss in a linter while you're at it.
• Add a staging environment of some sort. With a bit of scripting you can do something phoenixable (quickly create/destroy) in AWS or even in a local VM (run a local container, or maybe check out vagrant) if you want. You have options.
• Use something like git tags/docker labels to version your code.
• Maintain a changelog

There are free tools or modules you can download to help you with most of these steps. Pretty much every language has testing libraries and ways to measure code coverage. You can also easily find linters.

Why bother?

I have been on more calls at 2AM than I care to think about - sometimes consecutive nights! It's a dark place to be in (no pun intended) when you are debugging production from 2-6AM.

An investment in a good pipeline will save you a lot of pain and heartache and ultimately give you confidence in you and your organization's ability to release quality software. Even better, your customers will be more satisfied.

Make Progress

There's a lot you can do improve your development pipelines. You don't have to do it all. Start slow, start easy. You don't even need to follow my suggestions, start with what's right for you/your team. Identify quick wins. There are a lot of tools out there to help you in this process.

Eventually you can get to a point where most things are automated. Some of my favorite parts of an automated pipeline is after a deploy to production, you have monitors that watch your metrics, determine that your new release is sending page times through the roof, and triggers an automatic rollback or at least a notification that you may need to rollback. Then I can wake up at 2AM push the button to roll back and go back to sleep!

Here are the some key principles

• Leave your code/pipeline a little better every time you touch it, whether it's an additional unit test or automating some piece.
• Look for the quick wins. Some of them may save you heaps of time.
• If you have a long release cycle, a good pipeline can help you ensure quality leading up to a release.
• Focus on quality. That new feature doesn't help anyone if it errors out all the time.
• Don't rush to production if you can help it.
• Don't test in production!
• Sleep better

I love a good quality pipeline. What have you seen work well? What would you add? What would you like to see me talk more about? I'm happy to go into more details if folks are interested.