Reduce release risk with Progressive Delivery

Add the feature flag and wrap up the feature code.
Turn off feature flags that no one can see and access new features.
Deploy the new version on production with canary deployment on the container/application level
Testing in production. Turn on the feature flag to QA and beta users.
Release each feature independently and progressively. If needed, roll back the feature in one second without redeployment and git revert.

Introduction

Minimizing delivery risk is essential for software engineering. Testing in Production (TIP) and Progressive Delivery are proven solutions to reduce delivery risk.

Testing in production (TIP) is a software development practice in which new code changes are tested on live user traffic rather than in a staging environment.

Progressive Release helps mitigate the risk of unexpected issues affecting users by making a release available only to a specific percentage of a snap’s user base. This percentage can be initially small, perhaps 10% or 20%, and increases as confidence in a release grows.

A classical solution: Using the Traffic Routing to progressively switch from the current to the new version of the application. Today, many cloud services allow you to achieve Traffic Routing easily. Microservice architecture often uses a service mesh (App Mesh, Istio, Linkerd, Open Service Mesh) or an ingress controller (Contour, Gloo, NGINX, Skipper, Traefik) for traffic routing.

Problem

The traffic routing approach has a significant shortcoming when dealing with real-world problems:

If only one feature encounters a problem, you must roll back all the features.
The rollout and rollback can only be operated by DevOps/Ops team, which takes time.
It's hard to test each feature with a specific target audience.
It's hard to release a feature to a specific customer progressively.
It's hard to deliver experimental features.
It can't handle the new version's business and product impact, which also causes user/income loss.
Etc.

Solution

Feature Flags is a modern engineering technology that decouples code deployments from feature releases, giving you control over who sees each feature and when they see it.

With feature flags, you can progressively release or roll back individual features to or from a specific group of users without redeployment. You can toggle a feature on and off to subsets of users.

Implementation of feature flag

In this chapter, we will use an open-source feature flag tool FeatBit to demonstrate.

1. Build your feature flag environment

You can learn how to build the system through FeatBit's GitHub repo.

2. Connect a feature flag SDK to projects

We need to install the feature flag SDK to your project. Here's a demo of how to install the feature flag SDK in a front-end application:

npm install featbit-js-client-sdk

Then we need to initiate SDK with users' basic information. Here is a screen capture of what it looks like.

Secret, a key to connect to the remote feature flag server.
API URL, the address of the remote feature flag server.
Configurable user information that you will use to roll out features to the target audience.

3. Use the feature flag to wrap up and control the feature code.

In the web app project, find the code which runs the "notification module", and use a feature flag to wrap up the related code. As shown below:

The front-end app is written with VUE (a framework of javascript/typescript). If the flag code featureStore.flags['notification'] execute and returns true, the feature "notification module" runs and is displayed in the front-end APP. The feature won't be executed if it returns false.

4. Turn off the feature flag.

The new feature should be delivered to users only after it's deployed to production. As shown below, we need to turn off the feature flag "notification", and let it return variant false.

Testing in production

We need to configure the portal UI to control who sees the feature and when they see it. (user sees the feature only if the flag returns true).

Turn on the feature flag for the notification module, and let only the QA team and beta users see (access) this feature. Test it in production.

If the user is in the QA group or is one of the beta users for the "notification module"，featureStore.flags['notification'] should return true.
If not, featureStore.flags['notification'] return false

In the Segment page of FeatBit, create a new segment QA team, input the QA members into the "Including users" list in the "Targeting users" tab, then click Review and save。

On the feature flag list page, choose the "Notification" flag and go to its Targeting page.

a. Create a customized "Targeting rule": If the user belongs to the QA segment, then the feature flag featureStore.flags['notification'] returns true; otherwise returns false.

b. You can add beta users directly to the true value list in the Individual targeting tab. If the user belongs to this list, featureStore.flags['notification'] returns true; otherwise returns false.

Complete the setting and save it. Below are demonstrations in the video:

Releasing a feature progressively

It's easy to progressively release the "notification module" feature to users. You can change the default rule to serve: Rollout percentage, then input the percentage of all users who receive a true value. As shown below, we roll out the feature to 10% of users.

Releasing a feature to a specific customer progressively.

In the real world, some SaaS providers release a feature only to one of their biggest clients.

For example, we can create a rule that only users under a specific customer's domain can see the "notification module". We can configure the rule to serve: Rollout percentage.

As shown below, the "notification module" has been delivered progressively only to users under the domain sho.saas-housekeeper.chouldbu.

Progressive release by a custom attribute

For example, users in the same Slack organization should see the same feature. So you can't release the new feature progressively dispatched by the user; you need to roll out the new feature dispatched by the organization. So instead of a percentage rollout identified by default user keyId, you can use a customized attribute as a rollout key. As shown below, I use team-region as the rollout key for a housekeeping project.

The users of the same regional team will have the same features.

Defects of feature flags

You need to inject a "feature flag" code into your program. Sometimes you fear that your feature flag code may cause the problem.

Over my 5 years of feature flags practice, it's an ultra-small probability event (but it happened). It's also the number one reason that some teams fear using feature flags.

Combined solution

Many mature companies combined the Traffic Routing method and Feature Flags method to build a robust delivery process:

Step-1. Use feature flags to wrap up new features and turn off new features for all users.
Step-2. Build new versions and deploy the new version online.
Step-3. Using traffic routing solutions to progressively transfer the traffic from the old version to the new version. This prevents the new version containers from breaking the existing services.
Step-4. Turn on the feature flag for each feature, and let only the QA team and beta users access this feature. Test it in production.
Step-5. Releasing features progressively. This can be done by developers, product managers, customer success, or marketing.

Conclusion

Minimizing delivery risk is always critical for the teams who serve clients seriously. Combining traffic routing and feature flag solutions helps teams significantly reduce delivery risk.

In reality, this combined solution will also help you to ship new features faster. Because you're fearless in shipping new features, the risk is under control. This allows you significantly innovate faster and increase your business income.

This solution is useful for both microservice and monolith architecture.

Feature Flag can be used in both front-end and back-end applications.

Blog