When it comes to releasing new features, it is often difficult to anticipate how users will react and interact with them. This is where A/B testing comes in useful. It provides a mechanism to test and evaluate two variations of an app to determine which is better by releasing them to an isolated user segment before a full deployment. This level of flexibility allows developers to quickly experiment with new features without affecting the production environment.

What is an A/B Test

An A/B test is a type of testing in which two variations of an app are released for testing to an isolated user segment before being deployed to the entire user base to collect metrics and feedback. One variation is designated variation A, and the other is designated variation B. Variation A usually represents the current version of the app and is considered a control or benchmark variation, while variation B represents the version of the app with the new feature to be tested.

Conducting a demo A/B test

Here are two variations of a demo website built with Golang. I aim to determine if changing the design of the home page will influence a higher number of visits to the Store page.

Before we dive deeper, let's take a moment to discuss a few additional technologies that will help streamline the A/B testing process.

The role feature flags play

Feature flags are virtual switches that you can integrate into your app to control what features users can and cannot see. An added benefit of using feature flags is that you can easily switch from one variation to the other with ease and target specific user segments when releasing new features. They let you launch new features and change your software configuration without (re)deploying code.

Let's see how we can set this up using ConfigCat's feature flag service. For context, ConfigCat supports simple feature toggles, user segmentation, and A/B testing and has a generous free tier for low-volume use cases or those just starting out.

Creating feature flags for the test

Let's create two feature flags. One will be responsible for showing the old home page (Variation A) and the other will be responsible for showing the new home page (Variation B).

To create the feature flags, you'll need to sign up for a free ConfigCat account.

1. In the dashboard create two feature flags as shown below:

2. I'll turn off the newHomePage feature flag, for now, to focus on collecting metrics from the old home page first. More on this in a later step.

Creating user segments

1. I'll create two user segments within my ConfigCat dashboard. Each segment will consist of 50 users represented by their email addresses. One segment will see variation A and the other will see variation B.

2. I'll link the feature flags to the segments. In this way, when a flag is switched on, the variation it controls will only be shown to users in the segment it is linked to.

Integrating the feature flags into Go

Let's look at how we can integrate the feature flags into the Golang app.

1. Install the ConfigCat's Go SDK, then add it to the import statement in main.go:

go get github.com/configcat/go-sdk/v7

import (
  "github.com/configcat/go-sdk/v7"
)

2. Create the client using your ConfigCat's SDK Key. You should keep this key private and outside of your code base.

client := configcat.NewClient("YOUR_CONFIGCAT_SDK_KEY")

3. Add the user object. Here, I am using the user's email address. This is required to help ConfigCat to decide whether the user belongs to a particular segment.

var user = &configcat.UserData{Email: "email1@example.com"}

4. Create variables to store the value of each feature flag and create if-statements to render the variation of the home page it is responsible for:

func homePageHandler(w http.ResponseWriter, r *http.Request) {
    isNewHomePageEnabled := client.GetBoolValue("newhomepage", false, user)
    isOldHomePageEnabled := client.GetBoolValue("oldhomepage", false, user)

    if isNewHomePageEnabled { newHomePageTemplate.Execute(w, nil)} 
    if isOldHomePageEnabled { oldHomePageTemplate.Execute(w, nil)} 

}

With that set, let's look at how we can collect metrics for the test.

Collecting metrics

I plan to collect metrics for variation A over a period of three weeks and then do the same for variation B. To do this, I will use a product analytics tool called Amplitude.

1. Sign up for a free Amplitude account.

2. Switch to the Data section by clicking the dropdown at the top left.

3. Click the Sources link under Connections in the left sidebar, then click the + Add Source button on the top right.

4. Select the GO SDK from the SDK sources list and enter the following details to create a source:

5. You should be automatically redirected to the implementation page. We'll revisit the instructions on this page soon but first, let's add an event.

Adding an event

To keep track of the number of visits to the store page, I'll create an event. In a later step, I'll trigger and log this event to Amplitude when the store page is rendered.

1. Click the Events link in the left sidebar under Tracking Plan to access the events page.

2. Click the + Add Event button at the top right to create an event and fill in the following details:

3. Click the Save changes button in the left sidebar.

Integrating Amplitude into Go

Click the Implementation link in the left sidebar to see the integration instructions page.

1. Install the Amplitude CLI with the following command:

sh
npm install -g @amplitude/ampli


2. Install the Amplitude Go SDK:

go get github.com/amplitude/analytics-go

3. Run the following command to pull the necessary configurations we added in Amplitude's dashboard into the Golang app.

ampli pull

Sending the page visit event to Amplitude

1. Import Amplitude in main.go

import (
    "github.com/configcat-labs/ab-testing-in-go-sample/ampli"
)

2. Call the PageView event method on the ampli instance to send an event when the store page is visited:

func storePageHandler(w http.ResponseWriter, r *http.Request) {
    ampli.Instance.PageView("user@example.com") // New*
    storePageTemplate.Execute(w, nil)
}

Let's check the logged requests to make sure the Go app is connected to Amplitude.

Checking for logged requests

1. Under Connections in the left sidebar, click on Sources.

2. Clicking on the Visit the store button on the home page will redirect the user to the store page. When the store page is rendered the event will be logged to Amplitude as shown in the Successful Requests graph:

Setting up an analysis chart

To effectively analyze the incoming events from the app, I'll set up an Analysis chart. This will give me visual insights into how each variation is performing in real-time.

1. Switch to the Analytics dashboard, by clicking the dropdown arrow on the top left next to Data.

2. In the analytics dashboard, click the New button in the left sidebar.

3. Select Analysis, then select Segmentation.

4. Select the event as shown below:

5. Click Save on the top right to save the chart.

I'll leave this to run for three weeks to see how Variation A performs. When the three-week period is up, I'll switch off the oldHomePage feature flag and turn on the newHomePage feature flag which will release variation B to users in User segment B.

Analyze the test results

Depending on how long you decide to run your tests, at the end of the time period you can use the Compare to past dropdown option at the top right of the analysis chart to compare the two variations. For example, if the total number of visits to the store page was 40 for the past three weeks for variation A, you can compare that to the three-week period where you released variation B to determine which is performing better.

To keep track of your feature flag changes you can add it to your charts on Amplitude.

Conclusion

By test-driving new features through an A/B test, we can see what works and what doesn't. This method allows you to test and refine features without affecting the entire user base. Feature flags play an important role in this process by allowing us to switch between variations and set up user segmentation. ConfigCat's dashboard is simple to use allowing you and your teammates to get up and running with feature flags quickly. You can learn more about the various languages and frameworks that ConfigCat supports here.

Stay on top of the latest posts and announcements from ConfigCat on Twitter, Facebook, LinkedIn, and GitHub.