A/B testing a Vue app with AlephBet

In this article, we will set up A/B testing for a simple Vue.js app and process the experiments on a Node.js server, saving them in a SQLite database.

Before we get started, if you need a refresher on A/B testing or want to learn about how A/B testing is used by Netflix to drive user experience, I can recommend this series on the Netflix tech blog.

All code of this post is on Github.

Goal

The idea is to build a simple counter app with a button that increments a counter with every click. The A/B test could (for example) test if users prefer a blue button or a green button.

When the user clicks the button, the event is posted to the node server, which will store the results in a database for later analysis.

While the set up is simple, it should demonstrate the principles involved relatively clearly.

Technologies

Vue 2 app on which we want to perform A/B testing
Node/Express server to process experiment results
SQLite database to store experiment results
AlephBet framework for A/B testing

Setting up the Vue project

First, set up a basic Vue project. We will use Vue 2 here.

vue create client

Next, we will make some changes to the HelloWorld component, which will be the subject of our tests. The Vue component will have a single button and counter. The .is-blue and .is-green CSS classes will be used later for the A/B test.

// client/src/components/HelloWorld.vue

<template>
  <div class="hello">
    <p id="counter"> {{ counter }}</p>
    <button id="increment-btn" @click="increment">Increment</button>
  </div>
</template>

<script>
export default {
  name: 'HelloWorld',
  data() {
    return {
      counter: 0,
    }
  },
  methods: {
    increment: function() {
      this.counter++;
    }
  }
}
</script>

<style scoped>
#counter {
  font-size: xxx-large;
}

#increment-btn {
  border: none;
  padding: 15px 32px;
  font-size: x-large;
  margin: 4px 2px;
  cursor: pointer;
}

.is-blue {
  background-color: #34495e;
  color: white;
}

.is-green {
  background-color: #41b883;
  color: white;
}

</style>

Setting up AlephBet

As mentioned before, we will use AlephBet for managing the A/B tests.

Configuration

First, install the dependency:

npm install alephbet

Next, we can configure our tests. Set up a new file in the Vue project src/analytics/ab-testing.js.

After importing AlephBet, we will set up a wrapper method makeExperiment to create new experiments.

// client/src/analytics/ab-testing.js

const AlephBet = require("alephbet");

/**
 * Set up a new A/B testing experiment with AlephBet
 * @param name Name of the experiment
 * @param variants Object of the experiment variants
 * @param adapter Adapter of the experiment
 * */
const makeExperiment = (name, variants, adapter) => {
    return new AlephBet.Experiment({
        name: name,
        variants: variants,
        tracking_adapter: adapter,
    });
};

AlephBets uses Google Analytics by default as the experiment adapter. Since we want to set up our own backend, we need a custom adapter. For now, we will simply log all events to the console.

// client/src/analytics/ab-testing.js

/**
 * Wrapper for an A/B testing adapter for AlephBet experiments.
 * */
const makeAdapter = () => {
    return {
        experiment_start: async function (experiment, variant) {
            console.log(experiment, variant, 'participate')
        },
        goal_complete: async function (experiment, variant, event_name) {
            console.log(experiment.name, variant, event_name)
        },
    };
};

Defining the variants

Next, we can define our experiment variants. We will store these in a JS object experimentVariants with the key being the name of the experiment.

// client/src/analytics/ab-testing.js

// Experiment variant presets
const experimentVariants = {
    "button color": {
        green: {
            activate: function () {
                document.getElementById("increment-btn").className = "is-green";
            },
            weight: 50,
        },
        blue: {
            activate: function () {
                document.getElementById("increment-btn").className = "is-blue";
            },
            weight: 50,
        },
    },
};

In the inner object, we define two variants blue and green. Each variant has an activate function that is called by AlephBet when the variant is activated for a user. In our case, the activate function adds the .is-green or .is-blue CSS class, respectively, to the increment button.

The weight specifies the likelihood that a user is allocated the variant.

Lastly, export the two wrapper methods and the object with the variants.

module.exports = {
    makeExperiment: makeExperiment,
    makeAdapter: makeAdapter,
    experimentVariants: experimentVariants,
}

Configuring the Vue component

The actual experiment is configured in the HelloWorld component, specifically in the mounted method of the component.

Start by importing the functions we just created as well as AlephBet. We also need to define the variable for the goal in the most outer scope of the component.

// client/src/components/HelloWorld.vue

import {
  experimentVariants,
  makeAdapter,
  makeExperiment,
} from "@/analytics/ab-testing";
import AlephBet from "alephbet";

let goal;
...

The goal variable captures the completion of the goal of the experiment - in this case when the user clicks the button.

In the mounted method, set up the goal with the experiment. We set unique: false since we want every click to be registered and not merely the first click.

// client/src/components/HelloWorld.vue

...
  mounted() {
    const name = "button color";
    const variants = experimentVariants[name];
    const adapter = makeAdapter();
    const experiment = makeExperiment(name, variants, adapter);
    goal = new AlephBet.Goal("button clicked", {unique: false});
    experiment.add_goal(goal);
  },
...

Finally, we need to actually register the goal completion when the button is clicked. As the increment function is called on click, we can simply add one line to that method.

// client/src/components/HelloWorld.vue

...
increment: function() {
      goal.complete()
      this.counter++;
    }
...

That is the basic set up of the client complete. Fire up your application and head over to localhost:8080. You should now either see a blue or a green increment button. AlephBet actually stores the variant in localStorage so that one user is always shown the same variant. Hence, if you'd like to see the other variant, delete the alephbet entry from localStorage and refresh the page (you might have to do this a few times until you randomly get allocated to the other group).

If you open the console, you will also notice that the adapter logs the start of the experiment as well as every goal completion, that is every button click.

Next on the list is the set up of our experiment tracking server.

Node.js server setup

Start by setting up a second directory server on the same level as the client Vue app, then setup the npm project and install the dependencies.

mkdir server
cd server
npm init
npm install express cors sqlite3

index.js

Next, create a file in server called index.js and add the following content:

// server/index.js

const express = require("express")
const app = express()
const cors = require('cors')

app.use(cors())

// Server port
const PORT = 5555;

// Start server
app.listen(PORT, () => {
    console.log(`Server running on port ${PORT}`)
});

// Root endpoint
app.get("/", (req, res, next) => {
    res.json({"message":"Ok"})
});

// A/B testing endpoint
app.post("/track_experiment", (req, res) => {
    const experiment = req.query.experiment;
    const variant = req.query.variant;
    const event = req.query.event;

    if (experiment === null || variant === null || event === null) {
        res.status(400);
        return;
    }

    console.log(experiment, variant, event);
    res.json({"message":"Ok"})
})

// 404 not found for other requests
app.use(function(req, res){
    res.status(404);
});

We won't go into detail here, but essentially we're setting up a simple server running on PORT 5555 with the /track_experiment endpoint, to which we can send our experiment events from the counter app.

database.js

To persistently store the experiment results, we use a simple SQLite database. The setup here is very basic and could (should!) be improved but it is sufficient for this proof of concept.

In the server directory create the following file database.js:

// server/database.js

const sqlite3 = require('sqlite3').verbose()

const DB_FILE = "db.sqlite"

let db = new sqlite3.Database(DB_FILE, (error) => {
    if (error) {
        // Error opening db
        console.error(error.message)
        throw error
    }
    else{
        console.log('Connected to the SQLite database.')

        const sql = `
        CREATE TABLE experiment (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            name text, 
            variant text, 
            event text
            );
        `
        db.run(sql, (err) => {/* table already exists */});
    }
});

module.exports = db

This sets up a single table experiment with the columns id, name, variant, and event.

Now that we've set up the database, we can fill out the rest of the API endpoint.

Back to index.js

First, import the db object at the start of the file.

// server/index.js
...
const db = require("./database.js")
...

We can now update the /track_experiment to insert the incoming experiment data into the database. The final endpoint should look like this.

// server/index.js

...

// A/B testing endpoint
app.post("/track_experiment", (req, res) => {
    const experiment = req.query.experiment;
    const variant = req.query.variant;
    const event = req.query.event;

    if (experiment === null || variant === null || event === null) {
        res.status(400);
        return;
    }

    // Insert into database
    const sql = 'INSERT INTO experiment (name, variant, event) VALUES (?, ?, ?)'
    const params = [experiment, variant, event];

    db.run(sql, params, function (error, result) {
        if (error){
            res.status(400).json({"error": error.message})
            return;
        }
        res.json({
            "message": "success",
            "data": params,
            "id" : this.lastID
        })
    })
})

...

We can try this out by starting the server node server/index.js and sending a test request to the endpoint with curl.

curl --request POST "http://localhost:5555/track_experiment?experiment=myname&variant=myvariant&event=myevent"

> {"message":"success","data":["myname","myvariant","myevent"],"id":1}%

Success!

Integrating the Vue app with the server

Server and DB are running, thus we can now connect the client to the server.

Head back to the client directory. We will edit the ab-testing.js file.

We first need to add a method to post the event to the tracking server.

// client/src/analytics/ab-testing.js

/**
 * Post an experiment result to the tracking server.
 * */
const postResult = (experiment, variant, event) => {
    let URL = "http://localhost:5555/track_experiment"
    URL += `?experiment=${experiment}&variant=${variant}&event=${event}`
    fetch(URL, {
        method: 'POST'
    }).catch(console.error)
}

Almost done. Now, in the makeAdapter wrapper function we want to use this new method. Update the code as follows:

// client/src/analytics/ab-testing.js

...

const makeAdapter = () => {
    return {
        experiment_start: async function (experiment, variant) {
            postResult(experiment.name, variant, 'participate')
        },
        goal_complete: async function (experiment, variant, event_name) {
            postResult(experiment.name, variant, event_name)
        },
    };
};

...

That's it. Any experiment results will now be posted to the server and saved in the database:

Final thoughts

Setting up basic A/B tests with AlephBet isn't rocket science. While this demo project is fairly simple, I hope it serves as a decent introduction.

Feel free to reach out for feedback!

Happy A/B testing 🎉

Blog