Introduction

In the last years AB-testing has become a default way to take data-driven decisions about the product development in many companies. There are teams of analysts working together with product managers to take all the crucial decisions.

If you are an analyst or lead an analytics team you know that the experiments are not always perfect. As any regular process the process of AB testing has some common pitfalls and areas for optimization. In this article I am telling about my experience in a big tech-company with good established processes, the problems we faced and the ways we have solved them. It focuses rather on organizational part of the process than on technical or mathematical side of AB testing. The latter will be covered in a separate article.

Process description

The process described in this paper is AB Testing of new features in user products. The Company operates in the technology sector and runs dozens of popular internet sites and mobile apps. All the products are constantly being improved. To make sure only changes that increase the product key metrics are launched, the product team runs AB tests, comparing the version before changes and the updated version of the website or app. The current procedure assumes that the product manager of the new feature is an internal customer of this process. The process can be simplified and described as presented in the picture:

A manager launches a new feature and asks an analyst to perform an AB test. If the AB test data is valid and the key metrics in the test group are higher than in the control group, the feature is launched into production. In another case, there is an iterative process of problem detection. The latter may take much time.

The general concern of all stakeholders is that it takes too long to assess the results of an experiment.

To get a better understanding of product managers’ view on the issue we can perform an I/P analysis to understand which areas have problems from the point of view of internal customers, what are our strengths and weaknesses.

X axis is the importance of the process feature to the customer
Y axis is customer’s assessment of how good is it done now

After visualising the I/P matrix, it becomes evident that the priorities are flawed due to the wrong incentives of the product managers (who are supposed to be internal customers). Product managers are interested in launching fast all the features they were working on (caring about sunk costs). They care less about the precision of the numbers and the quality of the product. Analysts are treated as an external resource; optimisation of their labour is not a priority.

It would be helpful to redraw the matrix from the point of view of the business owner.

From the point of view of the business owner, the most crucial goal is to keep a high quality of the product while launching new features fast. The time needed to test features matters much less than the time to launch the feature.

A bigger process

The process of AB-testing itself is not separate. It is a part of the process of launching a new feature.

Redefinition of the process customer and the borders of the process itself let us think broader about the potential ways to solve the problems we face. The process that we actually want to optimize is launching new features.

We can ask 5 Whys to understand better the problems of this bigger process:

In our case, defects are the most critical problem of the AB testing process. The quality of the product delivered to an AB test might sometimes be not good enough, which leads to several iterations of problem detection and explaining.

It can be fixed by improving the QA process, which is interesting because the potential optimisation for AB-testing does not depend on the analytical team at all.