This article is part of the Aspecto Hello World series, where we tackle microservices-related topics for you. Our team searches the web for common issues, then we solve them ourselves and bring you complete how-to guides. Aspecto is an OpenTelemetry-based distributed tracing platform for developers and teams of distributed applications.

Note: this tutorial assumes you are familiar with OpenTelemetry, traces, and spans. If you want to learn more about OpenTelemetry, check out The OpenTelemetry Bootcamp. This is a free, vendor-neutral, six-episode video series that brings you everything you need to know to get started with OpenTelemetry, from zero to hero.

Introduction

The evolution of OpenTelemetry (OTEL) in recent years has made it a lot easier for developers that are interested in better understanding their microservices – to instrument their services and gain that desired view.

But so far, the usage has been mainly for debugging production issues.

What if I told you there’s a way of utilizing OpenTelemetry’s power to prevent production issues, by using it in your integration test environment?

Sounds interesting? read on as I show you how it can be done easily.

What it would look like to use OpenTelemetry in an integration test

The end goal is to instrument our service under test while the test is running and make assertions on the created spans.

We call this: Trace-Based Testing.

Now you might be starting to think practically on such an implementation and say – “Oh, so I need to integrate the OpenTelemetry SDK now in my test run? Where do I store the spans? How do I get them while the test is already running so that I can assert?”

Well, those questions are legit indeed.

Luckily, you don’t need to implement all that on your own.

This use case is exactly what has led to the creation of Malabi, an open-source that wraps the OpenTelemetry SDK and does all this setup for you so that you can simply add it to your project and start asserting.

P.S. You can read more about the concept of Trace-Based testing and Malabi here.

How does Malabi help?

(Practical guide below, this is the theoretical part)

The way it works is simple.

You add Malabi to your project by using npm or yarn and add 3 lines of code at the top of the main file of the service.

Then, Malabi uses OpenTelemetry SDK for you and creates spans as your service is run (in the context of an integration test – for example in CI).

Malabi then stores these spans in memory and exposes an endpoint that lets you access these spans, and gives you utility functions to extract the data that you need for your assertions.

The practical part – how to take your existing NodeJs microservice and utilize OpenTelemetry to make assertions in an integration test

The following code is the ExpressJS code of the microservice that we want to test.

It is a simple service that uses SQLite as an in-memory database to store & retrieve data about users. It also stores some of the fetched data in a redis cache for faster retrieval.

Here is the code in the index.ts file:

Part 1 – The microservice code

* Note that you can find the complete code in the Malabi examples folder:

index.js file:

import * as malabi from 'malabi';
malabi.instrument();
malabi.serveMalabiFromHttpApp(18393);

import axios from 'axios';
import express from 'express';
import body from "body-parser";
import User from "./db";
import { getRedis } from "./redis";
import Redis from "ioredis";
let redis: Redis.Redis;

getRedis().then((redisConn) => {
   redis = redisConn;
   app.listen(PORT, () => console.log(`service-under-test started at port ${PORT}`));

})
const PORT = process.env.PORT || 8080;

const app = express();
app.use(body.json())
app.get('/',(req,res)=>{
   res.sendStatus(200);
})
app.get('/todo', async (req, res) => {
   try {
       const todoItem = await axios('https://jsonplaceholder.typicode.com/todos/1');
       res.json({
           title: todoItem.data.title,
       });
   } catch (e) {
       res.sendStatus(500);
       console.error(e, e);
   }
});

app.get('/users', async (req, res) => {
   try {
       const users = await User.findAll({});
       res.json(users);
   } catch (e) {
       res.sendStatus(500);
       console.error(e, e);
   }
});

app.get('/users/:firstName', async (req, res) => {
   try {
       const firstName = req.param('firstName');
       if (!firstName) {
           res.status(400).json({ message: 'Missing firstName in url' });
           return;
       }

       let users = [];
       users = await redis.lrange(firstName, 0, -1);
       if (users.length === 0) {
           users = await User.findAll({ where: { firstName } });
           if (users.length !== 0) {
               await redis.lpush(firstName, users)
           }
       }

       res.json(users);
   } catch (e) {
       res.sendStatus(500);
       console.error(e, e);
   }
});

app.post('/users', async (req, res) => {
   try {
       const { firstName, lastName } = req.body;
       const user = await User.create({ firstName, lastName });
       res.json(user);
   } catch (e) {
       res.sendStatus(500);
   }
})

In the above file, you see all the endpoints of the microservice. Mostly self-explanatory – fetching, storing data in SQLite DB / Redis as cache.

But notice the top three lines where the Malabi magic happens:

import * as malabi from 'malabi';
malabi.instrument();
malabi.serveMalabiFromHttpApp(18393);

Basically, we require Malabi (after running npm install –save-dev malabi of course).

Then, Malabi instruments our service – meaning it will create spans (in memory) as it runs.

At that point, we tell it to serve the created spans from port 18393.

In part 2, you will see how we use Malabi util functions to query this endpoint and use Jest to make assertions on them. But first, let’s continue to understand our service.

This db.ts file that handles SQLite with Sequelize:

import { Sequelize, DataTypes } from 'sequelize';

const sequelize = new Sequelize({
   dialect: 'sqlite',
   storage: ':memory:'
});

const User = sequelize.define('User', {
   firstName: {
       type: DataTypes.STRING,
       allowNull: false
   },
   lastName: {
       type: DataTypes.STRING
   }
});

User.sync({ force: true }).then(() => {
   User.create({ firstName: "Rick", lastName: 'Sanchez' });
})

export default User;

The redis.ts file:

import { RedisMemoryServer } from 'redis-memory-server';
import Redis from "ioredis";
const redisServer = new RedisMemoryServer();

export async function getRedis() {
   const host = await redisServer.getHost();
   const port = await redisServer.getPort();
   const redis = new Redis(port, host);
   return redis;
}

Part 2 – The Test Code

service-under-test.spec.ts file:

This file will be run using Jest.

Notice we have the port of the service itself (to call the actual running service, which you would run in the CI environment or locally).

We also have the Malabi utility functions – fetchRemoteTelemetry, clearRemoteTelemetry that like their name suggests – fetch the spans from the endpoint for assertions and clear the in-memory cache (which is useful to clean up between tests to maintain a clean slate each time).

Take a look at the code, more info follows below:

const SERVICE_UNDER_TEST_PORT = process.env.PORT || 8080;
import axios from 'axios';
import { fetchRemoteTelemetry, clearRemoteTelemetry } from 'malabi';
const getTelemetryRepository = async () => await fetchRemoteTelemetry({ portOrBaseUrl: 18393 });

describe('testing service-under-test remotely', () => {
   beforeEach(async () => {
       // We must reset all collected spans between tests to make sure spans aren't leaking between tests.
       await clearRemoteTelemetry({ portOrBaseUrl: 18393 });
   });

   it('successful /todo request', async () => {
       // call to the service under test - internally it will call another API to fetch the todo items.
       const res = await axios(`http://localhost:${SERVICE_UNDER_TEST_PORT}/todo`);

       // get spans created from the previous call
       const telemetryRepo = await getTelemetryRepository();

       // Validate internal HTTP call
       const todoInteralHTTPCall = telemetryRepo.spans.outgoing().first;
       expect(todoInteralHTTPCall.httpFullUrl).toBe('https://jsonplaceholder.typicode.com/todos/1')
       expect(todoInteralHTTPCall.statusCode).toBe(200);
   });

   it('successful /users request', async () => {
       // call the service under test
       const res = await axios.get(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users`);

       // get spans created from the previous call
       const telemetryRepo = await getTelemetryRepository();

       // Validating that /users had ran a single select statement and responded with an array.
       const sequelizeActivities = telemetryRepo.spans.sequelize();
       expect(sequelizeActivities.length).toBe(1);
       expect(sequelizeActivities.first.dbOperation).toBe("SELECT");
       expect(Array.isArray(JSON.parse(sequelizeActivities.first.dbResponse))).toBe(true);
   });

   it('successful /users/Rick request', async () => {
       // call the service under test
       const res = await axios.get(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users/Rick`);

       // get spans created from the previous call
       const telemetryRepo = await getTelemetryRepository();

       const sequelizeActivities = telemetryRepo.spans.sequelize();
       expect(sequelizeActivities.length).toBe(1);
       expect(sequelizeActivities.first.dbOperation).toBe("SELECT");

       const dbResponse = JSON.parse(sequelizeActivities.first.dbResponse);
       expect(Array.isArray(dbResponse)).toBe(true);
       expect(dbResponse.length).toBe(1);
   });

   it('Non existing user - /users/Rick111 request', async () => {
       // call the service under test
       const res = await axios.get(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users/Rick111`);

       // get spans created from the previous call
       const telemetryRepo = await getTelemetryRepository();

       const sequelizeActivities =  telemetryRepo.spans.sequelize();
       expect(sequelizeActivities.length).toBe(1);
       expect(sequelizeActivities.first.dbOperation).toBe("SELECT");

       const dbResponse = JSON.parse(sequelizeActivities.first.dbResponse);
       expect(Array.isArray(dbResponse)).toBe(true);
       expect(dbResponse.length).toBe(0);

       expect(telemetryRepo.spans.httpGet().first.statusCode).toBe(200);
   });

   it('successful POST /users request', async () => {
       // call the service under test
       const res = await axios.post(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users`,{
           firstName:'Morty',
           lastName:'Smith',
       });

       expect(res.status).toBe(200);

       // get spans created from the previous call
       const telemetryRepo = await getTelemetryRepository();

       // Validating that /users created a new record in DB
       const sequelizeActivities =  telemetryRepo.spans.sequelize();
       expect(sequelizeActivities.length).toBe(1);
       expect(sequelizeActivities.first.dbOperation).toBe("INSERT");
   });


   /* The expected flow is:
       1) Insert into db the new user (due to first API call; POST /users).
       ------------------------------------------------------------------
       2) Try to fetch the user from Redis (due to the second API call; GET /users/Jerry).
       3) The user shouldn't be present in Redis so fetch from DB.
       4) Push the user object from DB to Redis.
   */
   it('successful create and fetch user', async () => {
       // Creating a new user
       const createUserResponse = await axios.post(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users`,{
           firstName:'Jerry',
           lastName:'Smith',
       });
       expect(createUserResponse.status).toBe(200);

       // Fetching the user we just created
       const fetchUserResponse = await axios.get(`http://localhost:${SERVICE_UNDER_TEST_PORT}/users/Jerry`);
       expect(fetchUserResponse.status).toBe(200);

       // get spans created from the previous calls
       const telemetryRepo = await getTelemetryRepository();
       const sequelizeActivities = telemetryRepo.spans.sequelize();
       const redisActivities =  telemetryRepo.spans.redis();

       // 1) Insert into db the new user (due to first API call; POST /users).
       expect(sequelizeActivities.first.dbOperation).toBe('INSERT');
       // 2) Try to fetch the user from Redis (due to a second API call; GET /users/Jerry).
       expect(redisActivities.first.dbStatement).toBe("lrange Jerry 0 -1");
       expect(redisActivities.first.dbResponse).toBe("[]");
       // 3) The user shouldn't be present in Redis so fetch from DB.
       expect(sequelizeActivities.second.dbOperation).toBe("SELECT");
       //4) Push the user object from DB to Redis.
       expect(redisActivities.second.dbStatement.startsWith('lpush Jerry')).toBeTruthy();
   });
});

Once we have fetched the spans, we can use Jest’s expect function to make assertions, as we would in any other test regardless of trace-based testing.

Examined example 1 – test named “successful /users request”:

Let’s examine the above code – for example, the test named “successful /users request”.

First, we call the service to fetch all users.

Then, we use the fetchRemoteTelemetry wrapped by the getTelemetryRepository function to get the spans from Malabi.

After that, we use the sequelize accessor to filter only sequelize spans.

Once we have the sequelize spans at hand, we can assert to have only 1 as we only fetch the DB once.

We also know it’s a SELECT operation, so we assert that it’s a SELECT operation.

Examined example 2 – test named “successful create and fetch user”:

Let’s now examine a slightly more complicated test.

In the indicated test, we create a new user using POST /users. Then, we try to query for that user using GET /users/:firstName.

As expected, we assert for 200 as you would in any other test.

Now here again we use Malabi utilities to fetch relevant spans, and store them in variables – one for redis spans and another for sequelize:

const telemetryRepo = await getTelemetryRepository();
const sequelizeActivities = telemetryRepo.spans.sequelize();
const redisActivities =  telemetryRepo.spans.redis();

The first assertion – we want to make sure that the initial POST operation had caused a DB INSERT operation:

expect(sequelizeActivities.first.dbOperation).toBe('INSERT');

Since the user was just created, we expect it to not exist in redis, so we assert the redis query to be as we want it and expect the response from redis to be an empty array:

expect(redisActivities.first.dbStatement).toBe("lrange Jerry 0 -1");
expect(redisActivities.first.dbResponse).toBe("[]");

Since the user was not present in redis, we expect to have fetched the DB – so assert that the second DB operation select

expect(sequelizeActivities.second.dbOperation).toBe("SELECT");

And now we expect redis to have received push command to make sure in real life(not in test runs since we clean up everything), subsequent runs would not have to fetch the DB (but take from redis):

expect(redisActivities.second.dbStatement.startsWith('lpush Jerry')).toBeTruthy();

That would be it! I hope you can see how simple it can be to use OpenTelemetry & Malabi to write powerful integration tests, in a much easier way than before.

P.S. Malabi is a relatively new library implementing a new approach, and its authors (myself included) would love to hear your thoughts on it and hear any improvements/suggestions you have. So feel free to open a discussion in GitHub or contact me via Twitter DMs.

Blog

How to Improve Your Integration Tests Using OpenTelemetry

Tom Weiss