End-to-end testing with our Chaos Stream Proxy

Testing the behavior of a video player when there is something wrong with the video source used to be a tedious task: You had to either manually create or hunt down different broken streams to test different errors, and even then you couldn't always know for sure that the same error would always occur with the same timing.

To address this problem, Eyevinn has released its Chaos Stream Proxy as open-source! As the name suggests, this is a very handy tool for proxying an adaptive bitrate stream and deterministically introducing corruptions to it.

This tutorial will demonstrate how to use the Proxy in the end-to-end testing of the open-source Eyevinn WebPlayer, using Playwright.

Using the Chaos Stream Proxy

A demo of the Chaos Stream Proxy, which we'll use for the purpose of this tutorial, is currently running here.

The Proxy supports both HLS and MPEG-DASH streaming formats, and easily lets us add different corruptions by adding a stringified JSON object as a query parameter to the proxied URL. If we take this HLS URL:

https://maitv-vod.lab.eyevinn.technology/VINN.mp4/master.m3u8

it can be played with no problems, for example in the Eyevinn WebPlayer, hosted here. But, if we proxy it through the Chaos Stream Proxy, and set &statusCode=[{i:*,code:404}]:

https://chaos-proxy.prod.eyevinn.technology/api/v2/manifests/hls/proxy-master.m3u8?url=https://maitv-vod.lab.eyevinn.technology/VINN.mp4/master.m3u8&statusCode=[{i:*,code:404}]

we'll instead get 404 errors for all segments (because we set i:*; it's also possible to specify individual segments).

For specifying other types of corruptions, check out the project's README.

EPAS and the Eyevinn WebPlayer

The Eyevinn WebPlayer implements Eyevinn Player Analytics Specification (EPAS), which is an open specification that defines a standard for implementing analytics in any video/audio player. This means that we have a format for event reporting that we expect the WebPlayer to follow.

User interactions are easy enough to create for the sake of validating the EPAS implementation, and thanks to the Chaos Stream Proxy, manifest related errors are now also a piece of cake!

Setting up Playwright

Playwright is a powerful end-to-end test runner, which we're using to automatically test our code in a real browser environment. At the time of writing, we're adding a dependency for "@playwright/test": "^1.19.2" in our package.json(this is a Node.js project). We also add the script "test:e2e": "playwright test".

Our playwright.config.ts is in the project root, and looks like this:

import { PlaywrightTestConfig, devices } from '@playwright/test';

const config: PlaywrightTestConfig = {
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  testDir: "tests/",
  workers: 3,
  use: {
    trace: 'on-first-retry',
    // Necessary to get the media codecs to play video (default 'chromium' doesn't have them) 
    channel: 'chrome'
  },
  webServer: {
    command: 'npm run examples',
    port: 1234,
    reuseExistingServer: !process.env.CI,
  },
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
  ],
};
export default config;

testDir points to the folder where we put our test .spec.ts files; examples in the npm run examples command is a folder in our root directory, in which we put the HTML files that Playwright will use.

Note that channel: 'chrome' is necessary for our tests to run; the default setting is chromium, which doesn't include the necessary media codecs for video playback . This may cause issues with automated workflows and running tests on Firefox and Webkit, but works locally in Chrome.

In our examples folder, we add an index.html HTML file that includes a link to our Chaos Stream Proxy HTML:

<!DOCTYPE html>
<html>

<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <title>Examples</title>
</head>
<body>
  <a href="chaos-proxy/index.html">Chaos Stream Proxy Example</a>
</html>

As well as the Chaos Stream Proxy HTML index.html file in the sub-folder chaos-proxy:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <title>Chaos Stream Proxy Example</title>
  <script async src="../../packages/web-player-component/dist/web-player.component.js"></script>
</head>
<body>
  <!-- VOD: With segment delay of 1500ms and response code 400 on sixth (response of 400 will be sent after 1500ms): -->
  <eyevinn-video
    source="https://chaos-proxy.prod.eyevinn.technology/api/v2/manifests/hls/proxy-master.m3u8?url=https://maitv-vod.lab.eyevinn.technology/VINN.mp4/master.m3u8&delay=[{i:5,ms:1500}]&statusCode=[{i:5,code:400}]"
    muted autoplay>
  </eyevinn-video>
  <!-- VOD: With response of status code 404 on all segments: -->
  <eyevinn-video
  source="https://chaos-proxy.prod.eyevinn.technology/api/v2/manifests/hls/proxy-master.m3u8?url=https://maitv-vod.lab.eyevinn.technology/VINN.mp4/master.m3u8&statusCode=[{i:*,code:404}]"
  muted autoplay>
</eyevinn-video>
</body>
</html>

This file imports the compiled version of Eyevinn WebPlayer Component from the repository, which let's us embed the WebPlayer as web components with the <eyevinn-video> tags.

In our tests folder, we add chaos-proxy.specs.ts, which will be run automatically when we run the npm run test:e2e script:

import { test } from '@playwright/test';

  test('player sends error events when loading corrupt streams', async ({page}) => {
    const [request] = await Promise.all([
      page.goto('/chaos-proxy/index.html'),
      page.waitForRequest(req => req.url().match('https://sink.epas') && req.method() === 'POST' && req.postDataJSON().event === "warning" && req.postDataJSON().payload.code === "400"),
      page.waitForRequest(req => req.url().match('https://sink.epas') && req.method() === 'POST' && req.postDataJSON().event === "warning" && req.postDataJSON().payload.code === "404"),
    ]);
  });

We make asynchronous calls to the goto and waitForRequest methods, available on the pageobject. We check that the POST requests conform to the expected format: If both matching requests are made within 30 seconds, the test passes!

We run the test with npm run test:e2e, and this should be the result:

Success!

Bonus: Playwright Test for VSCode and debug mode

Playwright is developed my Microsoft, so perhaps it shouldn't come as a surprise that there is an excellent VS Code extension available.

With this extension, it's a breeze to run individual tests in isolation, add breakpoints and more!

It also gives easy access to debug mode, which lets us view and interact with the test in the test runner browser:

Blog

End-to-end testing with our Chaos Stream Proxy

Sebastian Ljungman

Using the Chaos Stream Proxy

EPAS and the Eyevinn WebPlayer

Setting up Playwright

Bonus: Playwright Test for VSCode and debug mode

Join Our Newsletter. No Spam, Only the good stuff.

Related