Introduction

Test automation is a way to automate a browser by simulating user actions like clicks. Applications become more complex as they are built on layers and entire networks of sub-systems, including UI and API layers, external databases, networks, and even third-party integrations. So, there is always a need for thorough testing to be done. This ranges from unit testing to end-to-end testing of applications. Test automation is one of the ways to ascertain the stability and the health of applications. That can lead to the success of the application when in production. The general stability of the application doesn't entirely depend on end-to-end testing but it surely helps to detect bugs in applications, among other benefits.

Below are some of the popular test automation frameworks being used;

• Selenium (WebdriverIO)
• Puppeteer
• Cypress
• Playwright

Playwright maybe the new kid on the block but it's gaining quite a trajectory among software developers and testers.

Project Background

The Playwright evaluation is a result of a need for to check if WebdriverIO is still a good test automation framework compared to some of the best non-Selenium modern test automation frameworks. MediaWiki is implemented in a large number of repositories. So, in order to ensure good code practices across all these repositories, an extensive amount of testing is performed. One of the tests performed is an end-to-end test. WebdriverIO is the current browser automation framework being used for implementing the end-to-end tests. However, with the recent increase in awareness about end-to-end testing, a number of equally competitive non-Selenium solutions have been introduced and one of them is Playwright.

At the time of choosing which automation framework to use in the implementation of the test automation, extensive research was done and an evaluation of WebdriverIO was carried out by Željko Filipin, who is a Senior Engineer in Test at Wikimedia Foundation and also one of my mentors. To verify if WebdriverIO is still a great testing framework for MediaWiki, a number of modern and non-Selenium automation frameworks have been evaluated against WebdriverIO. These evaluations were done by Soham Parekh who was Google Summer of Code 2020 intern and is also one of my mentors and Sim T.H. Tran. These evaluations can be found in the links below;

• Cypress Evaluation by Sim T.H. Tran.
• Puppeteer Evaluation by Soham Parekh.

So, this brings us to the current framework being evaluated against WebdriverIO, which is Playwright. The evaluations progress involved the following steps;

• Configuring and setting up of Wikimedia Core to run on continuous integration, in this case on Github Actions.
• Implementation of all MediaWiki Core tests in Playwright.
• Running MediaWiki Core tests in WebdriverIO.
• Analysis of both WebdriverIO and Playwright tests in terms of speed and stability.

WebdriverIO

WebdriverIO is a test automation framework that allows you to run tests based on the WebDriver protocol and Appium automation technology.

WebdriverIO provides the ability and options to run commands in both asynchronous and synchronous operations. For asynchronous operations, you can use the JavaScript usual async/await, however for synchronous, it can be done through node-fibers. Both of these options have their own benefits and issues which can be found on the WebdriverIO website.

Sample WebdriverIO Code

The code snippet below navigates to wikipedia.org in Chrome and checks if the correct title is being displayed.

const { remote } = require('webdriverio');

(async () => {
 const browser = await remote({
        capabilities: {
            browserName: 'chrome'
        }
  });
  await browser.url('https://www.wikipedia.org/');

  const title = await browser.getTitle();
  expect(title).toHaveTitle('The Free Encyclopedia');

  await browser.deleteSession();
})();

Mocha

The WebdriverIO WDIO runner currently supports Mocha, Jasmine, and Cucumber frameworks and WikiMedia Core test is currently using Mocha framework. Mocha is a feature-rich JavaScript test framework running on Node.js and in the browser, making asynchronous testing simple and fun.

WebdriverIO provides an adapter for Mocha which can easily be added as @wdio/mocha-framework.

Sample Mocha Code

The code snippet below navigates to wikipedia.org in Chrome and checks if the correct title is being displayed.

const { remote } = require('webdriverio');

describe('Wikipedia home page', async() => {
  const browser = await remote({
        capabilities: {
            browserName: 'chrome'
        }
  });

  it('should display correct page title', async() => {
    await browser.url('https://www.wikipedia.org/');

    expect(browser).toHaveTitle('The Free Encyclopedia');

    await browser.deleteSession();
  });
})();

WebdriverIO Advantages

WebdriverIO offers some great pros such as;

• Extendable, adding helper functions, or more complicated sets and combinations of existing commands is simple and really useful.
• It's Compatibility nature enables it to run on the WebDriver Protocol for true cross-browser testing as well as Chrome DevTools Protocol for Chromium-based automation using Puppeteer.
• Rich feature with huge variety of community plugins that allows you to easily integrate and extend your setup to fulfill your requirements.
• Stable features.
• Synchronous implementation of asynchronous browser commands.
• Excellent API documentation.
• Support for modern web and mobile frameworks.

WebdriverIO Disadvantages

WebdriverIO also presents some cons such as;

• Additional effort to set up browser driver with selenium-standalone or ChromeDriver especially in v5 and below.
• Much slower compared to frameworks like Playwright and Puppeteer.

To explore more WebdriverIO capabilities, you can check out the WebdriverIO and MediaWiki documentation.

Playwright

Playwright is a Node.js library to automate Chromium, Firefox, and WebKit with a single API.

While working with Playwright for the past few months, it has come across as easy to install and setup. The package takes care of installing all the browsers (Chromium, Firefox, and WebKit). Capturing screenshots is an out-of-the-box experience. However, video recording requires separate installation of playwright-video and ffmpeg but they all blend in with Playwright seamlessly.

Sample Playwright Code

The code snippet below navigates to wikipedia.org in Chrome and checks if the correct title is being displayed.

const { chromium } = require('playwright');
const expect = require('expect-playwright');

(async () => {
  let browser, page;
  browser = await chromium.launch();
  page = await browser.newPage();
  await page.goto('https://www.wikipedia.org/');

  const title = await page.$('.localized-slogan');
  await expect(title).toEqualText('The Free Encyclopedia');

  await browser.close();
})();

Jest

Jest is a delightful JavaScript testing framework with a focus on simplicity. Playwright can be integrated into a project in two ways and one of them is by using the jest-playwright which has rich features like:

• Multi-browser and device (like iPhones with given screen sizes) support.
• jest-dev-server integration which can start your webserver like create-react-app before running the tests.
• expect-playwright integration which provides common expect helper functions.

jest-playwright is added to the jest configuration as a preset which makes all the features available to be used at your disposal. It was inspired by jest-puppeteer.

Sample Jest Code

The code snippet below navigates to wikipedia.org in Chrome and checks if the correct title is being displayed.

const { chromium } = require('playwright');

describe('Wikipedia home page', async() => {
  let browser, page;
  browser = await chromium.launch();
  page = await browser.newPage();

  it('should display correct page title', async() => {
  await page.goto('https://www.wikipedia.org/');

  const title = await page.title();
  await expect(title).toBe('The Free Encyclopedia');

  await browser.close();
})

Playwright Advantages

Playwright offers some great pros such as;

• Test across all modern browsers with single API to automate Chromium, Firefox and WebKit.
• The API can be used in JavaScript & TypeScript, Python, C# and Java.
• It's simple to set up.
• Stable features.
• Bidirectional (events) – automating things like console logs is easy.
• Auto-wait for elements to be ready before executing actions (like click, fill).
• Intercept network activity for stubbing and mocking network requests.
• Seamless integration with Jest.

Playwright Disadvantages

Playwright also presents some cons such as;

• It is very new so the APIs are evolving.
• Has no support for IE11 or non-browser platforms.
• Documentations and community are not as good as the other framework yet.

To explore more on Playwright capability, you can check out the Playwright documentation and my MediaWiki Core fork.

Playwright Vs WebdriverIO

The below table is a comparison between Playwright and WebdriverIO in terms of;

• Performance
• Developer Experience
• Documentation
• Ecosystem/Community

Statistics

The above data was adopted from npm trends showing WebdriverIO and Playwright downloads. From the above charts, we see that WebdriverIO still trumps Playwright when it comes to popularity among the testing community.

MediaWiki Core Tests

The above visualized chart represents data that was collected by running MediaWiki Core tests in WebdriverIO and Playwright. A total of 10 tests were run using both frameworks on Github Actions platform.

The tests were run 40 times in both frameworks to ascertain stability and reliability. No flakiness and failures were encountered during the tests run. As shown above, there is consistency in time when the tests were run in WebdriverIO and Playwright.

The above chart further shows that Playwright is much faster than WebdriverIO in terms of speed.

Migrating From WebdriverIO

From the various evaluations done with Playwright, there seems to be great potential replacement in the future. Playwright seem to stand out in terms of both speed and stability. However, the question being asked is, is it worth migrating from WebdriverIO to another test automation framework? Let’s look at the advantages and disadvantages of a potential migration.

Advantages

• WebdriverIO is JavaScript based and is built over Node.js just like Playwright thus making it easier for developers to stick to the same programming language in the face of a potential switch to Playwright.
• The MediaWiki Core test implementation follows the Page Object Pattern, this would make it easy to write Playwright tests using the existing structure.
• Playwright has a number of useful features that come with it like the reset helper functions and single API to automate Chromium, Firefox and WebKit, among others. Some of these features are missing in WebdriverIO thus making Playwright a better option for a switch.

Disadvantages

• Investment in WebdriverIO tool. Most tests are already written in WebdriverIO. About 30 MediaWiki repositories use it, so migrating from it to another tool would require a great deal of time and manpower.
• Update of documentation. Most documentation is already written and has references to WebdriverIO. Updating the documentation would require some time.
• WebdriverIO incorporates Puppeteer as second automation driver tool allowing for the extra features which pretty much make up for some of the features that Playwright has, hence no much point in switching to Playwright.

What Next

• Refactor existing tests.
• Evaluate Puppeteer.

Summary

As of the above statistics and comparison, Playwright is a better alternative tool over WebdriverIO in terms of speed. However, WebdriverIO is a great tool given that it keeps evolving, new and modern features are being added to it and the community is still vibrant and supportive. With the recent release of WebdriverIO v7 which has some great feature update, and the fact that WebdriverIO incorporates a framework like Puppeteer as second automation driver tool makes WebdriverIO a powerful test automation framework. There is also a potential integration with Playwright in the near future. As of the time of the evaluation, there are no plans to switch to a different test automation framework but this could change in the future.

Acknowledgement

The completion of this project wouldn't have been a success without the support of my mentors, Željko Filipin, Vidhi Mody and Soham Parekh. I do appreciate the constant guidance, meetings, advice, code reviews, challenging, pair programmings, to mention but a few, that were offered during the project execution.

References

• Puppeteer, Selenium, Playwright, Cypress – how to choose? by Benjamin Gruenbaum.
• Using Jest with Playwright by Max Schmitt.
• State of JS 2020: Testing by Sacha Greif and Raphaël Benitte.
• Playwright Community.