Why visual regression testing

Your app code changes quite frequently - so does the UI. The problem with this flow is that sometimes unintentional UI changes result from incorrect code. We often forget to check the deep corners of the application that users might see for every potential regression; moreover, we don't have time for that, right 😅 Ultimately as with other testing types the goal is to ship with more confidence and o*wn the changes you make to the code*!

| Luckily, we can quickly establish automation that can help us with that

In this tutorial we will learn how to:

• setup a new next.js app
• use lost-pixel to enable CI visual regression testing

At this time, you might wonder what visual regression tests even are.

On top are two snapshots taken before/after a code change, below is how the engine compares those two to identify the visual regression.

Without further ado, let's jump into the code, and don't worry, as we will delve into some low-level dev-ops you don't need any specific knowledge, we will figure everything on the fly.

For the final code you could check out this repo

Next setup

This tutorial covers the most basic Next.js app, so you can follow official docs for the simple instructions for setting it up. First, let's add the *app page * and move the content from the index there so we have a clear path we will test!

Adding Lost Pixel

So far, our setup has the frontend application & the tooling to observe the frontend components/pages. However, to have complete control over the changes to the code and their repercussions on the UI, we use an open-source visual regression testing tool called Lost Pixel.

To set up the first tests, you need to add the lost-pixel.config.js or lost-pixel.config.ts at the root of your project by running one of the below commands:

npx lost-pixel init-js
or
npx lost-pixel init-ts

Don't forget to include lostpixel.config.ts file into your tsconfig when using typescript version of config so it's picked up correctly. npm install lost-pixel -D is as well required for the types.

This will generate the config that powers Lost Pixel. As we are setting up the open-source version which persists the snapshots of your components in your repo, you need to add the generateOnly key into your config - this will ensure that lost-pixel will not try to upload the images to the cloud, failOnDifference flag will exit the GitHub action with the error. Here is how your config should look like:

import { CustomProjectConfig } from 'lost-pixel';

export const config: CustomProjectConfig = {
  pageShots: {
    pages: [
      { path: '/app', name: 'app', id: 'app' },
    ],
    pageUrl: "http://172.17.0.1:3000",
  },
  generateOnly: true,
  failOnDifference: true
};

Note the 172.17.0.1 IP address, which is specific to the GitHub actions run.

GitHub actions setup

With our current setup, it's pretty straightforward to enable the visual-regression testing on the CI. We will use GitHub actions that have native integration with lost-pixel for this example. Still, lost-pixel also being a docker container, it's possible to integrate it into the CI platform of your choice.

Let's get to business!

In .github/workflows/test.yml

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x
          cache: 'npm'

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v2.15.0

Let's see what this workflow file does in a little more detail:

on: [push] // we want to execute this action on every push to our repository

jobs:
  build: // we set up one build job 
    runs-on: ubuntu-latest // it runs on ubuntu

    steps: // it will execute the steps outlined below
      - name: Checkout
        uses: actions/checkout@v2 // get the code of our repo

      - name: Setup Node
        uses: actions/setup-node@v2
        with:
          node-version: 16.x
          cache: 'npm' // get node to work in our process 

      - name: Install dependencies
        run: npm ci // install dependencies defined in package.json

      - name: Build Next app
        run: npm run build // build next.js application

      - name: Run Next app
        run: npm run start & // run next.js application in the background mode

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v2.15.0 // run lost-pixel visual regression tests

As you commit this code, we will see that GitHub has already picked the job and is executing the flow!
We don't yet have anything in .lostpixel baselines folder which means the action will fail and notify us that there are missing baselines. You can learn more about baselines and flow to update them here. So let's add another workflow to make sure we always have an easy way to update baselines!

In .github/workflows/test.yml

on: push

jobs:
  lost-pixel-update:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: "16"

      - name: Install dependencies
        run: npm ci --legacy-peer-deps

      - name: Build Next app
        run: npm run build

      - name: Run Next app
        run: npm run start &

      - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v2.15.0
        env:
          LOST_PIXEL_MODE: update

      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v4
        if: ${{ failure() && steps.lp.conclusion == 'failure' }}
        with:
          token: ${{ secrets.GH_TOKEN }}
          commit-message: update lost-pixel baselines
          delete-branch: true
          branch: "lost-pixel-update/${{ github.ref_name }}"
          title: "Lost Pixel update - ${{ github.ref_name }}"
          body: Automated baseline update PR created by Lost Pixel

You must have already guessed that this workflow file is not much different from the previous one with one little catch - it will automatically create a new PR. Don't forget to add the update mode env var so we have our baselines updated!

    - name: Lost Pixel
        uses: lost-pixel/lost-pixel@v2.11.0
        env:
          LOST_PIXEL_MODE: update // <--- this one here is really important for updater action to run

For this exercise we need a personal access token from github. To ensure that the action has access to the GitHub token which you, by the way, should never expose to the public - we will place it in the secrets of the repo.

Let's run the updater action:

If there are no baselines it will prompt a PR creating the Lost Pixel baselines, if there are existing baselines and there were some differences found it will as well prompt a new PR. In this case we see that there is one open PR with the new baseline. Let's verify it looks good and merge the pull request!

After PR merge we can see that Lost Pixel has ran the tests and all of them are passing 🍏🍏🍏

Amazing job, you now have visual regression testing running on your CI/CD pipeline 🚀

Closing thoughts

Whenever the Lost Pixel fails on CI, you need to decide if the change was intentional or not, which you could read up here. If it has been a deliberate change - you need to run update your baselines on CI. Another strategy could be generating the artefacts from the failed run and putting those manually into .lostpixel/baselines folder!

The implementation is finished, you could try making some changes to your /app page, like changing fonts, layout and color - as we have visual regression tests setup, we will be able to catch all the changes and ship with more confidence.

For the final code you could check out this repo as previously mentioned!