I got a chance recently working on a UI test automation project which is on top of the Google puppeteer. It's a tool highly customizable for developers to control the behavior of Google Chromium browser. The details of puppeteer can be found here.

We have a Google Chrome web extension project and the tests before each release a series of tests will be performed by interns. We have tried to maintain 100% coverage by executing the unit tests for the project. However, the integration tests like hitting APIs and monitor how the web extension UI behaves according to web pages or websites the web extension is on is still done manually. Indeed, manual tests are still important. But, we sometimes found out that the data collected by interns through manual tests are not that accurate for any kinds of reasons. We have surveyed UI test automation solutions like Selenium with WebDrivers and others but we ended up with Google puppeteer since we only support Google chrome at this stage.

The APIs of Google Puppeteer is pretty straightforward. Starting from creating a browser is very simple like codes below

const browser = await puppeteer.launch({
    headless, // if true then no UI will be shown
    ignoreDefaultArgs: true,
    ignoreHTTPSErrors: true,
    slowMo, // delay executing each actions performed on web broswer
    args, // other args
  })
return browser;

Due to the test target is the browser extension, I need to specify where the browser extension is so that the Google Chromium browser can locate the assets and then start the installation. extPath is the absolute path to the folder holding browser extensions assets.

  const args = new Array();
  args.push(`--disable-extensions-except=${extPath}`);
  args.push(`--load-extension=${extPath}`);
  const b = await puppeteer.launch({
    headless,
    ignoreDefaultArgs: true,
    ignoreHTTPSErrors: true,
    slowMo,
    args,
  });
  return b;

The codes above will show you the Google Chromium browser with the web extension icon on the top-right corner and probably open another tab landing to the website the web extension specify in menifest.json.

There are few key components composing the Google web extensions. Among which, the main UI commonly called popup.html is one of the test targets in the project. To the best of my knowledge, there no is API currently able to trigger the click event of the extension icon or button sitting on top-right corner to show the main UI which is the popup.html. However, there is a way around the issue, which is directly visiting the popup.html page. Therefore, you need to create a Page and specify the URL you want to visit. In codes below, there is a variable called extId. If you don't have this id you probably cannot load the popup.html.

  const page = await browser.newPage(); // create a new tab
  const chromeExtPath = `chrome-extension://${extId}/popup.html`; // where to find extId?
  await page.goto(chromeExtPath, { waitUntil: 'domcontentloaded', });
  if (bringToFront) await page.bringToFront();
  await page.reload();
  return page;

For the normal installation, you should find out the web extensions IDs at this directory ~/Library/Application Support/Google/Chrome/Default/Extensions if you're on the Macbook. However, you won't find any IDs in this directory related to the unpacked extensions.

➜  ls -l ~/Library/Application\ Support/Google/Chrome/Default/Extensions
total 0
drwx------@ 3 guoguang  staff  96 May 21 10:33 bmnlcjabgnpnenekpadlanbbkooimhnj
drwx------@ 3 guoguang  staff  96 May 31 12:57 chhjbpecpncaggjpdakmflnfcopglcmi
drwx------@ 3 guoguang  staff  96 May 26 13:37 djjjmdgomejlopjnccoejdhgjmiappap
drwx------@ 3 guoguang  staff  96 May 30 13:59 jifpbeccnghkjeaalbbjmodiffmgedin
drwx------@ 3 guoguang  staff  96 May 31 12:55 kbfnbcaeplbcioakkpcpgfkobkghlhen
drwx------@ 3 guoguang  staff  96 May 18 14:01 niloccemoadcdkdjlinkgdfekeahmflj
drwx------@ 3 guoguang  staff  96 May 16 10:31 nmmhkkegccagdldgiimedpiccmgmieda
drwx------@ 3 guoguang  staff  96 May 16 10:31 pkedcjkdefgpdelpbcmbmeomcjbeemfm

There are two methods address the missing extension id. First, load the unpacked extension in Google Chrome browser and also turn on Developer mode to show extension ids. However, this must go through the manual process, which is a bit problematic if you want to have a fully automated testing process. The second way is to generate the extension key and id on the fly. Following this thread on StackOverflow about how to create the key and id I have a shell script generating key and id as follows. You probably need to enlarge the key size to 4096 while creating the pem.key so as to make the web extension id and key works out for the extensions. Btw, the Google Chromium version I am using is 68.0.3419.0 Developer Build 64-bit.

  # genereate the web extension key.
  ret=$(openssl rsa -in ./extensions/key.pem -pubout -outform DER | openssl base64 -A)
  # based the key above to generate the **extension id**
  ret=$(openssl rsa -in ./extensions/key.pem -pubout -outform DER | shasum -a 256 | head -c32 | tr 0-9a-f a-p)

Not only the popup.html but how extensions interacting with web pages is also the test target. One important function of our web extension is to change its icon if a user is visiting our partners' website. In the beginning, I was trying the Message Passing to sync up testing data between web extension and web pages. I found that I need to change codes a lot in the extensions especially for the onMessage and onExtenalMesasge parts. To minimize the changes, I then used the Firebase as the medium to sync up the testing data. This probably is not the best choice but it really works for us currently.

  firebase.initializeApp(JSON.parse(data));
  firebaseDB = firebase.database();
...
  firebaseDB.ref('urlPatternTest/').update({
    uuid: testKey,
    host: '',
    icon: '',
    ts: (new Date()).getTime(),
    key: '',
  }, err => {
    if (err) reject(err);
    else resolve();
  });

The rest is about codes how we doing UI automation tests like switching tab, login/logout and etc.

async function selectTab(page, tabName) {
  const selector = 'div#buttonBox :first-child'; // querySelector
  let handle = await page.waitForSelector(selector); // blokc until the expected dom element loaded
  const text = await page.evaluate(sel => document.querySelector(sel).innerText, selector, handle); // evalute the selector and return the data
  if (text === tabName) {
    handle = await page.$(selector);
    await handle.click();
  } else {
    handle = await page.$('div#buttonBox :last-child');
    await handle.click();
  }
}
...
....
  let selector = 'input[type="email"]';
  await sgPage.waitForSelector(selector);
  const emailHandle = await sgPage.$(selector);
  await emailHandle.type(USER_EMAIL); // key in user email

  selector = 'input[type="password"]';
  const pwdHandle = await sgPage.$(selector);
  if (_.isNil(pwdHandle)) throw new TypeError('found no password input element!');
  await pwdHandle.type(USER_PWD); // key in use pwd
...

I really enjoy using Puppeteer so far since its simplicity getting UI test done with its full-fledged APIs and ways of intercepting data (HTML, image, URL, cookies and etc). Right now, we are still working on the project by adding more sophisticated test scenarios. Not sure if you will use Puppeteer in the near future. Any comments are so welcome!