As a software engineer, I’ve been reminded that “change” is the only “constant” we experience. Honestly, that premise is why I am just as excited today as I was when I started my career in 1992. I’ve realized there are some “almost-constants” that exist in an industry driven by change. 

In this post, I will talk about the continued mindset toward allowing less-than-ideal test coverage for features before they are released. 

I am not saying that product owners and development managers don’t care about quality. Instead, the reality of having a deep backlog of required features in a world driven by arriving to market before the competition requires choices to be made.

An agile team member might be able to ship one additional feature in the time it takes to generate an end-to-end test programmatically, which certainly impacts the bottom line. At least, that’s the mindset … until this same end-to-end test could have prevented a critical bug that results in a far greater financial impact.

I wanted to see if there was a better way to solve this quality challenge using a suite of testing tools from Tricentis.

About Tricentis

Based in Austin, TX, Tricentis creates solutions that “transform testing into a catalyst for innovation.” Their end goal is to help customers release higher-quality software, faster.

Their current suite of products includes 12 products that range from end-to-end tests for applications and data, to test automation with enterprise SaaS, Salesforce, and ServiceNow.

For this publication, I wanted to focus on the Testim product, which is focused on creating resilient UI tests for web and mobile applications in minutes.

Using the Magic 8 Ball as an example

In 2022, I decided to try the Render PaaS and created my first GO-based RESTful API. I decided to build an API that would mimic the Magic 8 Ball toy by Mattel. You can read all about it here. 

Since the focus of this article is end-to-end tests and improving quality, I decided to create a client for the Magic 8 all API using the Vue.js framework. The resulting client is very simple and contains three perspectives:

• Home – the main page for the application.

• Random Answer – simulates the Magic 8 Ball and returns a random answer.

• All Answers – provides a list of all 20 answers that were found on the original toy.

Below is a screenshot of the home page:

Here is what the Random Answer page looks like – providing the “You may rely on it.” random response:

The All Answers page appears as shown below:

The source code for the client can be found here, and you can even launch the client from the Render PaaS using the following URL:

https://magic-eight-ball-client.onrender.com/

Please note: Both the magic-eight-ball service and client are running as free instance types, so they could require a minute or two to respond if either shuts down due to inactivity.

For this publication I thought it would be fun to ensure quality with the client by adding end-to-end tests that would validate the following items:

• Validate that the Home screen responds accordingly 

• Validate that the Random Answer screen provides a response

• Validate that the “ask again” link works as expected on the Random Answer screen

• Validate that all 20 original responses exist on the All Answers screen

Adding Tricentis Into the Mix

Tricentis allows customers to set up a free account, providing 14 days of access to products of interest. I selected the Testim product for this publication.

After logging into the TestOps Dashboard, I used the New Test button, which provided an empty test panel:

I clicked on the plus icon in the main panel and selected the Start recording at this position option. The following modal appeared where I provided the publicly accessible URL for my Magic 8 Ball client hosted on Render PaaS.

Upon clicking the Create Test button, a new browser window appeared containing the Magic 8 Ball client.

At this point, I executed the validation plan for the Magic 8 Ball client application. At a high level, this translated to the following items:

• Navigate to the Random Answer screen and confirm that an expected response appeared.

• Use the “ask again” link and validate that an expected response was provided.

• Navigate to the All Answers link and confirm that all 20 original responses are listed.

• Navigate to the Home screen and validate the contents.

When I reached the end of my validation tasks, I clicked the Stop button in the lower right-hand corner of the screen. Testim immediately closed the window resulting in a test panel now full of steps that mirrored my usage of the application.

Testim made it easy to add validation for items on the screen. Using the All Answers screen, I merely had to click on the next I wanted to ensure to always be available as a step in the recording process.  Then I could update the step as noted below:

In the example above the “It is certain.” answer must always be visible from the All Answers screen – making an error and stopping the test when the text does not appear.

I saved the magic-eight-ball-client test in Testim and configured the Scheduler Library to run my test every 15 minutes.

Within a matter of minutes, I was able to provide full end-to-end coverage tests for my publicly available client application and schedule the tests to run every 15 minutes.

I decided to take a break from this effort until the scheduler had run for at least one full day.

The TestOps Dashboard

When I returned to Testim, I was quite impressed. Now, the TestOps Dashboard provides instant details for my test execution:

Out of the 95 executions, 86 passed and 9 failed. I looked at the scheduled executions, of which there were 92.

I was able to quickly zero in on the failures and realized they were directly related to the inactivity shutdown of the free instance types for my client and service on the Render PaaS. I considered changing the test interval to run every 5 minutes, but realized it was better to demonstrate what failures look like.

Testim also includes a Test Runs view, which provides a summary of all 95 test events that were initiated during my evaluation of their product.

The really cool part about using Testim is that I was able to extract all of this value just by using my application. There wasn’t a requirement to use a dedicated programming framework—with a distinct learning curve—resulting in more program code that my team would have to use and maintain. 

Testim and the CI Pipeline 

Testim also has the ability to plug into the CI pipeline process, firing the end-to-end tests at each desired step of the software development lifecycle.

Building upon this example, let’s assume that one of the software engineers decided to update a Magic 8 Ball answer from “It is certain.” to “Rock solid, yes!”

Even if a unit test missed catching this unexpected change, the simple quality tests I created to validate all the expected issues would fail, because “It is certain.” no longer exists. 

Simply including these lines to my GitLab .gitlab-ci.yml (CI pipeline) would catch change during the continuous integration process:

image: node:16.13.0
stages:
  - e2e
testim:
  stage: e2e
  image: docker:git
  variables:
    TESTIM_DOCKER: testim/docker-cli
    TESTIM_TOKEN: <TESTIM_TOKEN>
    TESTIM_PROJECT: <TESTIM_PROJECT>
    TESTIM_LABEL: <TESTIM_LABEL>
    GRID_NAME: <GRID_NAME>
  services:
    - docker:stable-dind
  script:
    - docker pull $TESTIM_DOCKER
    - docker run --rm -v "$(pwd)":/opt/testim-runner $TESTIM_DOCKER --token $TESTIM_TOKEN --project $TESTIM_PROJECT --label "$TESTIM_LABEL" --grid $GRID_NAME -r /opt/testim-runner/testim-report.xml
  artifacts:
    paths:
      - testim-report.xml
    reports:
      junit: testim-report.xml

As a result of this action, Testim would fail the CI execution because the end-to-end tests failed to pass 100%.

Imagine if there was a contract in place with Mattel that guaranteed the original 20 responses would always be used by the Magic 8 Ball client application. Not catching this unintended update could have resulted in unexpected legal consequences.

Conclusion

My readers may recall that I have been focused on the following mission statement, which I feel can apply to any IT professional:

“Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.”

- J. Vester

When I wrote this mission statement, I was thinking about a product like Testim — even though I had not even heard of Tricentis at the time. Testim adheres to my mission statement by allowing software engineers to remain focused on delivering focused features and solutions and still achieve important end-to-end test coverage.

In fact, with products like these, product owners and development managers no longer have to worry about allocating precious sprint time to understanding, creating, and maintaining code-based, end-to-end test frameworks. Rather, all they have to do is use their client-facing application with Testim to automatically create end-to-end tests that can be included in the CI process.

Have a really great day!