Why functional UI test automation is flaky

It is widely accepted that functional UI test automation helps save QAs a lot of time so that they can focus more on User Acceptant Testing and Exploratory Testing. However, there are some drawbacks that prevent many organizations and teams from successfully adopting it. The cost of writing and maintaining scripts as well as the level of flakiness are typically higher than other types of testing. That is also the reason why some people often criticize UI automation testing.

Nowadays, web application evolves really quickly with Agile-style short development cycles. Moreover, In favor of agile development and deployment, the architecture of web applications on both backend and frontend side is getting increasingly complex with multiple components separated logically and physically that are only linked together via web services. Even with a substantial amount of automated unit tests and integration tests, doing functional UI test automation is still necessary to make sure the software will behave as desired.

Therefore, instead of abandoning functional UI test automation, we had been actively looking for a way to mitigate its defects in our team, and as a result, maximize its benefits and ROI. When we take a closer analysis, we realize that there are three main causes that reduce the effectiveness of UI automation testing:

1. The time spent on writing good and stable XPath/CSS locators, and update them whenever the underlying HTML markup changes.

2. The network itself is sometimes flaky which leads to test failures.

3. The code does not handle some rare cases such as race condition.

The third cause is actually by bugs inside the software and should really be fixed by the engineering team (don’t call it flaky tests!). The second one can be mitigated by putting the application under test (AUT) and the tests in the same data center with a robust network (e.g. Amazon Web Services). Therefore, we focused on resolving the first cause.

Testable UI

To solve this problem, we need to understand why unit testing is highly robust and reliable. If you have googled around the internet on unit testing, you can see thousands of articles about writing testable code. Indeed, if the code is not structured carefully, the effectiveness of unit testing will be greatly affected, and it will take longer to write unit tests with good coverage.

Similarly, having a good XPath locator for an element is always easy even with QAs if the element were composed of unique and stable attributes. They may be made up by the developers themselves, or by the JavaScript libraries used inside the software. The list of these attributes, along with their changes, should be communicated timely between developers and QAs. We call this kind of UI “testable UI”, and the responsibility to write testable UI lies in the hand of the developers.

However, we do not let our developers writing automated UI tests. One reason is the budget — it depends on teams and companies with here we need to focus our development resources on build new features. Another reason is that QAs often do a better job in writing tests, because they focus on the users’ perspective (that’s what the word “functional” is for), while developers’ first priority is often to prove their code works (no offense!). Therefore, we came up with a very simple workflow:

1. Developers write the code.

2. They then write very simple automated tests to prove that they have covered all expected scenarios.

3. These tests are attached to corresponding Jira issues.

4. QAs write complete automated tests based on “hints” in developers submitted tests.

We call simple UI tests that developers write “functional UI unit tests”. They are very short and dead simple scripts that can be thrown away later. They should be composed without much time thinking about structure, design, coding convention etc.

This workflow brings about many benefits for both developers and QAs:

1. QAs can verify very quickly if the developers have understood and covered expected scenarios.

2. It encourages developers to make the UI testable, with all elements easy to locate. QAs can also verify the testability by taking a look at the functional unit tests.

3. The direct consequence of #1 and #2 is that the communication overhead will be reduced. QAs rarely need help from developers to resolve issues in the test scripts, or to clarify their understanding about scenarios.

4. QAs spend less time on writing XPath/CSS locators and instead can pay more attention to the stories and the test project’s structure, which should be treated as a complete and serious software project on it own.

For the next step, we need a good tool to ease this workflow.

Read the full article here