I was watching the Data from an API and I adamantly believe this to be wrong. But like all absolutes I need to clarify.

The Situation

I would like to have you get through this point in the training, but to allow you to skip I'll try to describe the situation.

He has a website which uses a card model for content. There can be several cards on the page and they each have a name and category and rarity.

He writes a test which will verify the card details are correct on the page.

Cards are provided by another API call. In the referenced chapter he demonstrates that by using the same API used by the developer we can cover all of the cards rather than write them by hand.

Breaking Things Apart

I will avoid saying this doesn't add to test coverage, but I want to explain why it provides limited coverage.

How Much Coverage

By writing this test can we say that we know the website displays all cards after the tests run?

✅ If the cards were provided at random, the test is unlikely to succeed.
❌ If a card has new content fields we are not checking it is displayed

Random

Basically any difference between the order the API provides and how the UI displays the cards runs the risk of failure.

This leaves us with needing to recreate the same logic as the developers, assuming the API is consistent and the UI is doing the sorting. I'm not saying this isn't usable to help quality, double or triple implementation has been a common approach in highly critical systems. It is common to reduce the amount of excess logic in tests (who is correct, tester or dev).

When we have a passing test we would know that none of these other considerations are true and have covered what we know.

New Card Types

He mentioned that using the API would allow tests to automatically be added as the new cards are added. This assumes we don't add to what a card can have.

If we are tracking the addition of cards we can update our test to handle these new card types. If the cards are from a third party how do you know what they are adding.

Expectations

What is actually happening here is that we are defining our expectations of a working system. However we are not being explicit about these expectations.

Mock

Wiremock as a Web Server for Testing

Jesse Phillips ・ Dec 1 '19 ・ 2 min read

#testing #mocking

I believe the card interface should be tested with mocking the card data, we test around the expectation of valid card content. Do cards handle unicode, can html be injected.

Confirm Expectations

If the card API is controlled by your development then you may have some insight into the changes occurring to cards. If you have card types mocked then these new types can be added and verified, even if the new types are available in the backed yet.

If you have little insight into the card API changes then an API confirmation could be created. The idea here is to pull the cards and verify they match the data payload expected to the current functionality. If you are really concerned about card changes you want to be alerted to not only changes to used content but unknown additions to content which might be important for the card.

Concerns

One of the primary concerns of the suggested API testing is that it creates the appearance of large coverage but can distract from finding edge cases or miss the lack of coverage due to complexity in the system.

By isolation of UI testing we can simplify the system under test. It creates another concern that the mocks aren't updated to meet the real world data. While I provided a suggestion for mitigation, reality doesn't mean you'll go that far.