Introduction

We are starting a new project at Filmin, so we are having some discussions to take decisions. We want tests to be one of the cornerstones of this project, so we needed to decide how to implement them.

However, we have not found yet a clear, unique definition of the different kind of testing strategies. What is the scope of an integration test? Should it also test the whole framework? What is the difference between acceptance tests and end-to-end tests?

As we did not have any unique guide, what we did was to have a dialogue between all the team members, interchanging our expertise and viewpoints, to get to a consensus.

These are the testing strategies we agreed on. Note that it is not meant to be an exhaustive list, neither an academic research: these are testing strategies that best suits us for the new project.

Testing strategies

We are using hexagonal architecture, so for each kind of test we will write which layers it applies to. Besides, as we are using PHP, we will write the tool we will use for that testing strategy.

Unit

Unit tests check every part of the software in isolation. Unit tests do not go beyond its scope (i.e., memory) to run,
so they use stubs for interfaces defined within the Domain Layer. Their execution must be fast.

• Who needs to understand them: developer.
• Apply to:
  • Domain layer
  • Application layer
• Tool: PHPUnit

Integration

Integration tests check that a concrete implementation of a
Domain interface works as expected —but it is not only restricted to elements in Domain.
An example could be a query to the database to check if filtering or pagination work as expected.

They usually require external infrastructure in order to run, such as a Database or an AMQP instance. Locally, these
tests will run against the local Docker instance. In CI/CD, the pipeline will prepare the required infrastructure using
and building a Docker image.

Usually we will need fixtures for the database, i.e., elements in it that we can query or act on. In that case, we will
load the fixtures querying the database directly, loading the bunch of SQL files with queries that we will need. Each test case will be wrapped in a transaction, so they will be isolated.

• Who needs to understand them: developer.
• Apply to:
  • Infrastructure layer
• Tool: PHPUnit + Docker with all infrastructure elements needed.

Functional tests

Functional tests are a "unitary" test of the whole stack (including the framework). They should check that the whole
layers (Domain -> Application -> Infrastructure) work properly together. They should check that the expected changes
in infrastructure are correctly done, for example, that a row is inserted in the database or that a message is sent to
Rabbit.

Functional tests also check the response: its status code, its possible responses —including errors—, its possible
formats —JSON, XML, Proto...

Like integration tests, functional tests require infrastructure to run. We will follow the same strategy as with
integration tests.

As integration tests, functional tests also require loading fixtures to the database previously. We will follow the
same strategy as with integration tests.

• Who needs to understand them: developer.
• Apply to:
  • The whole stack, including the framework.
• Tool: PHPUnit + Docker with all infrastructure elements needed.

We will implement functional tests for all endpoints.

Acceptance tests

Acceptance tests need context to be run and understood. They may consist on testing the same endpoint multiple times,
with different use cases. Acceptance tests check the behavior of the application, without regarding infrastructure
details. For example, we will not check the database if the tests does not say so.

These kinds of test should be able to be written using Gherkin syntax.

The difference between functional and acceptance tests is blurry, so the developer will have to decide each time the
place where the test belongs.

• Who needs to understand them: product owner.
• Apply to:
  • Application and Domain layers, if we want to test only business rules.
  • The whole stack, including the framework, if we want to test the system as a whole.
• Possible tools:
  • PHPUnit
  • Behat
  • Codeception

End-to-end tests

End-to-end tests checks user behavior, or, as we greed on, an interface. In the case the project is an API, this kind of tests will not apply.

• Who needs to understand them: user.
• Apply to:
  • The whole stack, including the framework and even the web server.
• Tools:
  • Cypress
  • Selenium
  • Panther

Benchmark tests

Benchmark tests check that a concrete functionality gives a response within an accepted threshold. For example, we
may want that a core endpoint call or function gives a response in less than 0.5 seconds.

• Who needs to understand them: developer.
• Apply to:
  • Infrastructure/Application/Domain: if we want to test how performant the code is.
  • The whole stack, including the framework and even the web server: if we want to test the whole infrastructure.
• Tools:
  • PHPBench (for code)
  • JMeter (for infrastructure)

Summary

As said, these are the kind of tests that we agreed on, not pretending to be an exhaustive list nor an academic research. Thus, it is open to future modifications or insertions, depending on what we learn.

And you? Which kind of tests do you implement in your project? Do your testing strategies agree with ours?