OpenAPI is a specification for describing RESTful APIs in JSON and YAML format, aiming at being understandable by both humans and machines.

OpenAPI definitions are language-agnostic and can be used in several ways:

An OpenAPI definition can be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.

– The OpenAPI Specification

This article demonstrates how to write integration tests that compare API responses to OpenAPI 3+ definitions to validate that the former conform to the latter.

We will use the OpenAPI HttpFoundation Testing package in a fresh Laravel installation for which we'll also generate a Swagger UI documentation using the L5 Swagger package.

The first part of the post will describe the issue and provide further context – if you're just here for the code, you're welcome to skip ahead and go to the Laravel example section straight away.

The issue

APIs are commonplace nowadays and the decent ones come with documentation describing the endpoints and how to use them. These documents come in various shapes and forms, but a fact they can't escape is that they need to be updated every time the API changes.

To many developers, API documentation is an afterthought – it's boring, tedious, and often unrewarding. Using annotations to keep the code and the documentation in one place helps to some extent, but these are often painful to write and even the most willing developer is likely to introduce errors that won't necessarily be caught by reviewers.

The result is documentation and API getting out of sync, leading to frustrated consumers.

Another goal of API maintenance is to ensure that existing endpoints keep functioning properly – regressions can and will appear over time, and may go unnoticed if there isn't a proper testing strategy in place.

One approach is to write integration tests controlling the API's behaviour and automatically detecting breaking changes as soon as they are introduced. While this is a good strategy, it isn't foolproof – there's no guarantee that the expectations set in the tests will always 100% match the documentation.

How can we make sure they remain in sync?

A solution

Let's assume that we have an API documented with OpenAPI, some integration tests, and that we want to align the expectations of the latter with the operations described in the former.

OpenAPI comes with a growing number of tools built on top of it, pushing its utility far beyond the documenting aspect.

One tool destined for the PHP community and maintained by The PHP League is OpenAPI PSR-7 Message Validator, a package validating PSR-7 HTTP messages against OpenAPI definitions.

The idea is to compare HTTP requests and responses with OpenAPI definitions and return errors when they don't match.

In other words, we can use this package to add an extra layer on top of our integration tests that will compare API requests and responses with the OpenAPI definitions describing our API and fail the test if they don't match.

Here is a fancy diagram to illustrate the relationship between the API, the integration tests and the OpenAPI definition:

The definition describes the API, and the tests use the definition to make sure the API behaves the way the definition says it does.

All of a sudden, our OpenAPI definition becomes a reference for both our code and our tests – it acts as the API's single source of truth.

PSR-7

One caveat of the OpenAPI PSR-7 Message Validator package is that, as its name indicates, it only works with PSR-7 messages.

The issue is that not all frameworks support this standard by default – in fact, a lot of them use Symfony's HttpFoundation component under the hood, whose requests and responses do not implement that standard out of the box.

The Symfony folks thought of this, however, and provided a bridge that converts HttpFoundation objects to PSR-7 ones. The bridge simply needs a PSR-7 and PSR-17 factory, for which they suggest using Tobias Nyholm's PSR-7 implementation.

All these pieces are brought together by OpenAPI HttpFoundation Testing, a package allowing developers to back their integration tests with OpenAPI definitions in applications relying on the HttpFoundation component.

Let's see how to use it in a Laravel project, which falls into this category.

A Laravel example

The code featured in this section is also available as a GitHub repository for reference.

First, let's create a new Laravel 10 project, using Composer:

$ composer create-project --prefer-dist laravel/laravel openapi-example "10.*"

Enter the project's root folder and instal a couple of dependencies:

$ cd openapi-example
$ composer require --dev osteel/openapi-httpfoundation-testing
$ composer require darkaonline/l5-swagger -W

The first one is the OpenAPI HttpFoundation Testing package mentioned earlier, which we instal as a development dependency as it's intended to be used as part of our test suite.

The second one is L5 Swagger, a popular package bringing Swagger PHP and Swagger UI to Laravel. We actually don't need Swagger PHP here, as it uses Doctrine annotations to generate OpenAPI definitions and we're going to manually write our own instead. We do need Swagger UI, however, and L5 Swagger conveniently adapts it to work with Laravel.

If you're wondering, the -W option is simply here to also update related dependencies and thus avoid conflicts.

To make sure Swagger PHP doesn't overwrite the OpenAPI definition, let's set the following environment variable in the .env file, at the root of the project:

L5_SWAGGER_GENERATE_ALWAYS=false

Let's also make YAML the default format:

L5_FORMAT_TO_USE_FOR_DOCS=yaml

Create a file named api-docs.yaml in the storage/api-docs folder (which you need to create), and add the following content to it:

openapi: 3.1.0

info:
    title: OpenAPI HttpFoundation Testing Laravel Example
    version: 1.0.0

servers:
    - url: http://localhost:8000/api

paths:
    '/test':
    get:
        responses:
        '200':
            description: Ok
            content:
            application/json:
                schema:
                type: object
                required:
                    - foo
                properties:
                    foo:
                    type: string
                    example: bar

This is a simple OpenAPI definition describing a single operation – a GET request to the /api/test endpoint, that should return a JSON object containing a required foo key.

Let's check whether Swagger UI displays our OpenAPI definition correctly. Start the PHP development server (from the project's root):

$ php artisan serve

Open localhost:8000/api/documentation – you should see the OpenAPI definition rendered as a Swagger UI documentation:

Expand the /test endpoint and try it out – it should fail with a 404 Not Found error because we haven't implemented it yet.

Let's fix that now. Open the routes/api.php file and replace the example route with this one:

Route::get('/test', function (Request $request) {
    return response()->json(['foo' => 'bar']);
});

Go back to the Swagger UI tab and try the endpoint again – it should now return a successful response.

Time to write a test! Open tests/Feature/ExampleTest.php and replace its content with the following:

<?php

namespace Tests\Feature;

use Osteel\OpenApi\Testing\ValidatorBuilder;
use Tests\TestCase;

class ExampleTest extends TestCase
{
    /**
     * A basic test example.
     *
     * @return void
     */
    public function testBasicTest()
    {
        $response = $this->get('/api/test');

        $validator = ValidatorBuilder::fromYamlFile(storage_path('api-docs/api-docs.yaml'))->getValidator();

        $result = $validator->validate($response->baseResponse, '/test', 'get');

        $this->assertTrue($result);
    }
}

Let's unpack the above. If you're unfamiliar with Laravel, $this->get() is a test method provided by the MakesHttpRequests trait to perform a GET request to the provided endpoint, executing the request's lifecycle without leaving the application. It returns a response that is identical to one you would obtain if you'd perform the same request from the outside.

We then create a validator using the Osteel\OpenApi\Testing\ValidatorBuilder class, to which we feed the YAML definition we wrote earlier via the fromYamlFile static method (the storage_path function is a helper returning the path to the storage folder, where we placed the definition).

Had we had a JSON definition instead, we could have used the fromJsonFile method (note that other methods are available and listed in the documentation).

The builder returns an instance of Osteel\OpenApi\Testing\Validator, on which we call the get method, passing the path and the response as parameters ($response is an instance of Illuminate\Testing\TestResponse here – a wrapper for the underlying HttpFoundation object, which can be retrieved through the baseResponse public property).

The above is basically the equivalent of saying:

I want to validate that this response conforms to the OpenAPI definition of a GET request to the /test path.

It could also be written this way:

$result = $validator->get($response->baseResponse, '/test');

That's because the validator exposes a shortcut method for each of the HTTP methods supported by OpenAPI (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS and TRACE), to make it simpler to test responses for the corresponding operations.

Note that the specified path must exactly match one of the OpenAPI definition's paths.

You can now run the test, which should be successful:

$ php artisan test tests/Feature

Open routes/api.php again, and change the route for this one:

Route::get('/test', function (Request $request) {
    return response()->json(['baz' => 'bar']);
});

Run the test again – it should now fail because the response contains baz instead of foo, and the OpenAPI definition says the latter is expected.

Our test is officially backed by our OpenAPI definition!

The above is an oversimplified example for demonstration purposes, but in real-life situations, a good practice would be to overwrite the MakesHttpRequests trait's call method, so it performs both the request and the OpenAPI check.

This way, our test would boil down to this line:

$this->get('/api/test');

This could be implemented as a new MakesOpenApiRequests trait that would "extend" the MakesHttpRequests one, and that would first call the parent call method to get the response. It would then work out the path from the URI, and validate the response against the OpenAPI definition before returning it, for the calling test to perform any further assertions, as needed.

The OpenAPI HttpFoundation Testing package has more to offer, like extension points and caching to speed up the execution of your tests (Laravel's caching API is supported out of the box) – please refer to the documentation for details.

Conclusion

While this setup is a great step towards improving an API's robustness, it is no silver bullet – it requires that every single endpoint is covered with integration tests, which is not easily enforceable in an automated way and ultimately still requires some discipline and vigilance on the part of developers.

It may even feel a bit coercive at first since, as a result, maintainers are basically forced to keep the documentation up to date to write successful tests.

The added value, however, is that said documentation is now guaranteed to be much more accurate, leading to happy consumers which, in turn, should lead to less frustrated developers, who will spend less time hunting down pesky discrepancies.

All in all, making OpenAPI definitions the single source of truth for both the API documentation and integration tests is in itself a strong incentive to keep them up to date. They naturally become a priority, where they used to be an afterthought.

As for maintaining the OpenAPI definition itself, doing so manually can feel a bit daunting. Annotations are a solution, but I personally don't like them and prefer to maintain a YAML file directly. IDE extensions like this one for VSCode make it much easier, but if you can't bear the sight of a YAML or JSON file, you can also use tools like Stoplight Studio to do it through a more user-friendly interface.

And since we're talking about Stoplight, this article about API Design-First vs Code First by Phil Sturgeon is a good starting point for API documentation in general and might help you choose an approach to documenting that works for you.

Resources

• The OpenAPI Specification
• The OpenAPI HttpFoundation Testing package
• Laravel example repository
• Swagger UI
• The L5 Swagger Laravel package
• OpenAPI.Tools
• The OpenAPI PSR-7 Message Validator package
• The HttpFoundation component
• PSR-7: HTTP message interfaces
• The PSR-7 Bridge
• PSR-7 implementation
• API Design-First vs Code First