When we implement a testing stage for our code in the CI/CD pipeline, we will need to see the results of the tests and how they are progressing on every commits. In this tutorial I am going to share about how to use Test Reports feature on AWS CodeBuild to display test results from PHPUnit. I am also going to share the AWS CDK code so you can easily replicate it to your other projects/services.

Test Reports is a feature in AWS CodeBuild launched end of last year. It is still in preview at the time I wrote this article, so it might still subject to change. UPDATE: Test Reporting is now generally available.

Before Test Reports, the easiest way to see the test results aside from reading the terminal output is probably by generating the results as HTML and upload it to S3 bucket for you to download. But it will be hard to track the trends from commits to commits. Here I am going to show how the results are visualized by Test Reports.

Requirements

To run this, you will need an AWS Account. If you have a new AWS account, the resources we are going to create here are still under the free tier. You are also going to need Docker, Docker Compose, and AWS CDK installed.

The Code

You can access the sample code I wrote on GitHub. You can clone it or fork it on your GitHub.

Running The Code Locally

To run unit test locally, you can execute

docker build -t phpunit-codebuild-test-reports .
docker run -v ${PWD}:/app phpunit-codebuild-test-reports \
    ./vendor/bin/phpunit

You will see output that the unit test succeed.

PHPUnit 9.1.1 by Sebastian Bergmann and contributors.

..                                                                  2 / 2 (100%)

Time: 342 ms, Memory: 4.00 MB

OK (2 tests, 4 assertions)

Tests Code

The two source code below shows the PHP class that we want to test and the test class itself. Class MyClass has two simple method to add two integers and to concatenate two strings.

<?php

class MyClass {

    public function method1(int $a, int $b): int {
        return $a + $b;
    }

    public function method2(string $a, string $b): string {
        return $a . $b;
    }
}

The test class MyClassTest will have two testcases, each for the method.

<?php

use PHPUnit\Framework\TestCase;

class MyClassTest extends TestCase {

    public function testMethod1() {
        $object = new MyClass();
        $this->assertEquals(2, $object->method1(1, 1), 'Test Method 1 Case 1');
        $this->assertEquals(3, $object->method1(1, 2), 'Test Method 1 Case 2');
    }

    public function testMethod2() {
        $object = new MyClass();
        $this->assertEquals('11', $object->method2('1', '1'), 'Test Method 2 Case 1');
        $this->assertEquals('12', $object->method2('1', '2'), 'Test Method 2 Case 2');
    }
}

In the PHPUnit configuration phpunit.xml.dist, we need to configure the type of the result to JUnit junit so it can be recognized by CodeBuild Test Reports. We can set the target file anywhere as long as it's consistent with what we will put in the CodeBuild configuration. I usually put it inside build directory.

<?xml version="1.0" encoding="UTF-8"?>

<phpunit bootstrap = "vendor/autoload.php">
    <testsuites>
        <testsuite name="Project Test Suite">
            <directory>tests</directory>
        </testsuite>
    </testsuites>
    <filter>
        <whitelist>
            <directory suffix=".php">src/</directory>
        </whitelist>
    </filter>
    <logging>
        <log type="junit" target="build/reports.xml"/>
    </logging>
</phpunit>

CodeBuild Configuration

CodeBuild uses Build Specification or buildspec to configure what commands to run and other settings like runtime. The buildspec.yml configuration for CodeBuild will look like the code below. We configure the location of the test report in the reports directive. We set the test report to use the generated reports file that we already configured before build/reports.xml. This test report will use JUnit XML report format.

version: '0.2'
phases:
  install:
    runtime-versions:
      php: '7.3'
    commands:
    - composer install
  build:
    commands:
    - ./vendor/bin/phpunit
reports:
  test-report:
    files:
      - build/reports.xml
    file-format: JunitXml

AWS CDK Code

By using AWS CDK we can model and track our AWS infrastructure using familiar programming language like Typescript, Javascript, Java, .NET, and Python. Because most PHP projects use Javascript, we are going to use Typescript for our AWS CDK code. It doesn't need a lot of requirements to be able to use Typescript if you already have Node installed for your PHP projects. The full CDK code can be read here since I am going to explain part per part.

In this tutorial we are not going to build whole CI/CD Pipeline using AWS CodePipeline. We are just going to use a single AWS CodeBuild. It can stand on it's own for receiving trigger events from git services like AWS CodeCommit or Github. While AWS CodeBuild support executing build commands in stages, if you are looking for full-blown pipeline that supports various AWS or third party services, you can try AWS CodePipeline.

First thing we are going to discuss is the part createSource.

createSource(): codebuild.Source {
    const secret = cdk.SecretValue.secretsManager('GITHUB_OAUTH_TOKEN');
    new codebuild.GitHubSourceCredentials(this, 'GithubCredentials', {
        accessToken: secret,
    })
    const repo = ssm.StringParameter.valueForStringParameter(this, 'GITHUB_REPO');
    const owner = ssm.StringParameter.valueForStringParameter(this, 'GITHUB_OWNER');
    const source = codebuild.Source.gitHub({
        owner: owner,
        repo: repo,
        webhook: true,
        webhookFilters: [
            codebuild.FilterGroup.inEventOf(codebuild.EventAction.PUSH).andBranchIs('master'),
        ]
    });

    return source;
}

In this method, we are setting up the Source for the CodeBuild to fetch the source code. I am going to use GitHub as I hosted the source code there. Here I do configuration when creating the source object by calling codebuild.Source.gitHub. Before deploying the infrastructure you will need to set up some configurations so the CodeBuild will be able to integrate to your repository. You will need to obtain your GitHub OAuth Token and save it securely using AWS Secrets Manager.

aws secretsmanager create-secret \
    --name GITHUB_OAUTH_TOKEN \
    --secret-string abcdefg1234abcdefg56789abcdefg

You also need to store configuration for your repository by storing GITHUB_REPO and GITHUB_OWNER to AWS Systems Manager Parameter Store.

aws ssm put-parameter \
    --name GITHUB_OWNER \
    --type String \
    --value owner

aws ssm put-parameter \
    --name GITHUB_REPO \
    --type String \
    --value repo

The webhookFilters part specify that we only accept trigger event when a code is pushed to master branch.

Secondly is the simple part createProject. This will receive the source object we created before to create CodeBuild project.

createProject(source: codebuild.Source): codebuild.Project {
    return new codebuild.Project(this, 'Build', {
        source: source,
    });
}

And the third part is addTestReportPermissionToProject. Here we need to add permission for the CodeBuild to update the test report.

addTestReportPermissionToProject(project: codebuild.IProject) {
    //"arn:aws:codebuild:your-region:your-aws-account-id:report-group/my-project-*";
    const pattern = {
        partition: 'aws',
        service: 'codebuild',
        resource: `report-group/${project.projectName}-*`
    };
    const reportArn = cdk.Arn.format(pattern, cdk.Stack.of(this));

    project.addToRolePolicy(new iam.PolicyStatement({
        resources: [
            reportArn,
        ],
        effect: iam.Effect.ALLOW,
        actions: [
            "codebuild:CreateReportGroup",
            "codebuild:CreateReport",
            "codebuild:UpdateReport",
            "codebuild:BatchPutTestCases"
        ]
    }));
}

The test report resource ARN will use format arn:aws:codebuild:<your-region>:<your-aws-account-id>:report-group/<project-name>-*. We will use cdk.Arn.format helper to build the ARN for the IAM statement.

Deploying

If you already installed AWS CDK in your environment, you can go ahead with the following command

cdk deploy

It will show permission changes. Type y when you are prompted to proceed with the change.

Do you wish to deploy these changes (y/n)? y

It will then show progress on building the resources using CloudFormation.

Result

You now can see the created CodeBuild in your console. Here you will see the CodeBuild project that we created.

If you click on the project details, you will see the build history. As soon as you push a new code in your repository, a new entry will pop up.

The build entry will first show the build output from terminal execution.

You will be able to see the report history in the Reports tab after the build finished and the reports uploaded.

The summary of the history will show the pass rate and duration for the test executed in the run. The success will be shown in green and the failures will be shown in red.

You can also see the trend from past executions.

Clean Up

To avoid additional cost incurring, you may want to remove the resources by executing the following command.

cdk destroy

Summary and What's Next?

Now we can display reports from PHPUnit execution and track the trends by using AWS CodeBuild. We also know how to create CodeBuild and integrate to GitHub repo using AWS CDK. From here, there are couple of things that we can do to improve the developer experience and security.

1. Creating Pipeline The build system we created here is pretty simple. You might want to create more sophisticated pipeline using AWS CodePipeline, for example to send notification when the test fails.
2. Adding Permission to Developers IAM Group to Browse the List If you already have an IAM Group set up for your developer teams, you may want to add permissions to the group to browse and view the test reports.

Thanks a lot for reading this article. I would love to hear feedback or ideas from you. Feel free to drop your comments in the Comment section.

References

Here are some references that you can read to understand more about the Test Reports feature.

1. Working with test reporting in AWS CodeBuild
2. Test Reports with AWS CodeBuild