How to setup an automated testing pipeline with Codecov and GitHub Actions.

Outline

Scope of the tutorial
Introduction
Prerequisites (optional)
Code coverage with Codecov
Project Setup
Codecov & GitHub Actions
Codecov in action
Metadata (badges)
Conclusion
Helpful links

Scope of the tutorial

This tutorial teaches how to integrate unit testing into a development workflow with Github Actions & Codecov.

Non-scope: This tutorial does not teach about unit testing with PHP.

Introduction

Unit testing in software engineering is a development practice in which individual components (functions, modules, classes, etc) of a software system are tested independently as single units to validate that they function as they're intended to by the software engineer. This practice is very important because it safeguards the software against future breaking changes by acting as an "alerting system" of some sort that notifies the software engineers involved, once the tests are executed. Usually, unit testing is combined with other forms of automated testing such as integration testing, acceptance testing, etc.

Automated testing, in general, is very important, but it's also crucial that tests are written to account for all (or most) possible edge cases that exist. Hence, we are faced with an important question: how do we measure this metric?

In this tutorial, we will learn about code coverage and how to set up an automated testing pipeline with Codecov & GitHub Actions, using PHP for demonstration.

Prerequisites (optional)

The following prerequisites are optional and not required to understand this tutorial:

Knowledge of PHP and automated testing with PHPUnit.
PHP, MySQL & XDebug installed locally.
Composer installed locally.

Code coverage with Codecov

Code coverage in software engineering is an automated testing metric that measures the percentage of lines of code that are covered by the written tests. This gives an insight into the extent to which your codebase is tested. Code coverage is usually determined by a number of criteria such as function coverage, edge coverage, statement coverage, and the likes (helpful link at the end of this tutorial).

Codecov is a tool that helps to measure code coverage and automatically generates reports based on the result obtained. It comes with features that help to enforce a minimum code coverage percentage in your project and can be integrated with a number of Continuous Integration (CI) tools.

In the sections below, we would learn how to set up Codecov in a software development project using GitHub Actions as our CI tool.

Project setup

The sample project is a simple API built with vanilla PHP and is framework agnostic. Clone the project from this repo and follow the guidelines detailed in the Readme file for the local development setup.

The structure of the project is as shown below:

php_crud_app
├─ .env
├─ .github
│  ├─ PULL_REQUEST_TEMPLATE.md
│  └─ workflows
│     └─ test.yml
├─ .gitignore
├─ api
│  ├─ AlternativeUserRepository.php
│  ├─ DbConnection.php
│  ├─ UserController.php
│  └─ UserRepository.php
├─ bootstrap.php
├─ codecov.yml
├─ composer.json
├─ composer.lock
├─ envLoader.php
├─ phpunit.xml
├─ public
│  └─ index.php
├─ README.md
├─ setup.sql
└─ tests
   └─ Feature
      └─ UserTest.php

Codecov & GitHub Actions

In order to setup Codecov with GitHub Actions, the following secret tokens are required:

GitHub Personal Access Token (PAT): This is a secret token unique to every GitHub account, which allows its owners to perform actions that require authentication such as repository-actions (clone, push & pull), setting up C pipelines with GitHub Actions, etc.
Codecov token: This is required for granting GitHub Actions access to upload the coverage metrics to Codecov.

Creating a GitHub PAT

To create a personal access token for GitHub, follow the steps below:

Step 1: Login to your GitHub account and click on "settings"

Step 2: Click on Developer settings

Step 3: Click on Personal access tokens

Step 4: If you haven't created a PAT before, click on 'Create token'. Else, choose an existing PAT and click on "Edit token".

Step 5: Select the checkbox labeled "workflows" and click on "Update token"

That's it. Ensure you save the PAT in a secure location and do not expose it to anyone.
Next, we will set up Codecov.

Setting up Codecov for the project

In order to setup Codecov for the project, follow the steps below:

Step 1: Log in to Codecov with your GitHub account.

Step 2: On the Codecov tab, click on "Analytics" and click on "Add new repository"

Step 3: Scroll down to the repo of your choice and click on "setup repo"

Step 4: Copy the "Upload token" and keep it safe. We would use it soon.

Creating repo secrets

We need to save the GitHub PAT and Codecov token earlier generated as repo secrets in order for GitHub Actions to have access to it.

To create a new secret for the repo, follow the steps below:

Step 1: Navigate to the target repo on GitHub and click on the Settings tab

Step 2: Click on "New repository secret"

Step 3: Name the GitHub PAT as "MY_PAT", paste its value, and click on "Add secret" to save. Follow the same steps for "CODECOV_TOKEN".

Setting up an automated testing pipeline with GitHub Actions

Now that we've successfully generated the required PAT and Codecov tokens, we can set up the testing pipeline.

In the root of your project folder, create the directory: .github/workflows. Next, create a file in the workflows directory called test.yml. Copy the contents below and paste them into the file:

name: Test Suite
on: [push]

jobs:
  test:
    runs-on: ubuntu-18.04
    steps:
      - name: Checkout to code repo
        uses: actions/checkout@v2
        with:
          token: ${{ secrets.MY_PAT }}

      - name: Setup PHP with Xdebug
        uses: shivammathur/setup-php@v2
        with:
          php-version: "7.4"
          coverage: xdebug

      - name: Update dependencies in lock file
        run: composer update

      - name: Install dependencies
        run: composer install

      - name: Run tests
        run: composer test

      - name: Upload Code Coverage to Codecov
        uses: codecov/codecov-action@v2
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          file: ./coverage.xml
          flags: unit_tests
          name: codecov-umbrella
          fail_ci_if_error: true
          verbose: true

Let's try to break down critical aspects of the test.yml file:

on "push": This line instructs GitHub Action to run every time a commit is pushed to the repo
Setup PHP with XDebug: This section installs PHP with XDebug, which is a PHP extension that is critical to generating the coverage report file labeled "coverage.xml". For projects written in other programming languages, this section of the CI pipeline is usually where necessary setup for running the unit tests is done.
Run tests: This is the section in which the command required for running unit tests is inserted.
Upload Code Coverage to Codecov: This section uploads the coverage report to Codecov, which is generated after running unit tests.

Codecov config file

The Codecov config file controls critical aspects of Codecov features to be applied in your project, which is available through the Codecov bot.

In your project's root directory, create a file called codecov.yml and past the content below:

comment:
  layout: "reach, diff, flags, files"
  behavior: default
  require_changes: false
  require_base: no
  require_head: yes

coverage:
  range: "70..100"
  round: down
  precision: 2
  status:
    project:
      default:
        # basic
        target: 60%
        threshold: 0%
        # advanced settings
        if_ci_failed: error
        informational: false
        only_pulls: false

This config file controls important features such as pull request comment, target coverage percentage amongst others. For a more detailed understanding of this file, check out the official documentation.

Codecov in action.

For every pull request made, the GitHub Actions pipeline runs for every commit pushed to the repo, and a coverage report is commented by the Codecov bot, as shown below:

The pipeline is currently failing because the overall code coverage for the project is 0.00%, and we set the minimum code coverage to be 60%, from the codecov.yml file. This is a result of the lack of adequate tests in the project.

Once adequate unit tests have been added to the project, and the minimum code coverage percentage is met, the pipeline passes as demonstrated below:

This is how the summary looks like on the Codecov dashboard:

With that, we've successfully set up the unit testing pipeline for a project.

Metadata (badges)

Displaying badges on your project's Readme is a good way to track metrics (code coverage, CI status) at a glance.
Let's add badges for Codecov and GitHub Actions.

Codecov badge

To add a Codecov badge, visit the link: https://codecov.io/gh/<your-organisation>/<your-project>/settings/badge.

As an example, for the sample project in this tutorial, the link will be: https://codecov.io/gh/olorondu-emeka/php_crud_app/settings/badge.

Copy the markdown link and paste it into your Readme file.

GitHub Actions badge

Toadd a badge for GitHub Actions, follow this template: ![pipeline-name](https://github.com/<your-organisation>/<your-project>/actions/workflows/<pipeline-file>/badge.svg).

For example, the link for the GitHub badge for this project is:
![Github Actions](https://github.com/olorondu-emeka/php_crud_app/actions/workflows/test.yml/badge.svg).

The final result on the ReadMe file is as shown below:

Conclusion

So far, we've seen the importance of unit testing. Using a continuous integration tool (such as GitHub Actions) and Codecov is definitely an important step in ensuring code quality in any project, as the process is fully automated and feedback via metrics(badges and codecov comment) is displayed for every commit made.

Blog