I maintain several GitHub Actions, all of which are implemented in Python as container actions. This post explains how to test a GitHub Action using a GitHub Actions workflow, including using the workflow as a required check on Pull Requests. Although some of this post is specific to testing an action that is implemented in Python, much of the post is more generally applicable to testing actions regardless of implementation language.
Table of Contents: The rest of this post is organized as follows:
In this section, I'll walk you through a GitHub Actions workflow for testing a GitHub Action.
Preliminaries
First, within the .github/workflows directory of the repository, create a YAML file for the workflow. I usually name this workflow build.yml. Start by giving the workflow a name, and configuring the events that will trigger it to run. In this example, the workflow will run on both pushes and pull requests for the branch main. The snippet below also sets up a job that will run on an Ubuntu runner.
Our first set of tests are our unit tests. Although I implement actions as container actions, I conduct the unit testing outside of the Docker container. Since this is a Python example, we need two steps here. First, the actions/setup-python action is used to set up Python. In this example, Python 3.10 is used. The second step below uses the Python module unittest to execute our unit tests. This example assumes that there are unit tests in tests/tests.py. If any of the unit tests fail, then the failure will cause a non-zero exit code from Python, which will in turn fail the workflow. In this way, we can use this as a required PR check to ensure that all of our unit tests pass before merging a PR.
-name:Setup Pythonuses:actions/setup-python@v4with:python-version:'3.10'-name:Run Python unit testsrun:python3 -u -m unittest tests/tests.py
Build the Docker Container
Next, since this is a container action, we want to ensure that the Docker container successfully builds.
-name:Verify that the Docker image for the action buildsrun:docker build . --file Dockerfile
Integration Test
Earlier, we ran unit tests external from any container. Now, we want to test the full integration of our action. For this, we want to use the action itself. Ordinarily, you run an action with uses: username/repository@version. However, we don't want to do that here. If we want to use this workflow as a PR check, we want to make sure that we run a version of the action that incorporates any changes from the PR we are checking. We can do this by specifying uses: ./ which will direct the GitHub Actions framework to look for an action at the root of what we earlier checked out with the actions/checkout step.
In this example, I'm running the action we are testing twice with different inputs. You can have as many of these steps as needed. Keep in mind that each test here will be slower than each of your unit tests. After all, each of these is fully running the action, rather than simply testing one small unit; and additionally, the GitHub Actions framework must also build your Docker container.
-name:Integration test 1uses:./with:input-one:somethinginput-two:true-name:Integration test 2uses:./with:input-one:something elseinput-two:false
Validate the Integration Test Results
We now need a way to detect if the results of the above integration tests are correct. The various actions that I maintain produce files (e.g., jacoco-badge-generator produces coverage badges, and generate-sitemap produces an XML sitemap) or edits existing files (e.g., javadoc-cleanup inserts canonical links and a few other things into the head of javadoc pages). In cases like these, I use Python's unittest module to validate the results. In this case, I define unit test cases in tests/integration.py that verify that the files produced by the action are correct. If any of those tests fail, then Python will exit with a non-zero exit code which will cause the workflow to fail.
-name:Verify integration test resultsrun:python3 -u -m unittest tests/integration.py
Complete Example Workflow
Here's the complete example.
name:buildon:push:branches:[main]pull_request:branches:[main]jobs:build:runs-on:ubuntu-lateststeps:-uses:actions/checkout@v3-name:Setup Pythonuses:actions/setup-python@v4with:python-version:'3.10'-name:Run Python unit testsrun:python3 -u -m unittest tests/tests.py-name:Verify that the Docker image for the action buildsrun:docker build . --file Dockerfile-name:Integration test 1uses:./with:input-one:somethinginput-two:true-name:Integration test 2uses:./with:input-one:something elseinput-two:false-name:Verify integration test resultsrun:python3 -u -m unittest tests/integration.py
Real Examples
I use this approach in several actions that I maintain. I maintain a website about these.
Features information on several open source GitHub Actions for workflow automation that we have developed to automate parts of the CI/CD pipeline, and other repetitive tasks. The GitHub Actions featured include jacoco-badge-generator, generate-sitemap, user-statistician, and javadoc-cleanup.
actions.cicirello.org
If you'd like to see a couple real examples of my approach to testing a GitHub Action within GitHub Actions, then take a look at the following three examples.
Testing the generate-sitemap Action
This first real example is generate-sitemap. Files relevant to the example are as follows:
The generate-sitemap GitHub action generates a sitemap for a website hosted on GitHub
Pages, and has the following features:
Support for both xml and txt sitemaps (you choose using one of the action's inputs).
When generating an xml sitemap, it uses the last commit date of
each file to generate the <lastmod> tag in the sitemap entry. If the file
was created during that workflow run, but not yet committed, then it instead uses
the current date (however, we recommend if possible committing newly created files first).
Supports URLs for html and pdf files in the sitemap, and has inputs
to control the included file types (defaults include both html and pdf files in the sitemap).
Now also supports including URLs for a user specified list of
additional file extensions in the sitemap.
The jacoco-badge-generator can be used in one of two ways: as a GitHub Action or as a command-line
utility (e.g., such as part of a local build script). The jacoco-badge-generator parses a jacoco.csv
from a JaCoCo coverage report, computes coverage percentages
from JaCoCo's Instructions and Branches counters, and
generates badges for one or both of these (user configurable) to provide an easy
to read visual summary of the code coverage of your test cases. The default behavior directly
generates the badges internally with no external calls, but the action also provides an option
to instead generate Shields JSON endpoints. It supports
both the basic case of a single jacoco.csv, as well as multi-module projects in which
case the action can produce coverage badges from the combination of…
The javadoc-cleanup GitHub action is a utility to tidy up javadocs prior to deployment to
an API documentation website, assumed hosted on GitHub Pages. It performs the following
functions:
Improves mobile browsing experience: It
inserts <meta name="viewport" content="width=device-width, initial-scale=1"> within the <head> of
each html file that was generated by javadoc, if not already present. Beginning with Java 16, javadoc
properly defines the viewport, whereas prior to Java 16, it does not.
Strips out any timestamps inserted by javadoc: The timestamps cause a variety of version control
issues for documentation sites maintained in git repositories. Javadoc has an option -notimestamp to
direct javadoc not to insert timestamps (which we recommend that you also use). However, at the present
time there appears to be a bug (in OpenJDK 11's javadoc, and possibly other versions)…
Vincent A. Cicirello - Professor of Computer Science at Stockton University - is a
researcher in artificial intelligence, evolutionary computation, swarm intelligence,
and computational intelligence, with a Ph.D. in Robotics from Carnegie Mellon
University. He is an ACM Senior Member, IEEE Senior Member, AAAI Life Member,
EAI Distinguished Member, and SIAM Member.
actions.cicirello.org
cicirello.org