Github actions

Out of the box self-hosted actions runners are static so must be run on their own VM or something like a StatefulSet in kubernetes. This makes them quite costly. Wouldn't it be nice if they spun up per commit, ran a single job, and cleaned up afterwards?

In the next few posts I'm going to attempt to build a system that does this. This first post will run step by step through how I initially dockerized a self-hosted action.

What makes an action runner?

Self-hosted action runners can be registered against a given repo by following the instructions given in the Settings -> Actions menu in a repo. In short, the instructions are:


$ mkdir actions-runner && cd actions-runner
$ curl -O -L https://github.com/actions/runner/releases/download/v2.164.0/actions-runner-linux-x64-2.164.0.tar.gz
$ tar xzf ./actions-runner-linux-x64-2.164.0.tar.gz
$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}
$ ./run.sh

√ Connected to GitHub

2020-02-01 21:01:15Z: Listening for Jobs

Note that ${TOKEN} here and in the rest of this post refers to the actions runner registration token which you receive from the UI.

This will start a self-hosted action runner which will listen for new commits and run actions configured for those commits. Both the config.sh and run.sh scripts call out to a .Net core binary which does the real work.

This is great, now you can run actions on your own infrastructure. If you work in a large corporation this is a huge deal, as generally build tools and other resources your builds may need are only accessible from internal networks. However, there are a few issues with how these self-hosted actions run:

If you have many repositories - let's say you have a private github organization - you now have at the very least a VM, kubernetes pod, or some resource which must be constantly running. If your organization has 100+ repos this can be quite costly resource-wise.
It's not straightforward to add new actions runners for new repos that are created.
It's also not straightforward to scale these actions if a repo gets a lot of commits daily.
The scripts which are run in the setup instructions require some manual intervention (at least by default, more on that later) so automating it may be tricky.

The question is, can we fix these issues? Let's start by abstracting the setup, and create a docker image to do so.

What is currently automatable?

Before we begin to create an image for the runner, let's go through the setup steps and see what currently needs manual intervention.

Initial download

The initial download is fully automatable, no manual intervention necessary:

$ cd /var/tmp
$ mkdir actions-runner && cd actions-runner
$ curl -s -O -L https://github.com/actions/runner/releases/download/v2.164.0/actions-runner-linux-x64-2.164.0.tar.gz
$ tar xzf ./actions-runner-linux-x64-2.164.0.tar.gz

We should have a directory with the following contents:

$ ls -la
total 73680
drwxr-xr-x  4 chaospie chaospie     4096 Dec 18 20:41 ./
drwxrwxrwt 12 root     root         4096 Feb  1 20:13 ../
-rw-rw-r--  1 chaospie chaospie 75400617 Feb  1 20:14 actions-runner-linux-x64-2.164.0.tar.gz
drwxr-xr-x  2 chaospie chaospie    16384 Dec 18 20:41 bin/
-rwxr-xr-x  1 chaospie chaospie     2671 Dec 18 20:40 config.sh*
-rwxr-xr-x  1 chaospie chaospie      623 Dec 18 20:40 env.sh*
drwxr-xr-x  4 chaospie chaospie     4096 Dec 18 20:41 externals/
-rwxr-xr-x  1 chaospie chaospie     1666 Dec 18 20:40 run.sh*

Configuring the runner

Let's configure our runner.

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration

Enter the name of runner: [press Enter for sky]

It seems configuration wants manual input. Let's see if anything is mentioned in the help for config.sh:

 ./config.sh --help

Commands:,
 ./config.sh          Configures the runner
 ./config.sh remove   Unconfigures the runner
 ./run.sh             Runs the runner interactively. Does not require any options.

Options:
 --version  Prints the runner version
 --commit   Prints the runner commit
 --help     Prints the help for each command

Nothing. It doesn't look there is a parameter for setting the name of the action runner. We can cheat for now:

$ echo my-runner | ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration

Enter the name of runner: [press Enter for sky]
√ Runner successfully added
√ Runner connection is good

# Runner settings

Enter name of work folder: [press Enter for _work]
√ Settings Saved.

Great, a fully automated config setup! Note this also defaults the second manual input request, the work folder, to _work.

Running the runner

There are no params or manual input requests for run.sh, so this is clearly automatable.

Quick test in a docker container

Let's do a quick test in a docker container:

$ docker run -ti --rm ubuntu:18.04 bash

$ mkdir actions-runner && cd actions-runner

$ curl -O -L https://github.com/actions/runner/releases/download/v2.164.0/actions-runner-linux-x64-2.164.0.tar.gz
bash: curl: command not found

# Ah! There is no curl in the default ubuntu 18.04 image, let's just install it for now.

$ apt update && apt install -y curl
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
...
done.

$ tar xvf actions-runner-linux-x64-2.164.0.tar.gz
./
./bin/
./bin/System.Security.Cryptography.OpenSsl.dll
./bin/System.Memory.dll
./bin/System.Runtime.Serialization.Json.dll
./bin/System.IO.Compression.FileSystem.dll
...

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}
Must not run with sudo

It seems we can't run config.sh as root, or with sudo! This makes sense, generally you shouldn't run builds with an elevated user. It seems like there will be a bit of work to get this containerized.

Dockerizing a runner

Let's create a simple Dockerfile to fix this:

FROM ubuntu

ENV RUNNER_VERSION=2.164.0

RUN useradd -m actions \
    && apt-get update && apt-get install -y wget

RUN cd /home/actions && mkdir actions-runner && cd actions-runner \
    && wget https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
WORKDIR /home/actions/actions-runner
USER actions

Build and run:

$ docker build -t actions-image .
Sending build context to Docker daemon  70.66kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...

$ docker run -ti --rm actions-image

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}
Libicu's dependencies is missing for Dotnet Core 3.0
Execute ./bin/installdependencies.sh to install any missing Dotnet Core 3.0 dependencies.

A new error! Seems we are missing dependencies. Conveniently the actions runner setup gives us a script to set everything up. Let's update our Dockerfile:

FROM ubuntu

ENV RUNNER_VERSION=2.164.0

RUN useradd -m actions \
    && apt-get update && apt-get install -y wget

RUN cd /home/actions && mkdir actions-runner && cd actions-runner \
    && wget https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz

WORKDIR /home/actions/actions-runner

# Here we install the dependencies needed by the runner
RUN /home/actions/actions-runner/bin/installdependencies.sh

USER actions

Build and run again:

$ docker build -t actions-image .
Sending build context to Docker daemon  70.66kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...

$ docker run -ti --rm actions-image

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}
touch: cannot touch '.env': Permission denied
./env.sh: line 36: .path: Permission denied
Unhandled exception. System.UnauthorizedAccessException: Access to the path '/home/actions/actions-runner/_diag' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at GitHub.Runner.Common.HostTraceListener..ctor(String logFileDirectory, String logFilePrefix, Int32 pageSizeLimit, Int32 retentionDays)
   at GitHub.Runner.Common.HostContext..ctor(String hostType, String logFile)
   at GitHub.Runner.Listener.Program.Main(String[] args)
./config.sh: line 79:    44 Aborted                 (core dumped) ./bin/Runner.Listener configure "$@"

Ah, more errors! This time permissions issues. If we check the permissions on the files in that directory we can see the owner is wrong. The owner of all these files should be the actions user:

$ ls -la
total 73676
drwxr-xr-x 4    1001     115     4096 Dec 18 20:41 ./
drwxr-xr-x 1 actions actions     4096 Feb  1 20:43 ../
-rw-r--r-- 1 root    root    75400617 Dec 18 20:44 actions-runner-linux-x64-2.164.0.tar.gz
drwxr-xr-x 2    1001     115    16384 Dec 18 20:41 bin/
-rwxr-xr-x 1    1001     115     2671 Dec 18 20:40 config.sh*
-rwxr-xr-x 1    1001     115      623 Dec 18 20:40 env.sh*
drwxr-xr-x 4    1001     115     4096 Dec 18 20:41 externals/
-rwxr-xr-x 1    1001     115     1666 Dec 18 20:40 run.sh*

Another Dockerfile update:

FROM ubuntu

ENV RUNNER_VERSION=2.164.0

RUN useradd -m actions \
    && apt-get update && apt-get install -y wget

RUN cd /home/actions && mkdir actions-runner && cd actions-runner \
    && wget https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
WORKDIR /home/actions/actions-runner

# Here we change owner to user actions on the actions user's home directory
RUN chown -R actions ~actions && /home/actions/actions-runner/bin/installdependencies.sh

USER actions

And finally run again:

$ docker build -t actions-image .
Sending build context to Docker daemon  70.66kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...

$ docker run -ti --rm actions-image

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration

Enter the name of runner: [press Enter for 0ccb204b3990] my-runner

√ Runner successfully added
√ Runner connection is good

# Runner settings

Enter name of work folder: [press Enter for _work]

√ Settings Saved.

$ ./run.sh

√ Connected to GitHub

2020-02-01 21:22:24Z: Listening for Jobs

Great! Now let's make it generic for any repository. To do that, let's create an entrypoint.sh as follows:

#!/usr/bin/env bash

OWNER=$1
REPO=$2
TOKEN=$3
NAME=$4

echo ${NAME} | ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}

./run.sh

And update our Dockerfile:

FROM ubuntu

ENV RUNNER_VERSION=2.164.0

RUN useradd -m actions \
    && apt-get update && apt-get install -y wget

RUN cd /home/actions && mkdir actions-runner && cd actions-runner \
    && wget https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
WORKDIR /home/actions/actions-runner

RUN chown -R actions ~actions && /home/actions/actions-runner/bin/installdependencies.sh

USER actions

# Add the script and make it the entrypoint
COPY entrypoint.sh .
ENTRYPOINT ["./entrypoint.sh"]

Let's build and test it:


$ docker run -ti --rm actions-image
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"./entrypoint.sh\": permission denied": unknown.

Woops! I forgot to make the script executable. Make the script executable to fix:

$ chmod +x entrypoint.sh

$ docker build -t actions-image .
Sending build context to Docker daemon  70.66kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...
Step 9/9 : ENTRYPOINT ["./entrypoint.sh"]
 ---> Using cache
 ---> a6535399773a
Successfully built a6535399773a
Successfully tagged actions-image:latest

$ docker run -ti --rm actions-image ${OWNER} ${REPO} ${TOKEN} my-runner

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration

Enter the name of runner: [press Enter for a05fbb6da3db]
√ Runner successfully added
√ Runner connection is good

# Runner settings

Enter name of work folder: [press Enter for _work]
√ Settings Saved.


√ Connected to GitHub

2020-02-01 21:28:51Z: Listening for Jobs

Great, it all works as expected! There are a few problems however:

You need to manually run a container for each runner you wish to start.
Each runner you start needs to be manually cleaned up.
Any runner that is started can take any job for a new commit if it's idle, so cleaning up runners has the potential to accidentally kill builds that are in progress.
The registration token is a pain to retrieve. Recently github released an API to generate this token. In the next post we will automate the retrieval of this token.
In the current state runners will not unregister, meaning you must do this from the UI.

The code up to this point can be seen here.

Conclusion

In this post we have created a very simple docker image for automating self-hosted action runner registration. In the next post we will improve this by looking into the actions runner source code which contains many hidden parameters that will allow us to automate a lot more than we have so far.

Blog

Ephemeral Self-Hosted Github Actions Runners

wayofthepie

Table of Contents