TheLazy.Dev
Portfolio GitHub

Blog

github |docker |bash

Hacking Together A GitHub Actions Runner Orchestrator

wayofthepie

wayofthepie

Posted on February 2, 2020

Hacking Together A GitHub Actions Runner Orchestrator

Table of Contents

In the last post, we built a docker image that allows us to connect and disconnect actions runners when we want. But we need to do this manually when we need one.

In this post, I'm going to hack together an orchestrator that launches an actions runner per commit, with bash and cron!

Run a single build and cleanup

Right now when we run a container and connect an actions runner, it will stay alive until we manually kill it, building any new commits. It would be nicer if it ran a single build then cleaned itself up.

Back in the previous post we found the Runner.Common Constants.cs. I mentioned that there were some other interesting pieces of code in that class, outside of what we were looking for at the time. One of those is the CommandLine Flags class. This contains the help option, which looks like it maps to config.sh --help. But it also contains a once option.

Digging about a bit more and seeing where this is used uncovers a variable called RunOnce in Runner.Listener CommandSettings.

...

public bool RunOnce => TestFlag(Constants.Runner.CommandLine.Flags.Once);

...
Enter fullscreen mode Exit fullscreen mode

This looks promising. It is most likely an option we can pass to ./run.sh. Let's find out by updating entrypoint.sh!

#!/usr/bin/env bash

OWNER=$1
REPO=$2
PAT=$3
NAME=$4

cleanup() {
    token=$(curl -s -XPOST -H "authorization: token ${PAT}" \
        https://api.github.com/repos/${OWNER}/${REPO}/actions/runners/registration-token | jq -r .token)
    ./config.sh remove --token $token
}

token=$(curl -s -XPOST \
    -H "authorization: token ${PAT}" \
    https://api.github.com/repos/${OWNER}/${REPO}/actions/runners/registration-token | jq -r .token)

./config.sh \
    --url https://github.com/${OWNER}/${REPO} \
    --token ${token} \
    --name ${NAME} \
    --work _work

# add the --once option here
./run.sh --once

cleanup
Enter fullscreen mode Exit fullscreen mode

Rebuild and launch the runner:

$ docker run -ti --rm actions-image ${OWNER} ${REPO} ${PAT} ${NAME}

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.


√ Connected to GitHub

2020-02-02 18:05:11Z: Listening for Jobs
Enter fullscreen mode Exit fullscreen mode

For the next steps to work, you will need to have an action workflow set up, here is an example:

name: CI

on: [push]

jobs:
  build:

    runs-on: self-hosted

    steps:
    - uses: actions/checkout@v2
    - name: Run a one-line script
      run: echo Hello, world!
    - name: Run a multi-line script
      run: |
        echo Add other actions to build,
        echo test, and deploy your project.
        sleep 10
Enter fullscreen mode Exit fullscreen mode

Add this to the repository at .github/workflows/ci.yml. It should appear under the Actions tab in the repo. Now you can add a new commit to the repo, and your connected action runner should start the workflow:

...
2020-02-02 18:06:42Z: Running job: build
2020-02-02 18:07:00Z: Job build completed with result: Succeeded

# Runner removal

√ Runner removed successfully
√ Removed .credentials
√ Removed .runner
Enter fullscreen mode Exit fullscreen mode

Awesome! It ran the action workflow on the new commit, exited, and cleaned up. Exactly what we wanted. Now we can build a simple orchestrator.

A simple orchestrator

I'm going to keep this dead simple. Here is the workflow I am thinking of:

  • Every minute check for new commits
  • If a new commit is detected, launch a runner

That's it! This has a few problems, for example if there are multiple new commits this will just launch a single runner which will build one commit and the other commits may never be built. However, if we get this flow working we can improve as we go.

Detecting new commits

The simplest way I can think of detecting new commits is by using the /repos/:owner/:repo/commits API. This returns a lot of info, all we need is the SHA1 hash of the last commit. For example:

$ curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/wayofthepie/gh-app-test/commits \
    | grep sha \
    | head -n1 
"sha": "e0654f66fcb8061a439e404f1bf1b2c8b7116537",
Enter fullscreen mode Exit fullscreen mode

Simple! We can make this a bit cleaner with jq:

curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/wayofthepie/gh-app-test/commits \
    | jq -r .[0].sha
e0654f66fcb8061a439e404f1bf1b2c8b7116537
Enter fullscreen mode Exit fullscreen mode

jq -r .[0].sha says get me the first object in an array, and get the value of the sha field from that. Give to me as a raw string (not wrapped in quotes).

With this, we can write a shell script (orc.sh) that detects new commits and launches a runner:

#!/usr/bin/env bash
##
# Detects new commits in a given repo by checking every minute
# and storing a reference to the last seen commit in a file called
# prev.
##

PAT=$1
OWNER=$2
REPO=$3
prev=prev

# make sure we have values for all our arguments
[ -z ${PAT} ] || [ -z ${OWNER} ] || [ -z $REPO ] && {
    echo "Incorrect usage, example: ./orc.sh personal-access-token owner some-repo"
    exit 1
}

# make sure the prev file exists
touch ${prev}

# get the latest commit
latest_commit=$(curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/${OWNER}/${REPO}/commits |\
    jq -r .[0].sha)

# read the last commit we saw
prev_commit=$(cat ${prev})

echo "Latest commit is ${latest_commit}"
echo "Previous commit is ${prev_commit}"

# compare the commits, running an action runner if they differ
if [ "${latest_commit}" != "${prev_commit}" ]; then
    echo "Detected new commit, starting runner"
    docker run -d --rm actions-image \
        ${OWNER} \
        ${REPO} \
        ${PAT} \
        $(uuidgen)
fi

# update the previous commit store with the latest commit
echo ${latest_commit} > ${prev}
Enter fullscreen mode Exit fullscreen mode

To test it, create a new commit in the repo you are connection your actions runners to and run this script against that repo, e.g.:

$ ./orc.sh ${PAT} ${OWNER} ${REPO}
Latest commit is 4ceb832f5386c812c0a296968197a1d242f8e8fd
Previous commit is c7e9b463a90a4c5eeab16ba9ccf2afdc1b28153b
Detected new commit, starting runner
68101fbf5c801afaa378ebbb6dc988c5e264fa784bc2c1353025f125f56bbc22
Enter fullscreen mode Exit fullscreen mode

Awesome! To see the container:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
68101fbf5c80        actions-image       "./entrypoint.sh way…"   26 seconds ago      Up 25 seconds                           adoring_perlman
Enter fullscreen mode Exit fullscreen mode

And the logs:

$ docker logs -f adoring_perlman

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.


√ Connected to GitHub

2020-02-02 18:38:25Z: Listening for Jobs
Enter fullscreen mode Exit fullscreen mode

An issue with our process

Hmm it doesn't seem to be starting any jobs, and if we look at the repo we see that the check has failed!

Alt Text

The reason for this:

Alt Text

This is a problem. It seems you always need at least one self-hosted runner running, or commits will initially fail. For now we can manually kick off the checks again from the UI with the Re-run checks button, and our runner should start the job.

Alt Text

$ docker logs -f adoring_perlman
...
2020-02-02 18:38:25Z: Listening for Jobs
2020-02-02 18:42:37Z: Running job: build
2020-02-02 18:43:00Z: Job build completed with result: Succeeded

# Runner removal


√ Runner removed successfully
√ Removed .credentials
√ Removed .runner
Enter fullscreen mode Exit fullscreen mode

Given this issue, we need to change tactics!

Changing tactics

From a bit of testing it seems we can get around this issue by configuring a runner that we never actually run. Let's update the entrypoint script to allow registration only:

#!/usr/bin/env bash

OWNER=$1
REPO=$2
PAT=$3
NAME=$4

# if set this script will only run ./config.sh
# it will not run the actions runner
REGISTER_ONLY=$5

cleanup() {
    token=$(curl -s -XPOST -H "authorization: token ${PAT}" \
        https://api.github.com/repos/${OWNER}/${REPO}/actions/runners/registration-token | jq -r .token)
    ./config.sh remove --token $token
}

token=$(curl -s -XPOST \
    -H "authorization: token ${PAT}" \
    https://api.github.com/repos/wayofthepie/gh-app-test/actions/runners/registration-token | jq -r .token)

./config.sh \
    --url https://github.com/${OWNER}/${REPO} \
    --token ${token} \
    --name ${NAME} \
    --work _work

if [ -z ${REGISTER_ONLY} ]; then
    ./run.sh --once
    cleanup
fi
Enter fullscreen mode Exit fullscreen mode

The code up to this point can be found here. Now let's build:

$ docker build -t actions-image .
Sending build context to Docker daemon    105kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...
Successfully built f82508c0ebb8
Successfully tagged actions-image:latest
Enter fullscreen mode Exit fullscreen mode

And run, but this time run it so that it only configures and does not stay waiting for commits:

$ docker run -ti --rm actions-image ${OWNER} ${REPO} ${PAT} test true

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.
Enter fullscreen mode Exit fullscreen mode

We are setting the fifth parameter to true here, this will in turn set the ${REGISTER_ONLY} variable in our entrypoint.sh script, meaning ./run.sh will never be called. Now if we go to the UI and Settings -> Actions in our repo, we should see an offline action called test:

Alt Text

Great! Now let's commit to our repo again, and hopefully the check run will be queued and will not have failed:

Alt Text

Awesome! Finally let's spin up a runner to actually run the build:

$ ./orc.sh ${PAT} ${OWNER} ${REPO}
Latest commit is 1fb4c5fd212e8aba177a15b6016c68b7c005b74a
Previous commit is
Detected new commit, starting runner
4ecebf7f0a5fdd75c216d490ef67afa7a9d3e0c3b30848c150b148238eccbd7c

$ docker logs suspicious_ardinghelli -f

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.


√ Connected to GitHub

2020-02-02 19:14:33Z: Listening for Jobs
2020-02-02 19:14:36Z: Running job: build
2020-02-02 19:14:54Z: Job build completed with result: Succeeded

# Runner removal


√ Runner removed successfully
√ Removed .credentials
√ Removed .runner
Enter fullscreen mode Exit fullscreen mode

Woop! All working again. Now that's working again, let's add some improvements to orc.sh and add a scheduler.

Adding cron

Let's get orc.sh running every minute with cron. Just a small update first so that it writes the prev file in the directory itself is located in:

#!/usr/bin/env bash
##
# Detects new commits in a given repo by checking every minute
# and storing a reference to the last seen commit in a file called
# prev.
##

PAT=$1
OWNER=$2
REPO=$3

# the directory of this script 
cur_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
prev="${cur_dir}/prev"

# make sure we have values for all our arguments
[ -z ${PAT} ] || [ -z ${OWNER} ] || [ -z $REPO ] && {
    echo "Incorrect usage, example: ./orc.sh personal-access-token owner some-repo"
    exit 1
}

# make sure the prev file exists
touch ${prev}

# get the latest commit
latest_commit=$(curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/${OWNER}/${REPO}/commits |\
    jq -r .[0].sha)

# read the last commit we saw
prev_commit=$(cat ${prev})

echo "Latest commit is ${latest_commit}"
echo "Previous commit is ${prev_commit}"

# compare the commits, running an action runner if they differ
if [ "${latest_commit}" != "${prev_commit}" ]; then
    echo "Detected new commit, starting runner"
    docker run -d --rm actions-image \
        ${OWNER} \
        ${REPO} \
        ${PAT} \
        $(uuidgen)
fi

# update the previous commit store with the latest commit
echo ${latest_commit} > ${prev}
Enter fullscreen mode Exit fullscreen mode

I took the cur_dir var from this post: https://stackoverflow.com/questions/59895/how-to-get-the-source-directory-of-a-bash-script-from-within-the-script-itself.

Latest code up to this point is here.

To add orc.sh to cron, first move it to /var/tmp (we can pick a better location later) and then run cronab -e to edit the crontab for your user:

$ cp orc.sh /var/tmp
$ crontab -e 

# Edit this file to introduce tasks to be run by cron.
#
...
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
* * * * * /var/tmp/orc.sh ${PAT} ${OWNER} ${REPO}
Enter fullscreen mode Exit fullscreen mode

You can make sure it is running by checking the cron logs, it should run every minute:

$ journalctl -u cron -f
-- Logs begin at Sat 2019-08-10 18:05:10 IST. --
...
Feb 02 19:30:01 sky CRON[31363]: (chaospie) CMD (/var/tmp/orc.sh ${PAT} ${OWNER} ${REPO} )
Enter fullscreen mode Exit fullscreen mode

Now if you commit, a container should be created and run your actions! Note that because orc.sh did not know about any previous commit it will always launch a container on its initial run.

Conclusion

We now have a super simple way of launching actions runners per commit. However the way we are doing this has a lot of of problems:

  • orc.sh will launch a container when it initially runs, whether there is something to build or not.
  • We launch a runner per commit, if there are multiple workflows defined in a repo we will only ever build one and the others will lay idle. We can fix this by using information from the checks API to figure out when to launch a runner and how many we may need.
  • The cron checks every minute for new commits, we can do better than this. For example we could use webhooks so the builds run in realtime.
  • In my testing I have seen some cases where a runner fails to be cleaned up and sometimes you cannot even remove them from the UI. When this has happened I've had to register a new runner with the same name and then remove that.
  • Instead of bash + cron it would be better to create a webservice for this, especially if we are going down the webhooks route.
  • This can currently only run on a single machine and isn't really scalable as is.

In the next post I'll tackle some of these issues. The code up to this point can be found here.

💖 💪 🙅 🚩
wayofthepie
wayofthepie

Posted on February 2, 2020

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Hacking Together A GitHub Actions Runner Orchestrator
github Hacking Together A GitHub Actions Runner Orchestrator

February 2, 2020

Improving our github actions runner image
github Improving our github actions runner image

February 2, 2020

Ephemeral Self-Hosted Github Actions Runners
github Ephemeral Self-Hosted Github Actions Runners

February 1, 2020