Experimentations on Bazel: github-actions

In previous article Experimentations on Bazel: intro, we created a workspace/project. In this article, we'll setup github-actions to build the workspace.

Minimal config

First step, create the repository on github: https://github.com/new and push the content of the local repository into github by following instruction on the web, something like (name of the remote repository and the default branch depends of your setup):

git remote add origin git@github.com:davidB/sandbox_bazel.git 
git branch -M development
git push -u origin development

Then to enable github-action for CI, you need to define job in the file .github/worflows/ci.yml with the following content:

name: CI

on: [push, pull_request]

jobs:
  test:
    # virtual environments: https://github.com/actions/virtual-environments
    runs-on: ubuntu-20.04

    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, which is the CWD for
      # the rest of the steps
      - uses: actions/checkout@v2

      # build
      - name: Build the code
        run: bazel build //...

bazel and bazelisk are included into the provided virtual-environments (see virtual-environments/Ubuntu2004-README.md at main · actions/virtual-environments), so no need to install it. The job just need to checkout the code and run bazel.

git add .github/
git commit -m ":construction_worker: add ci.yml config for github-actions"
git push

Check the result on your actions page by example the on https://github.com/davidB/sandbox_bazel/actions this job took 23s.

You can notice in the log that bazel for the version defined into .bazelversion is downloaded.

2021/03/14 15:53:06 Downloading https://releases.bazel.build/4.0.0/release/bazel-4.0.0-linux-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading: 
Loading: 0 packages loaded
Analyzing: 0 targets (1 packages loaded, 0 targets configured)
INFO: Analyzed 0 targets (1 packages loaded, 0 targets configured).
INFO: Found 0 targets...
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
INFO: Elapsed time: 6.708s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action

Enable Cache

If you re-run the job, bazel will be downloaded again. You can improve a little the build by using cache.

name: CI

on: [push, pull_request]

jobs:
  test:
    # virtual environments: https://github.com/actions/virtual-environments
    runs-on: ubuntu-20.04

    steps:
      # Caches and restores the bazelisk download directory.
      - name: Cache bazelisk download
        uses: actions/cache@v2
        env:
          cache-name: bazel-cache
        with:
          path: ~/.cache/bazelisk
          key: ${{ runner.os }}-${{ env.cache-name }}-${{ github.ref }}
          restore-keys: |
            ${{ runner.os }}-${{ env.cache-name }}-development


      # Checks-out your repository under $GITHUB_WORKSPACE, which is the CWD for
      # the rest of the steps
      - uses: actions/checkout@v2

      # build
      - name: Build the code
        run: bazel build //...

git add .github/
git commit -m ":construction_worker: enable bazelisk cache on github-action"
git push

When you re-run the job (use the button on top-left in the page of CI jobs), no longer "Downloading https://releases.bazel.build/4.0.0/release/bazel-4.0.0-linux-x86_64..." in the output of the job.

The duration of the run increase to 26s, but the re-run now take 23s. Not a big gain (vs no-cache) because managing the cache (pre and post steps) or downloading bazel each time is equivalent. The gain is on the host of bazel archives, less load on its side (better metrics) and download of cache stay inside the github'server infrastructure.

But the using cache becomes more interesting to speed up CI by avoiding rebuild of unchanged packages. To simulate a long build, we'll create a fake long build, I'll explain it in the next article. Add the following code inside ./BUILD.bazel

genrule(
    name = "hello20s",
    outs = ["hello20s.txt"],
    cmd_bash = "sleep 20 && echo 'hello20s' >$@",
)

Then build it twice locally, the first time it will take at least 20s, and less than 1s the second time

❯ bazel build //...                                                                                                                18:07:00
INFO: Analyzed target //:hello20s (1 packages loaded, 1 target configured).
INFO: Found 1 target...
Target //:hello20s up-to-date:
  bazel-bin/hello20s.txt
INFO: Elapsed time: 20.067s, Critical Path: 20.01s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions

❯ bazel build //...                                                                                                                18:10:45
INFO: Analyzed target //:hello20s (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:hello20s up-to-date:
  bazel-bin/hello20s.txt
INFO: Elapsed time: 0.049s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

Now we'll enable the cache on github-actions an check that re-run will take care of the cache. So update .github/workflow/ci.yml with:

name: CI

on: [push, pull_request]

jobs:
  test:
    # virtual environments: https://github.com/actions/virtual-environments
    runs-on: ubuntu-20.04

    steps:
      # Caches and restores the bazelisk download directory, the bazel build directory.
      - name: Cache bazel
        uses: actions/cache@v2
        env:
          cache-name: bazel-cache
        with:
          path: |
            ~/.cache/bazelisk
            ~/.cache/bazel
          key: ${{ runner.os }}-${{ env.cache-name }}-${{ github.ref }}
          restore-keys: |
            ${{ runner.os }}-${{ env.cache-name }}-development

      # Checks-out your repository under $GITHUB_WORKSPACE, which is the CWD for
      # the rest of the steps
      - uses: actions/checkout@v2

      # build
      - name: Build the code
        run: bazel build //...

git add BUILD.bazel
git commit -m ":alembic: add target hello20s"
git add .github/
git commit -m ":construction_worker: enable bazel cache on github-action"
git push

🎉 first run in 45s, and re-run in 23s, the rule 'hello20s' was not executed the previous built output was used.

Do not forgot test

Currently the project doesn't include test, so the CI will failed if we run bazel test \\... on it, but we can prepare CI to run the future test.

name: CI

on: [push, pull_request]

jobs:
  test:
    # virtual environments: https://github.com/actions/virtual-environments
    runs-on: ubuntu-20.04

    steps:
      # Caches and restores the bazelisk download directory, the bazel build directory.
      - name: Cache bazel
        uses: actions/cache@v2
        env:
          cache-name: bazel-cache
        with:
          path: |
            ~/.cache/bazelisk
            ~/.cache/bazel
          key: ${{ runner.os }}-${{ env.cache-name }}-${{ github.ref }}
          restore-keys: |
            ${{ runner.os }}-${{ env.cache-name }}-development

      # Checks-out your repository under $GITHUB_WORKSPACE, which is the CWD for
      # the rest of the steps
      - uses: actions/checkout@v2

      # build
      - name: Build the code
        run: bazel build //...
      # - name: Run the test
      #   run: bazel test //...

git add .github/
git commit -m ":construction_worker: prepare to run test in CI"
git push

share the cache

Currently the cache is setup per git ref, meaning that each branch, PR, tag will have its own cache. But bazel cache has a good handle to check if cached content should be rebuild or not based on inputs like a hash(command + files). So you can also avoid PR and tag to rebuild everything by removing -${{ github.ref }} from the cache name.

name: CI

on: [push, pull_request]

jobs:
  test:
    # virtual environments: https://github.com/actions/virtual-environments
    runs-on: ubuntu-20.04

    steps:
      # Caches and restores the bazelisk download directory, the bazel build directory.
      - name: Cache bazel
        uses: actions/cache@v2.1.4
        env:
          cache-name: bazel-cache
        with:
          path: |
            ~/.cache/bazelisk
            ~/.cache/bazel
          key: ${{ runner.os }}-${{ env.cache-name }}

      # Checks-out your repository under $GITHUB_WORKSPACE, which is the CWD for
      # the rest of the steps
      - uses: actions/checkout@v2

      # build
      - name: Build the code
        run: bazel build //...
      # - name: Run the test
      #   run: bazel test //...

git add .github/
git commit -m ":construction_worker: share the CI cache between gitref"
git push

Keep the github-actions'version uptodate

Setup the dependabot service to create PR on project to keep uptodate the version of the action by adding the file .github/dependabot.yml

version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"

git add .github/
git commit -m ":construction_worker: setup dependabot to update version of github-actions"
git push

Conclusion

We setuped github-actions to run bazel with cache to speed up result. We also added the first rule in ./BUILD.bazel, next article will explain the rule and it will start digging into genrule.

The sandbox_bazel is hosted on github (not with the same history, due to errors), use tag to have the expected view at end of article: article/1_github-actions.

Blog