Today I will tell you a story behind making [Jupyter Docker Stacks](https://github.com/jupyter/docker-stacks) able to build `aarch64` images without many (almost any) compromises. If you want to read how it is implemented now, you can skip to the last section of this post. > _Jupyter Docker Stacks are a set of ready-to-run [Docker images](https://hub.docker.com/u/jupyter) containing Jupyter applications and interactive computing tools. You can use a stack image to do any of the following (and more):_ > - _Start a personal Jupyter Server with the JupyterLab frontend (default)_ > - _Run JupyterLab for a team using JupyterHub_ > - _Start a personal Jupyter Notebook server in a local Docker container_ > - _Write your own project Dockerfile_ More info in the [documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/). --- ## First, a short dive in history Jupyter Docker Stacks images were [first introduced](https://github.com/jupyter/docker-stacks/commit/8ab2e0ed8f1fe0dd6185df7a16bb7dec8faba46e) on 2015–07–19 and consisted of three images: `minimal-notebook`, `r-notebook` and `scipy-notebook`. From the very beginning these images were not independent and `minimal-notebook` was a parent image for `r-notebook` and `scipy-notebook` images. This is how it looked like back then:

A lot of things have changed since then and this is how it looks now:

The [oldest version](https://hub.docker.com/r/jupyter/minimal-notebook/tags?page=1&ordering=-last_updated) of the `minimal-notebook` image pushed to DockerHub seems to be pushed on Oct 24, 2015. We won’t talk here about switching the CI system, adding new images and changing the base image. Instead, we will focus on how to make a set of docker images support a new architecture. The build process was based on Makefile mostly. We were simply running `docker build` commands in the right order on one machine. This is how it essentially looked: ```bash OWNER?=jupyter ALL_IMAGES:= \ minimal-notebook \ scipy-notebook build/%: docker build -t $(OWNER)/$(notdir $@):latest ./$(notdir $@) --build-arg OWNER=$(OWNER) test/%: python3 -m tests.run_tests --short-image-name "$(notdir $@)" --owner "$(OWNER)" build-test-all: $(foreach I, $(ALL_IMAGES), build/$(I) test/$(I) ) push/%: docker push --all-tags $(DARGS) $(OWNER)/$(notdir $@) ``` [Since around 2019](https://github.com/jupyter/docker-stacks/issues/899), our users were actively asking if it is possible to build ARM images as well. As of 2022, we’ve fully accomplished this. --- ## Unique aspects of our Docker images Before we get to the implementation, there are some specifics to our images, that are not widely common for other Docker images: - these images are the final product of this repository, so we have to **_test these images_** - we have many images and they **_do depend on each other_** - we **_do not build our own software_**, but these images are mostly installing 3rd party software and configuring it - we **_tag the images by the version of this software_** and we tag images a lot (for example, if we install a specific version of `spark` in the `pyspark-notebook` image, we will use the version we installed as a tag). These things make it more difficult to implement multi-arch images. --- ## The process First, we made sure our Docker images were building just fine on the `aarch64` platform. This is (not a complete) list of the things we did: - Do not inherit from image with a fixed SHA tag, this won’t work. Instead of `FROM ubuntu:focal-20210609@sha256:376209...` it became `FROM ubuntu:focal-20210609` in [this Pull Request](https://github.com/jupyter/docker-stacks/pull/1368). This is slightly less secure, but seems to be a working solution. We’ve also recently moved to simple `FROM ubuntu:focal` to get image automatically whenever we build our images. - Get rid of arch hardcode in the Dockerfiles. This is how we make it work for `micromamba` installer: ```bash RUN set -x && \ arch=$(uname -m) && \ if [ "${arch}" = "x86_64" ]; then \ # Should be simpler, see arch="64"; \ fi && \ wget "https://micromamba.snakepit.net/api/micromamba/linux-${arch}/latest" ``` - Some python packages we were installing were missing pre-built package for `linux-aarch64`. We [asked to add the missing support](https://github.com/conda-forge/bottleneck-feedstock/issues/35). - When it took a lot of time to add this support to the external package, we simply didn’t install it for the `aarch64` platform. For example, [we still only install `r-tidymodels` on `x86_64`](https://github.com/jupyter/docker-stacks/blob/bd658995bd76b4bc4e30f67d2de02c03ae432e59/r-notebook/Dockerfile#L54-L63): ```bash # `r-tidymodels` is not easy to install under arm RUN set -x && \ arch=$(uname -m) && \ if [ "${arch}" == "x86_64" ]; then \ mamba install --quiet --yes \ 'r-tidymodels' && \ mamba clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}"; \ fi; ``` - If it is not currently possible to provide the image without substantial changes, we don’t provide an `aarch64` image. For now, there is [no `aarch64` tensorflow Linux package](https://github.com/conda-forge/tensorflow-feedstock/issues/136) available on conda-forge, so we don’t provide this image. --- ## The first try I will briefly describe how we initially implemented multi-arch images and the reasons why it didn’t work well in the end. Initially, we decided to use [`docker buildx build`](https://docs.docker.com/build/) command to build the images as [this is the recommended way to build multi-platform images](https://docs.docker.com/build/building/multi-platform/). To make this work we installed QEMU in GitHub runner and enabled `buildx` builds. [This is how it looked like](https://github.com/jupyter/docker-stacks/pull/1399/files) (simplified version): ```bash OWNER?=jupyter MULTI_IMAGES:= \ base-notebook \ minimal-notebook AMD64_ONLY_IMAGES:= \ r-notebook \ scipy-notebook build-multi/%: docker buildx build $(DARGS) -t $(OWNER)/$(notdir $@):latest ./$(notdir $@) --build-arg OWNER=$(OWNER) docker buildx build $(DARGS) -t build-multi-tmp-cache/$(notdir $@):latest ./$(notdir $@) --build-arg OWNER=$(OWNER) --platform "linux/amd64,linux/arm64" build-all-multi: $(foreach I, $(MULTI_IMAGES), build-multi/$(I)) $(foreach I, $(AMD64_ONLY_IMAGES), build/$(I)) push-multi/%: docker buildx build $(DARGS) $($(subst -,_,$(notdir $@))_EXTRA_TAG_ARGS) -t $(OWNER)/$(notdir $@):latest ./$(notdir $@) --build-arg OWNER=$(OWNER) --platform "linux/amd64,linux/arm64" push-all-multi: $(foreach I, $(MULTI_IMAGES), push-multi/$(I)) $(foreach I, $(AMD64_ONLY_IMAGES), push/$(I)) ``` There were some limitations on docker buildx build (using docker/buildx 0.5.1, this is what we used back then): 1. Can’t `--load` and `--push` at the same time 2. Can’t `--load` multiple platforms What does the command `--load` mean? It means that the built image can be referenced by `docker` CLI, for example when using the `docker tag` or `docker push` commands. Workarounds due to limitations: 1. We always build a dedicated image using the current system named as `OWNER/-notebook` so we always can reference that image no matter what during tests etc. 2. Also, we always build a multi-platform image during build-multi that will be inaccessible with `docker tag` and `docker push` etc, but this will help us test the build on the different platform and provide cached layers for later. 3. We let push-multi refer to rebuilding a multi image with `--push`. We can rely on the cached layer from build-multi now even though we never tagged the multi image. There were a few things that went wrong and they were kind of expected: - We expected the builds to be slower, due to usage of QEMU (but when we started to add `aarch64` support for more images, we had around 10x slowdowns and our builds were taking ~3 hours). - We were tagging `aarch64` images as if they had the same tags as `x86_64` - We did not test `aarch64` images - We were not creating build manifests for these images The worst thing was that no one knew how to overcome these issues with this approach. There were some things that didn’t work and were unexpected at all: - There is a QEMU bug, when [child process hangs when forking due to glib allocation](https://gitlab.com/qemu-project/qemu/-/issues/285#note_919207579). As a workaround, set `G_SLICE=always-malloc` in the QEMU guest. This meant that we had to add some workarounds to our Dockerfiles. You can also take a look at the [issue, where it first appeared](https://github.com/jupyter/docker-stacks/issues/1539) and how it was mitigated in the [upstream project](https://github.com/mamba-org/mamba/issues/1611). - Some software doesn’t work properly under QEMU. We were not able to run `Julia` in an emulated environment. So, we decided to use another approach. --- ## How we did it So we decided to: - use self-hosted `aarch64` GitHub runners and don’t use QEMU at all - use simple `docker build` commands - use many GitHub jobs to achieve better concurrency - to pass the images between different jobs we use `docker save` and `docker load` commands and upload/download the corresponding files using well-made GitHub actions `actions/upload-artifact` and `actions/download-artifact` - use `aarch64` runners only for the things that must be done on `aarch64` platform. For example, we build the image, test it, compute tags and create build manifests on our self-hosted runners, but the actual tagging and pushing build manifests is done on GitHub provided runners. - heavily rely on GitHub reusable workflows and local actions. This way there is almost zero code duplication. - one particular GitHub workflow does as little as possible. It took me a few months of work, 137 commits and lots of hours reading about Docker and GitHub workflows, runners and actions. If you want to see the implementation, here is [the PR](https://github.com/jupyter/docker-stacks/pull/1703). Before we dive into the details, I would like to point out, that there are few things to know about self-hosted GitHub runners. 1. The workflows are run on the host machine by default (not inside an isolated environment), so you need to make sure, you don’t preserve an environment between different runs, _especially when workflow fails_. I recommend cleaning your environment at the start of the workflow — for example, we do `docker system prune --all --force` whenever we need to use `docker` and it’s cache from the previous runs is not desirable. 2. Each self-hosted runner installation only allows one run simultaneously. You can install several runners on one machine, if you want to, but you’ll have to deal with build caches and artefacts. 3. I would strongly suggest to require an approval for all outside collaborators. Because PRs are run in GitHub self-hosted environment as well, making them run without an approval will harm the security of your project.

We have several GitHub workflows which let us build all our images: - [`docker.yml`](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/docker.yml) — is a root workflow, it runs some jobs (which call reuseable workflows) and manages their order. This file can be seen both as config, and GitHub based build system. - [`docker-build-test-upload.yml`](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/docker-build-test-upload.yml) — this is a reusable workflow to download parent image, build a new image, test and upload it. This is the only place, where self-hosted runners are used. We also compute tags and write build manifest files in this workflow. _Note: This workflow only manipulates with one-platform image._ - [`docker-tag-push.yml`](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/docker-tag-push.yml) is used to download built images (from GitHub artifacts), tag these images (using already computed tags) and then push these images (with correct tags). _Note: This workflow only manipulates with one-platform image. We also add `arch` prefix for all the tags. For example, we push [`jupyter/base-notebook:aarch64-python-3.10.8`](https://hub.docker.com/r/jupyter/base-notebook/tags?page=1&name=aarch64-python-3.10.8) tag (and most likely, though not guaranteed [`jupyter/base-notebook:x86_64-python-3.10.8`](https://hub.docker.com/r/jupyter/base-notebook/tags?page=1&name=x86_64-python-3.10.8))._ - [`docker-merge-tags.yml`](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/docker-merge-tags.yml) — this is actually the place, where we try to create multi-platform images and merge tags. To do this, we mostly use `docker manifest` subcommands. - [`docker-wiki-update.yml`](https://github.com/jupyter/docker-stacks/blob/main/.github/workflows/docker-wiki-update.yml) downloads build manifests and pushes them to [GitHub Wiki page](https://github.com/jupyter/docker-stacks/wiki). Now you know the high-level implementation of our build process and how you can build multi-arch Docker images.

mathbunnyru

How we made Jupyter Docker Stacks multi-arch

Join Our Newsletter. No Spam, Only the good stuff.