How to create your own Docker images that run on multiple CPU architectures.

With the recent introduction of Docker's buildx functionality it becomes possible and relatively easy for everybody to build and publish Docker images that work on multiple CPU architectures. This article focuses exclusively on Linux multi-architecture docker images, shows how to go about creating such images, and what to look out for to make it work in different host environments. It'll cover Ubuntu and Debian distributions in particular, which are used in a number of CI/CD pipelines such as Github Actions or Travis, but it's generally applicable to other Linux distributions too. You just need to make sure to check which kernel and userspace tool versions you've got.

The article assumes you're generally familiar with using Docker. If you don't know Docker yet, you can familiarize yourself with the basics with Docker's Getting Started guide. Make sure you get the Hello World example working before continuing here.

How Docker Buildx Compiles for Non-Native Architectures

Docker buildx multi-architecture support can make use of either native builder nodes running on different architectures or the QEMU processor emulator. We're only going to discuss QEMU here as it's a pure software solution that doesn't require you to have access to hosts that run on different CPU architectures.

QEMU works by simulating all instructions of a foreign CPU instruction set on your host processor. E.g. it can simulate ARM CPU instructions on an x86 host machine. With the QEMU simulator in place you can run foreign architecture binaries on your host. But to do so, you'd have to write every command with a prefix qemu-<arch> <your-command> on the command line.

Luckily, Linux also has built-in support for running non-native binaries, called binfmt_misc. Whenever Linux tries to execute a binary, it checks if there is a handler for that binary format registered with binfmt_misc. If there is, the handler is executed instead and pointed to the binary. The handler in turn executes the binary however it sees fit. An example of this is executing java byte code binaries with a JVM which interprets each java byte code. In our case we'll make use of binfmt_misc to transparently execute foreign CPU binaries with QEMU.

Software Requirements for Buildx Non-Native Architecture Support

There are several software requirements that need to be met so docker buildx can create multi-architecture images:

• Docker >= 19.03: Docker itself needs to be new enough to contain the buildx feature.
• Experimental mode for the docker CLI needs to be turned on since buildx is an experimental feature.
• Linux kernel >= 4.8: The kernel side of binfmt_misc needs to be new enough to support the fix-binary (F) flag. The fix-binary flag allows the kernel to use a binary format handler registered with binfmt_misc inside a container or chroot even though that handler binary is not part of the file system visible inside that container or chroot.
• binfmt_misc file system mounted: The binfmt_misc file system needs to be mounted such that userspace tools can control this kernel feature, i.e. register and enable handlers.
• Either a Host installation or Docker image based installation of QEMU and binfmt_misc support tools.
  • Host installation:
    • QEMU installed: To execute foreign CPU instructions on the host, QEMU simulators need to be installed. They need to be statically linked since dynamic library resolution depends on those dynamic libraries being visible in the file system at time of use, which is not typically the case inside a container or chroot environment.
    • binfmt-support package >= 2.1.7: You need to install a package that contains an update-binfmts binary new enough to understand the fix-binary (F) flag and actually use it when registering QEMU simulators.
  • Docker image based installation: You can use a Docker image that contains both QEMU binaries and setup scripts that register QEMU in binfmt_misc similar to what the binfmt-support package does.

If you happen to run on a system that has Docker Desktop >= 2.1.0 installed, e.g. on Mac OSX or Windows, you're in luck since it comes configured meeting all the above requirements. In this case you can skip the rest of this section. However, if you're running on a system where Docker Desktop is not available or installed, e.g. Linux, you'll have to install the necessary support yourself. The rest of this section assumes you're running on Linux x86. Now let's go through these requirements one by one.

Docker

Docker gained buildx support with version 19.03, so you need at least this version installed. You can check your docker version with:

$ docker --version
Docker version 19.03.5, build 633a0ea838

If you don't have docker installed on your system you can try to install it from your Linux distribution's default package sources. The package typically comes by the name of docker-ce or docker.io (see also the table of popular Linux environments below):

$ sudo apt-get install -y docker-ce
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  docker-ce
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/22.8 MB of archives.
After this operation, 109 MB of additional disk space will be used.
Selecting previously unselected package docker-ce.
(Reading database ... 120478 files and directories currently installed.)
Preparing to unpack .../docker-ce_5%3a19.03.5~3-0~ubuntu-bionic_amd64.deb ..
Unpacking docker-ce (5:19.03.5~3-0~ubuntu-bionic) ...
Setting up docker-ce (5:19.03.5~3-0~ubuntu-bionic) ...
Processing triggers for systemd (237-3ubuntu10.33) ...
Processing triggers for ureadahead (0.100.0-21) ...

It's quite possible though that the docker version that comes by default with your Linux distribution is not new enough. In that case you can add Docker's own package repository and get a newer docker version from there:

DOCKER_APT_REPO='https://download.docker.com/linux/ubuntu'
curl -fsSL "${DOCKER_APT_REPO}/gpg" | sudo apt-key add -
OS="$(lsb_release -cs)"
sudo add-apt-repository "deb [arch=amd64] $DOCKER_APT_REPO $OS stable"
sudo apt-get update
sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce

Docker Experimental Features

As of this writing (early 2020), buildx is an experimental feature. If you try to use it without turning on experimental features it'll fail:

$ docker buildx
docker: 'buildx' is not a docker command.
See 'docker --help'

You can turn on experimental Docker CLI features in one of two ways. Either by setting an environment variable

$ export DOCKER_CLI_EXPERIMENTAL=enabled

or by turning the feature on in the config file $HOME/.docker/config.json:

{
  ...
  "experimental" : "enabled"
}

If you choose the environment variable, put the setting into you shell startup script, e.g. $HOME/.bashrc for bash, otherwise the setting only sticks around in your current shell until you log out. Once you have turned on experimental features either way, you can check that it has taken effect with:

$ docker version
Client: Docker Engine - Community
 ...
 Experimental:      true
 ...

Note that this output also shows you the status of the Experimental flag of Server: Docker Engine. But this doesn't concern us for now. With experimental mode now turned on, you should have access to the docker buildx command:

$ docker buildx

Usage:  docker buildx COMMAND

Build with BuildKit

Management Commands:
  imagetools  Commands to work on images in registry

Commands:
  bake        Build from a file
  build       Start a build
  create      Create a new builder instance
  inspect     Inspect current builder instance
  ls          List builder instances
  rm          Remove a builder instance
  stop        Stop builder instance
  use         Set the current builder instance
  version     Show buildx version information

Run 'docker buildx COMMAND --help' for more information on a command.

Linux Kernel

You need a kernel that supports the binfmt_misc feature and has it enabled. In particular, the binfmt_misc support needed to use QEMU transparently inside containers is the fix-binary (F) flag which requires a Linux kernel version >= 4.8 (commit, commit). You can check your kernel version with:

$ uname -r
4.15.0

Binfmt_misc File System

The binfmt_misc kernel features are controlled via files in /proc/sys/fs/binfmt_misc/. This file system must be mounted. E.g. on a Ubuntu 18.04 (bionic) system the script responsible for mounting that file system is /lib/systemd/system/proc-sys-fs-binfmt_misc.automount which is part of the systemd package and runs automatically at boot time (and also during package installation). You can check if the file system is mounted with:

$ ls /proc/sys/fs/binfmt_misc/
register  status

Host Installation: QEMU

An easy way to install statically linked QEMU binaries is to use a pre-built package for your host Linux distribution. E.g. for Debian or Ubuntu you can install it with:

$ sudo apt-get install -y qemu-user-static
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  binfmt-support
The following NEW packages will be installed:
  binfmt-support qemu-user-static
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.1 MB of archives.
After this operation, 101 MB of additional disk space will be used.
Get:1 http://us-east-2.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 binfmt-support amd64 2.1.8-2 [51.6 kB]
Get:2 http://us-east-2.ec2.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 qemu-user-static amd64 1:2.11+dfsg-1ubuntu7.21 [10.0 MB]
Fetched 10.1 MB in 0s (22.0 MB/s)
Selecting previously unselected package binfmt-support.
(Reading database ... 120409 files and directories currently installed.)
Preparing to unpack .../binfmt-support_2.1.8-2_amd64.deb ...
Unpacking binfmt-support (2.1.8-2) ...
Selecting previously unselected package qemu-user-static.
Preparing to unpack .../qemu-user-static_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user-static (1:2.11+dfsg-1ubuntu7.21) ...
Setting up binfmt-support (2.1.8-2) ...
Setting up qemu-user-static (1:2.11+dfsg-1ubuntu7.21) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for systemd (237-3ubuntu10.33) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...

That has installed QEMU for a number of foreign architectures, e.g. 64-bit ARM (aarch64), as you can see by checking:

$ ls -l /usr/bin/qemu-aarch64-static
-rwxr-xr-x 1 root root 3621200 Oct 15 09:23 /usr/bin/qemu-aarch64-static

$ qemu-aarch64-static --version
qemu-aarch64 version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.21)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

Other Linux distributions might use different package managers or package names for the QEMU package. Alternatively you can install QEMU from source and follow the build instructions.

Host Installation: update-binfmts Tool

The update-binfmts tool is typically part of the binfmt-support package. If you look back at the installation of qemu-user-static above you'll see that it has automatically pulled in the recommended binfmt-support package, so in our case it's already installed. But if you've specified the --no-install-recommends flag (or that is set by default on your system), binfmt-support might not yet be installed. If it's missing on your system you can also install it manually with:

$ sudo apt-get install -y binfmt-support
Reading package lists... Done
Building dependency tree
Reading state information... Done
binfmt-support is already the newest version (2.1.8-2).
binfmt-support set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Here again, we need support for the fix-binary (F) flag, which was added to update-binfmts with version 2.1.7. You can check the version with:

$ update-binfmts --version
binfmt-support 2.1.8

Checking Your Host System

Putting everything together, you can check if the aforementioned environment is in place for using QEMU with docker buildx with the following check-qemu-binfmt.sh script:

Problem: QEMU Not Registered With (F) Flag

In some environments you can run into the situation that the appropriate kernel and update-binfmts support is present, but the qemu-user-static post-install script does not register QEMU with the fix-binary (F) flag. The checker script above will point that out. One such environment is e.g. AWS EC2 instances running Ubuntu 18.04 (bionic). In such a case you can fix up the installation by re-registering QEMU with the fix-binary (F) flag with the following reregister-qemu-binfmt.sh script:

Docker Image Based Installation

As an alternative to installing the QEMU and binfmt-support packages on your host system you can use a docker image to satisfy the corresponding requirements. There are several docker images that do the job, among them multiarch/qemu-user-static and docker/binfmt. They come loaded with QEMU simulators for several architectures and their own setup script for installing those QEMU simulators in the host kernel's binfmt_misc with the fix-binary (F) flag. The QEMU simulators stay registered and usable by the host kernel after running that docker image as long as the host system remains up (or you explicitly unregister them from binfmt_misc). That is what also makes them usable by later runs of docker buildx. Unlike the host installation of packages though, you'll need to re-run that docker image after every system reboot.

Using those images doesn't release you from having the right docker and kernel version on the host system, but you do get around installing QEMU and binfmt-support packages on the host. I like to use multiarch/qemu-user-static:

$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
Unable to find image 'multiarch/qemu-user-static:latest' locally
latest: Pulling from multiarch/qemu-user-static
bdbbaa22dec6: Pull complete
42399a41a764: Pull complete
ed8a5179ae11: Pull complete
1ec39da9c97d: Pull complete
df7dd9470aac: Pull complete
Digest: sha256:25d6e8bb037094525cd70da43edc06a62122028cb9ad434605affbd4fffb3a4f
Status: Downloaded newer image for multiarch/qemu-user-static:latest
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
...

Status of Popular Linux Environments

The following table shows the current status of docker buildx support on various popular Linux environments. Only Ubuntu >= 19.10 (eoan) and Debian 11 (bullseye/testing) come with sufficient support by default to be able to run docker buildx out of the box. All older versions of these Linux distributions need updates of various components in order to be compatible with docker buildx usage. For example Ubuntu 18.04 (bionic) requires re-registration of QEMU with the fix-binary (F) flag or usage of the docker image installation method for QEMU as described above as well as an upgraded docker package.

Building Multi-Architecture Docker Images With Buildx

With all the software requirements on the host met, it's time to turn our attention to how buildx is used to create multi-architecture docker images. The first step is setting up a buildx builder.

Creating a Buildx Builder

The docker CLI now understands the buildx command, but you also need to create a new builder instance which buildx can use:

$ docker buildx create --name mybuilder
mybuilder
$ docker buildx use mybuilder

You can check your newly created mybuilder with:

$ docker buildx inspect --bootstrap
[+] Building 3.5s (1/1) FINISHED
 => [internal] booting buildkit                                         3.5s
 => => pulling image moby/buildkit:buildx-stable-1                      2.9s
 => => creating container buildx_buildkit_mybuilder0                    0.6s
Name:   mybuilder
Driver: docker-container

Nodes:
Name:      mybuilder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le,
linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Note how the Platforms line reports support for various non-native architectures which you have installed via QEMU. If it only reports support for linux/amd64 and linux/386 you either still haven't met all software requirements, or you had created a builder before you have met the software requirements. In the latter case remove it with docker buildx rm and recreate it.

You can also see your just created mybuilder with buildx' ls subcommand:

$ docker buildx ls
NAME/NODE    DRIVER/ENDPOINT             STATUS  PLATFORMS
mybuilder *  docker-container
  mybuilder0 unix:///var/run/docker.sock running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
default      docker
  default    default                     running linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Using Buildx to Build

Alright, now we're ready to build multi-architecture docker images with buildx. To have something concrete to work with we're going to use the following example Dockerfile:

FROM alpine:latest
CMD echo "Running on $(uname -m)"

It's a simple stand-in for whatever you'd like to build yourself in your own Dockerfile. It uses the latest Alpine distribution - which itself is a multi-architecture docker image - and prints out the architecture on which it is executing. That will allow us to check which kind of image we're running.

The docker buildx build subcommand has a number of flags which determine where the final image will be stored. By default, i.e. if none of the flags are specified, the resulting image will remain captive in docker's internal build cache. This is unlike the regular docker build command which stores the resulting image in the local docker images list. The important flags are:

• --load: This flag instructs docker to load the resulting image into the local docker images list. However, this currently only works for single-architecture images. If you try this with multi-architecture images you'll get an export error:

$ docker buildx build ... --load ...
...
 => ERROR exporting to oci image format                                0.0s
------
 > exporting to oci image format:
------
failed to solve: rpc error: code = Unknown desc = docker
  exporter does not currently support exporting manifest lists

• --push: This flag tells docker to push the resulting image to a docker registry. Your image tag has to contain the proper reference to the registry and repository name. This currently is the best way to store multi-architecture images.

We're going to use the default Docker Hub registry. First we have to log in:

$ export DOCKER_USER='arturklauser'
$ docker login -u "$DOCKER_USER"
Password: *****
WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Now we can build and use the --push flag to push the image to Docker Hub. In our example we're going to build for three different architectures - x86, ARM, and PowerPC - which are specified with the --platform flag:

$ docker buildx build -t "${DOCKER_USER}/buildx-test:latest" \
    --platform linux/amd64,linux/arm64,linux/ppc64le --push .
[+] Building 1.5s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                    0.0s
 => => transferring dockerfile: 91B                                     0.0s
 => [internal] load .dockerignore                                       0.0s
 => => transferring context: 2B                                         0.0s
 => [linux/ppc64le internal] load metadata for docker.io/library/alpin  0.1s
 => [linux/amd64 internal] load metadata for docker.io/library/alpine:  0.2s
 => [linux/arm64 internal] load metadata for docker.io/library/alpine:  0.2s
 => CACHED [linux/ppc64le 1/1] FROM docker.io/library/alpine:latest@sh  0.0s
 => => resolve docker.io/library/alpine:latest@sha256:2171658620155679  0.0s
 => CACHED [linux/amd64 1/1] FROM docker.io/library/alpine:latest@sha2  0.0s
 => => resolve docker.io/library/alpine:latest@sha256:2171658620155679  0.0s
 => CACHED [linux/arm64 1/1] FROM docker.io/library/alpine:latest@sha2  0.0s
 => => resolve docker.io/library/alpine:latest@sha256:2171658620155679  0.0s
 => exporting to image                                                  1.2s
 => => exporting layers                                                 0.0s
 => => exporting manifest sha256:cc57b693aba3acedeec2e624b55e543919173  0.0s
 => => exporting config sha256:d90689831fec93dd68f90509031907cc262f89e  0.0s
 => => exporting manifest sha256:7997da18dbf6e4b2748736821ec6f1dd11d6e  0.0s
 => => exporting config sha256:c105b7a88a6002ad96802aa287be3872d86608a  0.0s
 => => exporting manifest sha256:6ed2267dc7082fbfc4454805b94326418ad14  0.0s
 => => exporting config sha256:5df18a6f10cc8591839d1b29732d3d202edf3c0  0.0s
 => => exporting manifest list sha256:57ca2d778839da0b6287bcbc99fc2299  0.0s
 => => pushing layers                                                   0.2s
 => => pushing manifest for docker.io/arturklauser/buildx-test:latest   0.7s

We can check the image with the imagetools subcommand which confirms that three architecture versions are included in the image:

$ docker buildx imagetools inspect "$DOCKER_USER/buildx-test:latest"
Name:      docker.io/arturklauser/buildx-test:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:57ca2d778839da0b6287bcbc99fc2299b0cea29e5fe4cff8492a1ba6e62fc8c5

Manifests:
  Name:      docker.io/arturklauser/buildx-test:latest@sha256:cc57b693aba3acedeec2e624b55e543919173a3e9ed76c2af2f2cd9713b84b78
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      docker.io/arturklauser/buildx-test:latest@sha256:7997da18dbf6e4b2748736821ec6f1dd11d6e8493f068fb23913507a9a271e2c
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

  Name:      docker.io/arturklauser/buildx-test:latest@sha256:6ed2267dc7082fbfc4454805b94326418ad14c530879a7c9d6f02f0961e2899c
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/ppc64le

Also, on the Docker Hub web site we see it reported as:

To verify that you've actually got what you've been promised, let's try to run the image:

$ docker run --rm "$DOCKER_USER/buildx-test:latest"
Unable to find image 'arturklauser/buildx-test:latest' locally
latest: Pulling from arturklauser/buildx-test
Digest: sha256:57ca2d778839da0b6287bcbc99fc2299b0cea29e5fe4cff8492a1ba6e62fc8c5
Status: Downloaded newer image for arturklauser/buildx-test:latest
Running on x86_64

As expected, since we're running on a 64-bit x86 host, the default architecture version that was used by docker was the amd64 which reports running on x86_64. If you check the local image in docker it confirms that:

$ docker inspect --format "{{.Architecture}}" "$DOCKER_USER/buildx-test:latest"
amd64

To pull and run a specific architecture version, use the image name including its full sha256 value that was reported by imagetools:

$ docker run --rm "$DOCKER_USER/buildx-test:latest@sha256:6ed2267dc7082fbfc4454805b94326418ad14c530879a7c9d6f02f0961e2899c"
Unable to find image 'arturklauser/buildx-test:latest@sha256:6ed2267dc7082fbfc4454805b94326418ad14c530879a7c9d6f02f0961e2899c' locally
sha256:6ed2267dc7082fbfc4454805b94326418ad14c530879a7c9d6f02f0961e2899c: Pulling from arturklauser/buildx-test
a5dee701e1e8: Pull complete 
Digest: sha256:6ed2267dc7082fbfc4454805b94326418ad14c530879a7c9d6f02f0961e2899c
Status: Downloaded newer image for arturklauser/buildx-test@sha256:6ed2267dc7082fbfc4454805b94326418ad14c530879a7c9d6f02f0961e2899c
Running on ppc64le

Since the sha256 value we requested here was that of the PowerPC image version, we see that the image is reporting to run on ppc64le as expected.

Optionally, we can pull and run non-native image versions by platform name. For that though we need to turn on another experimental feature, this time in the docker engine, that'll allow us to specify a --platform. If docker engine experimental features are not turned on you'll get an error instead:

"--platform" is only supported on a Docker daemon with experimental features enabled

Change the docker engine configuration file /etc/docker/daemon.json or create one if it doesn't exist already:

{
  ...
  "experimental" : true
}

After changing the configuration file you'll also need to restart dockerd for the change to take effect:

$ sudo systemctl restart docker

Let's purge the image that we've already pulled and try a different architecture:

$ docker rmi "$DOCKER_USER/buildx-test:latest"
Untagged: arturklauser/buildx-test:latest
Untagged: arturklauser/buildx-test@sha256:57ca2d778839da0b6287bcbc99fc2299b0cea29e5fe4cff8492a1ba6e62fc8c5

$ docker run --rm --platform linux/aarch64 "$DOCKER_USER/buildx-test:latest"
Unable to find image 'arturklauser/buildx-test:latest' locally
latest: Pulling from arturklauser/buildx-test
cde5963f3b93: Pull complete 
Digest: sha256:57ca2d778839da0b6287bcbc99fc2299b0cea29e5fe4cff8492a1ba6e62fc8c5
Status: Downloaded newer image for arturklauser/buildx-test:latest
Running on aarch64

Now we see that the architecture version of the image we've pulled and run is the one for 64-bit ARM aarch64, as can also be verified by looking at the image metadata:

$ docker inspect --format "{{.Architecture}}" "$DOCKER_USER/buildx-test:latest"
arm64

With this you've got to the point where you can start to build your own multi-architecture docker images with buildx.

Happy developing!

2020–01–12 - Eddies in Entropy

Appendix

The following script shows how you can use what was described above to build multi-architecture docker images in CI/CD pipelines like Github Actions or Travis.