Running applications on different operating systems and processor architectures is a common scenario, so it is a common practice to build separate distributions for different platforms. This is not easy to achieve when the platform we use to develop the application is different from the target platform for deployment. For example, developing an application on an x86 architecture and deploying it to a machine on an ARM platform usually requires preparing the ARM platform infrastructure for development and compilation.

Problem

I recently purchased the new Apple M1 Pro MacBook, which is a high-performance laptop. The M1 Pro (aarch64) processor is very fast and very friendly on the battery, but as I mentioned there are also some - not so small - issues associated with working on a new chip architecture.

I’ve tried to create a few multi-platform docker images with buildx and some are just not correctly emulated on the M1.
The parallelism of buildx was also an issue but the emulation gave me the most trouble.

Solution

Use buildx remote node(s) to build the image on a native architecture.

What do you need?

• More than one docker-enabled device in your network or on the internet like:
  • Virtual Machine / Server
  • Raspberry Pi
  • Laptop
  • Desktop
• ssh access to these devices

In my case I have a CX11 VM on HETZNER that is **amd64/x86_64** based and my own M1 Pro that is **arm64/aarch64** based. Since Hetzner now offers arm64 machines, you can also do it the other way around too.

How I did do this?

Step 1: Installation

Since the newer versions of Docker, Buildx (Buildkit) is a built-in function, to check its availability, use the following command.

❯ docker buildx version
github.com/docker/buildx v0.11.1 b4df08551fb12eb2ed17d3d9c3977ca0a903415a

Step 2: Setup ssh access for your remote node

The remote host where we also want to run Docker needs to be configured with a password-less SSH connection. This post will not explain how to do that as there are many articles explaining how to set up public/private key access to a device.

If you already have the keys you can copy them to your target device like so:

ssh-copy-id USERNAME@TARGET_VM_HOST

To run a command with the user environment available you need to make sure your sshd service allows it.
When running a command through ssh you will have a limited environment and that needs to be adjusted.

To have a correct environment where docker is known you need to set the #PermitUserEnvironment no property to PermitUserEnvironment yes in the /etc/ssh/sshd_conf file on your VM.
When this is adjusted you need to restart the sshd service. On my CX11 VM with the Ubuntu server, I used this command. It might be different for your device:

sudo systemctl restart sshd

Now you have to set the environment for ssh:

• login to your VM through the terminal (ssh USER_HERE@TARGET_VM_HOST)
• cd ~/.ssh create a file called environment with the following value in it

PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

the last /usr/local/bin is where the docker command lives. So now it will be available.

Check if you can run docker with the command below from local machine/laptop. It should work:

docker -H ssh://USERNAME_HERE@TARGET_VM_HOST info

Do not forget to change the command's values to appropriate ones for your situation.

Step 3: Create a buildx local node

Let’s first create the local platform. In my case that is the Apple M1 Pro node. It should build all the arm64/aarch64 targets.

docker buildx create \
  --name local_remote_builder \
  --node local_remote_builder \
  --platform linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/mips64le,linux/mips64,linux/arm/v8,linux/arm/v7,linux/arm/v6 \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000

I’ve played a bit with the settings but this seems to work best for me. I will update as needed or if I find settings that work even better.

Now we have a local config for a builder called local_remote_builder.

You can check if you have it by running docker buildx ls command:

NAME/NODE   DRIVER/ENDPOINT     STATUS PLATFORMS
local_remote_builder * docker-container
    local_remote_builder unix:///var/run/docker.sock running linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386

Step 4: Create a target buildx remote node

In my case, this will be the amd64/x86_64 builder node. The platforms corresponding to that architecture should be sent there.
We have already configured the SSH access and the environment and tested that docker can access it.

Now we need to add it to the buildx target

docker buildx create \
    --name local_remote_builder \
    --append \
    --node intelarch \
    --platform linux/amd64,linux/386 \
    ssh://USERNAME@TARGET_VM_HOST \
    --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 \
    --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000

Do not forget to change the command's values to appropriate ones for your situation.

This command will append the remote node to the local_remote_builder and call the node intelarch. You can check it again by running docker buildx ls command again:

NAME/NODE              DRIVER/ENDPOINT        STATUS  BUILDKIT             PLATFORMS
local_remote_builder * docker-container                                    
  local_remote_builder orbstack               running v0.11.6              linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386
  intelarch            ssh://aboozar@167.21.183.24 running v0.11.6              linux/amd64*, linux/386*, linux/amd64/v2, linux/amd64/v3

Step 5: buildx use and bootstrap

In order to start using this builder setup we need to tell buildx to start using it and we need to bootstrap it.

docker buildx use local_remote_builder
docker buildx inspect --bootstrap

output:

#1 [intelarch internal] booting buildkit
#1 starting container buildx_buildkit_intelarch
#1 ...

#2 [localremote_builder internal] booting buildkit
#2 pulling image moby/buildkit:buildx-stable-1 1.6s done
#2 creating container buildx_buildkit_local_remote_builder 0.6s done
#2 DONE 2.3s

#1 [intelarch internal] booting buildkit
#1 starting container buildx_buildkit_intelarch 3.6s done
#1 DONE 3.6s
Name:   localremote_builder
Driver: docker-container

Nodes:
Name:      local_remote_builder
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/arm64*, linux/riscv64*, linux/ppc64le*, linux/s390x*, linux/mips64le*, linux/mips64*, linux/arm/v7*, linux/arm/v6*, linux/amd64, linux/amd64/v2, linux/386

Name:      intelarch
Endpoint:  ssh://aboozar@167.21.183.24
Status:    running
Platforms: linux/amd64*, linux/386*, linux/amd64/v2

Step 6: Build a multi-platform image

The command below will build for amd64 and arm64 but will direct the amd64 build to the remote node and the arm64 build will be done on the local machine.

docker buildx build --platform=linux/amd64,linux/arm64/v8 --push -t name/target:tag .

This will result in something like this on the docker hub:

Conclusion

The most obvious issue in this setup is that I can only build for multiple platforms when within my own network.
That is not necessarily true if you used a remote ssh accessible host or IP. I did not do that and that limits me to these builds when I have my remote node available in the network. For now, that is not a real issue for me.

It solved my concurrency problem. I had one build where I had to start the server, in order to configure it, during the docker build. Buildx does this in parallel and that gave me a port conflict as one was a bit faster but not ready when the other also tried to start. That was a problem when I tried to build all on only my local node. When I added the remote node this problem went away as the port was used on a completely different machine.

Overall, I am currently content with this solution.