TheLazy.Dev
Portfolio GitHub

Blog

raspberrypi |docker |iot |virtualmachine

Running Raspberry Pi OS in a Docker Container

dokmic

Michael

Posted on November 25, 2024

Running Raspberry Pi OS in a Docker Container

Intro

Sometimes, testing your work on the Raspberry Pi OS is much easier without running it on real hardware. Things like Ansible Playbooks or a Kubernetes cluster, in most cases, can be tested in a virtualized environment.

There are plenty of tutorials and other Docker images running Raspberry Pi OS using QEMU, but unfortunately, all of them utilize the OS image at runtime as an SD card. Hence, they do not support mounting volumes to share the filesystem.

Image

The most obvious solution would be to extract the OS image and share the extracted folder with the virtual machine. Fortunately, QEMU provides such an option out of the box with the -virtfs flag:

-virtfs local,path=path,mount_tag=mount_tag ,security_model=security_model[,writeout=writeout][,readonly=on] [,fmode=fmode][,dmode=dmode][,multidevs=multidevs]
Enter fullscreen mode Exit fullscreen mode

Let's now try to download and extract all the OS files from a Raspberry Pi OS distribution:

curl https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2024-10-28/2024-10-22-raspios-bookworm-arm64-lite.img.xz \
| unxz -c - >/tmp/sd.img
Enter fullscreen mode Exit fullscreen mode

The extracted image can be inspected with the fdisk command:

$ fdisk -l /tmp/sd.img 
Disk /tmp/sd.img: 2732 MB, 2864709632 bytes, 5595136 sectors
43712 cylinders, 4 heads, 32 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Device     Boot StartCHS    EndCHS        StartLBA     EndLBA    Sectors  Size Id Type
/tmp/sd.img1    64,0,1      1023,3,32         8192    1056767    1048576  512M  c Win95 FAT32 (LBA)
/tmp/sd.img2    1023,3,32   1023,3,32      1056768    5595135    4538368 2216M 83 Linux
Enter fullscreen mode Exit fullscreen mode

As we can see, the image has two partitions. The first one, the boot partition, is of type FAT32, and the second is of type ext4.

We can mount those partitions using the loop device with mount, sfdisk, and jq in one line:

mount -o loop,offset="$(sfdisk -J /tmp/sd.img | jq '.partitiontable.sectorsize * .partitiontable.partitions[1].start')" /tmp/sd.img /tmp/sd
Enter fullscreen mode Exit fullscreen mode

Combining those all together in a Dockerfile would look something like this:

# syntax=docker/dockerfile:1.3-labs
FROM alpine:latest AS image

ADD https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2024-10-28/2024-10-22-raspios-bookworm-arm64-lite.img.xz /sd.img.xz

RUN --security=insecure \
  apk add --no-cache --virtual=.tools \
    jq \
    sfdisk \
  && unxz -ck /sd.img.xz >/tmp/sd.img \
  && mkdir -p /tmp/sd \
  && mount -o loop,offset="$(sfdisk -J /tmp/sd.img | jq '.partitiontable.sectorsize * .partitiontable.partitions[1].start')" /tmp/sd.img /tmp/sd \
  && mount -o loop,offset="$(sfdisk -J /tmp/sd.img | jq '.partitiontable.sectorsize * .partitiontable.partitions[0].start')" /tmp/sd.img /tmp/sd/boot/firmware \
  && cp -pr /tmp/sd /media/sd \
  && umount /tmp/sd/boot/firmware /tmp/sd \
  && rm -rf /tmp/sd /tmp/sd.img \
  && apk del .tools
Enter fullscreen mode Exit fullscreen mode

Mounting volumes requires a privileged mode. That is why we need the --security=insecure flag after the RUN instruction, which is only available in the labs specification.

Kernel

Now, if we try to run the extracted image using QEMU:

qemu-system-aarch64 \
  -serial mon:stdio \
  -nographic \
  -no-reboot \
  -machine virt \
  -cpu cortex-a72 \
  -m 1G \
  -smp 4 \
  -device virtio-net-device,netdev=net0 \
  -netdev user,id=net0 \
  -kernel /media/sd/boot/firmware/kernel8.img \
  -virtfs local,id=boot,mount_tag=boot,multidevs=remap,path=/media/sd/boot/firmware,security_model=none \
  -virtfs local,id=root,mount_tag=root,multidevs=remap,path=/media/sd,security_model=none \
  -append "console=ttyAMA0,115200 root=root rootflags=cache=mmap,msize=104857600,posixacl,trans=virtio,version=9p2000.L rootfstype=9p rootwait"
Enter fullscreen mode Exit fullscreen mode

The kernel will throw a panic since it cannot mount the root of type 9p:

[    1.144617] Disabling rootwait; root= is invalid.
[    1.153539] VFS: Cannot open root device "root" or unknown-block(0,0): error -19
[    1.154459] Please append a correct "root=" boot option; here are the available partitions:
...
[    1.156259] List of all bdev filesystems:
[    1.156335]  ext3
[    1.156346]  ext2
[    1.156386]  ext4
[    1.156414]  vfat
[    1.156440]  msdos
[    1.156469]  f2fs
[    1.156497] 
[    1.156625] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
Enter fullscreen mode Exit fullscreen mode

Even though the 9pfs support is present in the Linux kernel, the Raspberry Pi OS kernel was not built with the corresponding flags. Let's try to fix it by rebuilding the kernel ourselves:

FROM ubuntu:latest AS kernel

ENV ARCH=arm64
ENV CROSS_COMPILE=aarch64-linux-gnu-

WORKDIR /tmp/kernel

ADD https://github.com/raspberrypi/linux/archive/refs/tags/stable_20241008.tar.gz /kernel.tar.gz

RUN tar --strip-components=1 -xzf /kernel.tar.gz \
  && apt-get update \
  && apt-get install --mark-auto --no-install-recommends -y \
    bc \
    bison \
    flex \
    gcc \
    gcc-aarch64-linux-gnu \
    libc6-dev \
    libc6-dev-arm64-cross \
    libssl-dev \
    make \
  && make O=/tmp/build defconfig \
  && scripts/config --file /tmp/build/.config \
    --set-val CONFIG_9P_FS y \
    --set-val CONFIG_9P_FS_POSIX_ACL y \
    --set-val CONFIG_9P_FS_SECURITY y \
    --set-val CONFIG_NETWORK_FILESYSTEMS y \
    --set-val CONFIG_NET_9P y \
    --set-val CONFIG_NET_9P_VIRTIO y \
    --set-val CONFIG_PCI y \
    --set-val CONFIG_PCI_HOST_COMMON y \
    --set-val CONFIG_PCI_HOST_GENERIC y \
    --set-val CONFIG_VIRTIO_PCI y \
    --set-val CONFIG_VIRTIO_BLK y \
    --set-val CONFIG_VIRTIO_NET y \
  && make O=/tmp/build -j 3 Image.gz \
  && rm -rf /tmp/kernel \
  && mkdir -p /media/sd/boot/firmware \
  && cp /tmp/build/arch/$ARCH/boot/Image.gz /media/sd/boot/firmware/kernel8.img
Enter fullscreen mode Exit fullscreen mode

Configuration

Now, the command above should boot the operating system, which still fails to initialize completely:

[FAILED] Failed to start systemd-re…ount Root and Kernel File Systems.
See 'systemctl status systemd-remount-fs.service' for details.
[ TIME ] Timed out waiting for device /dev/disk/by-partuuid/385cce61-01.
Enter fullscreen mode Exit fullscreen mode

This is failing due to incorrect records in /etc/fstab as we provide the filesystem over 9pfs. That can be fixed by overwriting the prebaked /etc/fstab:

COPY <<EOF /media/sd/etc/fstab
proc /proc proc defaults 0 0
boot /boot/firmware 9p cache=mmap,msize=104857600,posixacl,trans=virtio,version=9p2000.L 0 2
root / 9p cache=mmap,msize=104857600,posixacl,trans=virtio,version=9p2000.L 0 1
EOF
Enter fullscreen mode Exit fullscreen mode

It also makes sense to update the boot options in cmdline.txt so we can reuse this file and thereby simulate Raspberry Pi's behavior:

COPY <<EOF /media/sd/boot/firmware/cmdline.txt
console=ttyAMA0,115200 init=/usr/lib/raspberrypi-sys-mods/firstboot root=root rootflags=cache=mmap,msize=104857600,posixacl,trans=virtio,version=9p2000.L rootfstype=9p rootwait
EOF
Enter fullscreen mode Exit fullscreen mode

Now, we can finally boot the OS and log in. However, a couple of services are still failing to initialize:

[FAILED] Failed to start rpi-eeprom…k for Raspberry Pi EEPROM updates.
[FAILED] Failed to start resize2fs_…root filesystem to fill partition.
Enter fullscreen mode Exit fullscreen mode

Since we are running in a virtualized environment, there is no need to run rpi-eeprom-update.service which is attempting to update Raspberry Pi's firmware. The second service tries to expand the root filesystem, but it fails since we are accessing this over 9P. Let's disable both of them:

RUN rm \
  /media/sd/etc/init.d/resize2fs_once \
  /media/sd/etc/systemd/system/multi-user.target.wants/rpi-eeprom-update.service
Enter fullscreen mode Exit fullscreen mode

Hardware

QEMU provides emulation of Raspberry Pi boards out of the box. Unfortunately, it is not so performant when running from Docker, especially since Docker is also running in a virtualized environment on MacOS. On top of that, it does not support PCI devices, which are required for the filesystem sharing.

Instead, we can use the generic platform virt, which is optimized to run in a virtualized environment like Docker for Mac.

In this case, we should explicitly specify the CPU (-smp 4) and RAM (-m 1G) resources available for the virtual machine. It also makes sense to stick to the cortex-a72 CPU as it is being used on an actual board.

Entrypoint

The last part is to get support for the cmdline.txt as Raspberry Pi OS relies on this file. At this moment, the contents of this file will be restored upon container restart. So that, if we change the -append parameter to read from the file, it does not help much:

-append "$(cat /media/sd/boot/firmware/cmdline.txt)"
Enter fullscreen mode Exit fullscreen mode

The QEMU command should be wrapped into a script to restart the virtual machine on soft reboots. In this case, the OS or a user can modify the cmdline.txt and apply these changes in the current container.

There is no way to intercept user reboots and differentiate them from kernel panics. The only possible option would be to read the serial port and restart the process when the kernel yields reboot: Restarting system. That is achievable using expect:

COPY --chmod=755 <<'EOF' /usr/local/bin/rpi
#!/usr/bin/expect

while {true} {
  set reboot false

  spawn -noecho qemu-system-aarch64 \
    -serial mon:stdio \
    -nographic \
    -no-reboot \
    -machine virt \
    -cpu cortex-a72 \
    -m 1G \
    -smp 4 \
    -device virtio-net-device,netdev=net0 \
    -netdev user,id=net0 \
    -kernel /media/sd/boot/firmware/kernel8.img \
    -virtfs local,id=boot,mount_tag=boot,multidevs=remap,path=/media/sd/boot/firmware,security_model=none \
    -virtfs local,id=root,mount_tag=root,multidevs=remap,path=/media/sd,security_model=none \
    -append "[exec cat /media/sd/boot/firmware/cmdline.txt] panic=-1"

  interact {
    -o -reset

    -nobuffer "reboot: Restarting system" {
      set reboot true
      expect eof
      return
    }
  }

  lassign [wait] pid spawn_id os_error exit_code

  if {$exit_code != 0 || !$reboot} {
    exit $exit_code
  }
}
EOF

CMD ["rpi"]
Enter fullscreen mode Exit fullscreen mode

In the script above, we modified the -append argument to add the panic=-1 kernel parameter. In this case, QEMU exits on reboot, and the expect script will reread cmdline.txt. So the OS or a user can make modifications in that file and then reboot to pick up the changes just like on a normal Raspberry Pi board.

Results

Here is the complete Dockerfile including all the instructions above:

Dockerfile 
# syntax=docker/dockerfile:1.3-labs
FROM alpine:latest AS image

ADD https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2024-10-28/2024-10-22-raspios-bookworm-arm64-lite.img.xz /sd.img.xz

RUN --security=insecure \
  apk add --no-cache --virtual=.tools \
    jq \
    sfdisk \
  && unxz -ck /sd.img.xz >/tmp/sd.img \
  && mkdir -p /tmp/sd \
  && mount -o loop,offset="$(sfdisk -J /tmp/sd.img | jq '.partitiontable.sectorsize * .partitiontable.partitions[1].start')" /tmp/sd.img /tmp/sd \
  && mount -o loop,offset="$(sfdisk -J /tmp/sd.img | jq '.partitiontable.sectorsize * .partitiontable.partitions[0].start')" /tmp/sd.img /tmp/sd/boot/firmware \
  && cp -pr /tmp/sd /media/sd \
  && umount /tmp/sd/boot/firmware /tmp/sd \
  && rm -rf /tmp/sd /tmp/sd.img \
  && apk del .tools

RUN rm \
  /media/sd/etc/init.d/resize2fs_once \
  /media/sd/etc/systemd/system/multi-user.target.wants/rpi-eeprom-update.service

COPY <<EOF /media/sd/etc/fstab
proc /proc proc defaults 0 0
boot /boot/firmware 9p cache=mmap,msize=104857600,posixacl,trans=virtio,version=9p2000.L 0 2
root / 9p cache=mmap,msize=104857600,posixacl,trans=virtio,version=9p2000.L 0 1
EOF

COPY <<EOF /media/sd/boot/firmware/cmdline.txt
console=ttyAMA0,115200 init=/usr/lib/raspberrypi-sys-mods/firstboot root=root rootflags=cache=mmap,msize=104857600,posixacl,trans=virtio,version=9p2000.L rootfstype=9p rootwait
EOF

FROM ubuntu:latest AS kernel

ENV ARCH=arm64
ENV CROSS_COMPILE=aarch64-linux-gnu-

WORKDIR /tmp/kernel

ADD https://github.com/raspberrypi/linux/archive/refs/tags/stable_20241008.tar.gz /kernel.tar.gz

RUN tar --strip-components=1 -xzf /kernel.tar.gz \
  && apt-get update \
  && apt-get install --mark-auto --no-install-recommends -y \
    bc \
    bison \
    flex \
    gcc \
    gcc-aarch64-linux-gnu \
    libc6-dev \
    libc6-dev-arm64-cross \
    libssl-dev \
    make \
  && make O=/tmp/build defconfig \
  && scripts/config --file /tmp/build/.config \
    --set-val CONFIG_9P_FS y \
    --set-val CONFIG_9P_FS_POSIX_ACL y \
    --set-val CONFIG_9P_FS_SECURITY y \
    --set-val CONFIG_NETWORK_FILESYSTEMS y \
    --set-val CONFIG_NET_9P y \
    --set-val CONFIG_NET_9P_VIRTIO y \
    --set-val CONFIG_PCI y \
    --set-val CONFIG_PCI_HOST_COMMON y \
    --set-val CONFIG_PCI_HOST_GENERIC y \
    --set-val CONFIG_VIRTIO_PCI y \
    --set-val CONFIG_VIRTIO_BLK y \
    --set-val CONFIG_VIRTIO_NET y \
  && make O=/tmp/build -j 3 Image.gz \
  && rm -rf /tmp/kernel \
  && mkdir -p /media/sd/boot/firmware \
  && cp /tmp/build/arch/$ARCH/boot/Image.gz /media/sd/boot/firmware/kernel8.img

FROM alpine:latest

RUN apk add --no-cache \
  expect \
  qemu-system-aarch64

COPY --from=image /media/sd /media/sd
COPY --from=kernel /media/sd /media/sd
COPY --chmod=755 <<'EOF' /usr/local/bin/rpi
#!/usr/bin/expect

while {true} {
  set reboot false

  spawn -noecho qemu-system-aarch64 \
    -serial mon:stdio \
    -nographic \
    -no-reboot \
    -machine virt \
    -cpu cortex-a72 \
    -m 1G \
    -smp 4 \
    -device virtio-net-device,netdev=net0 \
    -netdev user,id=net0 \
    -kernel /media/sd/boot/firmware/kernel8.img \
    -virtfs local,id=boot,mount_tag=boot,multidevs=remap,path=/media/sd/boot/firmware,security_model=none \
    -virtfs local,id=root,mount_tag=root,multidevs=remap,path=/media/sd,security_model=none \
    -append "[exec cat /media/sd/boot/firmware/cmdline.txt] panic=-1"

  interact {
    -o -reset

    -nobuffer "reboot: Restarting system" {
      set reboot true
      expect eof
      return
    }
  }

  lassign [wait] pid spawn_id os_error exit_code

  if {$exit_code != 0 || !$reboot} {
    exit $exit_code
  }
}
EOF

CMD ["rpi"]
Enter fullscreen mode Exit fullscreen mode

It lacks some features, like enabling SSH and changing the default user password, but those can be found in other tutorials or the official documentation.

Despite that, it has already been implemented and tested in the original repository, which prompted me to write this article. And of course, there is a dokmic/rpi ready-to-use image published on Docker Hub that supports both ARM64 and ARM architectures and has several configuration options to customize the emulated Raspberry Pi OS.

Translations of this article are allowed only upon permission.

💖 💪 🙅 🚩
dokmic
Michael

Posted on November 25, 2024

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Running Raspberry Pi OS in a Docker Container
raspberrypi Running Raspberry Pi OS in a Docker Container

November 25, 2024