RETHV - Building Reth for the RISC-V platform

We describe how we managed to run Reth, the Rust Ethereum execution client maintained by Paradigm and the open-source community, on a RISC-V environnement.

As we faced issues and learned many things, we wanted not only to share the finished PoC but also the journey itself. If you'd like to chat about running Reth on RISC-V in more details, feel free to contact us.

Last but not least, we are not experts on the topics described in the document - we might have missed easy fixes, misunderstood concepts or over-engineered some solutions, we encourage you to let us know.

Happy reading!

Introduction
The RISC-V environnement
- Bare Metal
- QEMU to the rescue
Cross-Compiling Reth
Cross-Compilation Target
MDBX Bindgen Configuration
Building Reth
Wrapping Up
Running Reth
Conclusion
Acknowledgments
Resources
Authors

Introduction

Our goal was to be able to run Reth on a RISC-V GNU Linux. Why? Because:

We believe it enhances client diversity. It's not only about what is the most used client implementation, but also what hardware it runs on.
It's an amazing challenge. We learned many things about cross-compilation, QEMU and the tools used in Reth.
Extending the previous reason, we just wanted to share our journey with the community. Maybe it'll help some of you get started on RISC-V, exploring QEMU and more.

We'll explain how we managed to configure a RISC-V environnement, how we cross-compiled Reth for the RISC-V target and present the tools used to achieve it. We'll also recap the limitations we currently know about.

The RISC-V environnement

In this first section, we describe how we failed to run Reth on a non-compatible bare-metal environnement, which led us to use a VM with the help of QEMU.

Bare Metal

At the beginning, we really wanted to run Reth on a bare-metal server with a RISC-V processor.

We used the Elastic Metal RV1 instances from Scaleway but faced a first error we previously encountered when trying to run Reth on ARM64 machines: the virtual memory layout on Linux.
In a nutshell, the Linux kernel can be compiled to use a specific virtual memory layout, but the hardware on the machine might not support all of them.
Scaleway's instances only support SV39 mode at best, while we need SV48 at least. Not using a proper virtual memory layout leads to an error which was documented in the Reth issue here .
On RISC-V Linux, the SV39 mode allows up to 256 GB of user-space virtual memory while the SV48 allows up to 128 TB - as specified in the kernel documentation.
Check-out the issue linked above to learn more about the underlying reasons of why this is an issue for Reth, onbjerg did an amazing job explaining it in simple terms.

When we started the project, Scaleway was the only cloud provider with RISC-V instances and we did not have a dev board at home, which led us to look for another setup: using QEMU to create a RISC-V VM.

QEMU to the rescue

We tried to follow the official documentation for running Linux on QEMU for RISC-V but faced too many compilation errors on macOS with Apple Silicon.

As we had a Ubuntu machine with an x86 CPU on Hetzner for development purposes, we decided to immediately switch.
After following the instructions to compile QEMU with RISC-V compatibility and running a compatible Debian image (amazing blog post with the instructions here), it worked!
You can connect to the VM using:

ssh root@localhost -p 2222 # Passord: root

root@debian:~# uname -r
6.8.12-riscv64
root@debian:~# cat /proc/cpuinfo
...
mmu             : sv57 # Sweet, this virtual memory layout allows up to 64 PB user-space virtual memory. Enough to store da blobs.
...

Note that the provided image only has a 10 GB-size disk. If you'd like to extend it consider taking a look at this SO answer and then resizing the FS using fdisk (be careful, when expanding the root partition, the first sector might not be the default one) and resize2fs.

The development environnement is ready. Now is the time to build and run Reth.

Cross-Compiling Reth

To compile Reth for the RISC-V 64 target, we have different options:

Compile natively inside the VM. Slow but less prone to errors and requires less configuration tweaks.
Compile using the cross-compiler. Requires to setup the tools first but leads to way faster compilation time.

The first option is actually really easy: just follow the official documentation to build Reth from source from within the VM.
The second option is a bit more complex: we can either use install the cross-compiler from a package manager or build it from source.

As your distribution may very from our own, we chose to build it from the official toolchain sources. In fact, we built a Docker image which contains the RISC-V toolchain built from source for both the amd64 and arm64 platforms.

From there, it's as easy as just running cargo build right? Well, there are two details.
The cross-compilation target must be specified and requires some extra parameters, and the MDBX auto-generated bindings requires extra-configuration.

Cross-Compilation Target

When cross-compiling in Rust using cargo, the target can be specified using the --target flag.
Here, we will inform cargo that the target is riscv64gc-unknown-linux-gnu.
One thing we discovered was that another target we aimed for, riscv64gc-unknown-none-elf, is not yet fully supported by Rust. Namely, it does not support std right now and therefore Reth cannot be cross-compiled for this target.

During the cross-compilation, the linker also must be specified manually, and we chose the easiest way (an environnement variable) to do so:

export CARGO_TARGET_RISCV64GC_UNKNOWN_LINUX_GNU_LINKER=riscv64-unknown-linux-gnu-gcc

But if we try to build Reth, the compilation will fail due to the custom mdbx-sys crate build process.

...
33.85 error: failed to run custom build command for `reth-mdbx-sys v1.0.0 (/build/crates/storage/libmdbx-rs/mdbx-sys)`
33.85
33.85 Caused by:
33.85   process didn't exit successfully: `/build/target/release/build/reth-mdbx-sys-c89ac4f3243e36e2/build-script-build` (exit status: 101)
33.85   --- stdout
33.85   cargo:rerun-if-changed=/build/crates/storage/libmdbx-rs/mdbx-sys/libmdbx
33.85
33.85   --- stderr
33.85   /build/crates/storage/libmdbx-rs/mdbx-sys/libmdbx/mdbx.h:80:2: warning: "The RISC-V architecture is intentionally insecure by design.   Please delete this admonition at your own risk,   if you make such decision informed and consciously.   Refer to https://clck.ru/32d9xH for more information." [-W#warnings]
33.85   /usr/include/stdint.h:26:10: fatal error: 'bits/libc-header-start.h' file not found
...

Let's fix it by tweaking the bindgen configuration.

MDBX Bindgen Configuration

To build the mdbx-sys crate bindings, the bindgen tool is used. As it uses clang to generate the C/C++/Rust bindings, it does not know about the RISC-V toolchain and we must specify it manually.
Hopefully, as stated in the bindgen's README, we can pass extra flags to clang using the BINDGEN_EXTRA_CLANG_ARGS environnement variables (we could also tweak the build.rs file in the crate, but we did not wanted to temper with the sources).

# The sysroot for the quartztech/riscv-gnu-toolchain image is in /usr/local/
export BINDGEN_EXTRA_CLANG_ARGS="--sysroot=/usr/local/sysroot"

Finally, we can build Reth.

Building Reth

Here's the command to build reth using our RISC-V cross-compiler:

cargo build --release --target riscv64gc-unknown-linux-gnu

Wrapping Up

Custom Docker image for RISC-V toolchain

Here's our Dockerfile used as the based image in the next sub-section.
It contains the RISC-V toolchain built from source.

FROM ubuntu:22.04 AS builder

WORKDIR /build

RUN apt-get update && apt-get install -y \
    autoconf \
    automake \
    autotools-dev \
    curl \
    python3 \
    python3-pip \
    libmpc-dev \
    libmpfr-dev \
    libgmp-dev \
    gawk \
    build-essential \
    bison \
    flex \
    texinfo \
    gperf \
    libtool \
    patchutils \
    bc \
    zlib1g-dev \
    libexpat-dev \
    ninja-build \
    git \
    cmake \
    libglib2.0-dev \
    libslirp-dev

RUN git clone https://github.com/riscv/riscv-gnu-toolchain .

# Newlib installation.
RUN ./configure --enable-multilib
RUN make

# Linux installation.
RUN ./configure --enable-multilib
RUN make linux

FROM ubuntu:22.04 AS runtime

WORKDIR /

COPY --from=builder /usr/local /usr/local

ENTRYPOINT [ "/bin/bash" ]

Using the non-trivial approach

For the braves, here's a Dockerfile which uses the Quartz base image to cross-compile Reth:

FROM quartztech/riscv-gnu-toolchain:latest

WORKDIR /build

ENV PATH=/opt/riscv/bin:$PATH

RUN apt-get update && apt-get install -y \
    curl \
    build-essential \
    git \
    pkg-config \
    libssl-dev \
    libclang-dev

RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH=/root/.cargo/bin:$PATH

RUN rustup target add riscv64gc-unknown-linux-gnu

RUN git clone https://github.com/paradigmxyz/reth .

ENV CARGO_TARGET_RISCV64GC_UNKNOWN_LINUX_GNU_LINKER=riscv64-unknown-linux-gnu-gcc
ENV BINDGEN_EXTRA_CLANG_ARGS="--sysroot=/usr/local/sysroot"

RUN cargo build --release --target riscv64gc-unknown-linux-gnu

ENTRYPOINT [ "/bin/bash" ]

Using a simpler approach

Now if you use a simpler method and just install the toolchain from a package manager, here's what the Dockerfile would look like (with Ubuntu as the base image at least):

FROM ubuntu:22.04

WORKDIR /build

RUN apt-get update && apt-get install -y \
    curl \
    build-essential \
    git \
    pkg-config \
    libssl-dev \
    libclang-dev \
    # All of this article for this change ^^, I love reinventing the wheel.
    gcc-riscv64-linux-gnu

RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH=/root/.cargo/bin:$PATH

RUN rustup target add riscv64gc-unknown-linux-gnu

RUN git clone https://github.com/paradigmxyz/reth .

ENV CARGO_TARGET_RISCV64GC_UNKNOWN_LINUX_GNU_LINKER=riscv64-linux-gnu-gcc

RUN cargo build --release --target riscv64gc-unknown-linux-gnu

ENTRYPOINT [ "/bin/bash" ]

Way simpler than everything else, sometimes it's all about installing one package that does the job.

Running Reth

The last step is to copy the binary built inside the Docker image to the RISC-V VM:

docker cp <containerId>:/build/target/riscv64gc-unknown-linux-gnu/release/reth .
scp -P 2222 reth root@localhost:/tmp
ssh root@localhost -p 2222
# ...
# ...
# ...
root@debian:~# cd /tmp
root@debian:/tmp# ls
reth
root@debian:/tmp# ./reth node --full
2024-06-24T20:00:40.554366Z  INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/mainnet
2024-06-24T20:00:40.588379Z  INFO Starting reth version="1.0.0-dev (81b5fbf57)"
2024-06-24T20:00:40.592833Z  INFO Opening database path="/root/.local/share/reth/mainnet/db"
2024-06-24T20:00:40.824511Z  INFO Configuration loaded path="/root/.local/share/reth/mainnet/reth.toml"
2024-06-24T20:00:40.908347Z  INFO Verifying storage consistency.
2024-06-24T20:00:41.025506Z  INFO Database opened
2024-06-24T20:00:41.029405Z  INFO
# ...

While it is definitely NOT an environnement suited for production, it works!

Conclusion

We managed to overcome some challenges to build and run Reth on RISC-V, which contributes to maximizing the diversity of the Ethereum infrastructure!

Now let's not celebrate too much: RISC-V is VERY early, slow and insecure (even MDBX reminds us of this, look back at the logs ^^). It's just a proof of concept and also a good excuse for us to test new things.
We also did not (but it's in the TODO list) synchronized a full-node entirely, and therefore can not attest that it works completely.

I really encourage you to try this on your own, hopefully you'll learn a few things in the process. I spent some time reading documentation on RISC-V, the Rust bindgen, cross-compilation and many more. As everyone has it's own way of approaching this deep-focus phase, I'll let you invest your time as you wish.

Acknowledgments

All the Reth contributors for their amazing work.
Onbjerg for his help on debugging compilation errors on ARM64 due to memory layout configuration.
Colin Atkinson for his post on running RISC-V in QEMU.

Resources

Authors

This project was made by the 🦀 at Quartz Technology.

Blog