I like Rust. I like the ecosystem, I like the performance, it's the perfect blend for me. Coupled with the safety features, I feel like I have a wise old friend guiding me through areas that could otherwise cause me some grand issues.

So why Rust? C is fine

As you saw in the last article, BPF programs are typically programmed in C. But why couldn't we use Rust? It can be just as low level as C.

The biggest win in my mind is the ergonomics that could be achieved (which I will hopefully be able to improve across these next few posts) along with the crates ecosystem that has enormous potential when compared to the C BPF ecosystem. It could be argued that because of the verifier the borrow checker is moot, but I don't see it in quite the same light. Sure the verifier will enforce specific checks and be arguably far more strict than the borrow checker but that
doesn't mean we can't get any use out of leaning on Rust's borrowing system.

Of course there is the subjective benefit that I'm more productive in Rust, and far more proficient with Rust than with C. Plus if I'm writing the supporting applications/libraries in Rust it'd be nice to stay in a single language when possible.

RedBPF

As it turns out there are a number of crates dedicated to writing BPF code in Rust! Many are in varying states of completeness and activity. The set of crates I've found most complete and promising is the redbpf collection. It's not perfect, but it shows great promise and the developers are super friendly and helpful. In this post I hope to dive into the design decisions and some new ideas for improving the networking portions of the redbpf crates.

If you remember all the way back to part 1, the problem we were originally trying to solve was a networking one, where we needed to validate a packet and multiplex a single ingress port to multiple ingress ports depending on parts of the payload.

Whats in the Box

The redbpf repository is divided into several crates:

• bpf-sys: provides bindings to libbpf and parts of BCC by using bindgen
• cargo-bpf: A cargo subcommand that handles the boilerplate of setting up and building a BPF project in Rust
  • also contains a library for generating additional bindings, and a development loader
• redbpf: A userspace library for loading and interacting with BPF programs
• rebpf-probes: A library to writing kprobes, uprobes, and XDP or Socket BPF programs in Rust
  • also provides a bindings module where it uses bindgen to generate bindings for libbpf
• rebpf-macros: A procedural macro crate which contains the proc-macros used to wrap BPF functions and code in redbpf-probes
• rebpf-tools: Sample projects generated by cargo-bpf to demonstrate how to structure code

Interestingly, bpf-sys is only used by cargo-bpf and redbpf, but not by redbpf-probes or redbpf-macros. I asked about this on Github and was told the reasons for both were historical and due to stability, but now that things have improved they plan to merge everything down to bpf-sys with libbpf. This will be great, because it caused a little confusion early on while investigating the code.

We can somewhat ignore redbpf-tools as it's pretty much just example code. I would prefer if it was in the examples/ dir, but that's just subjective preference I don't plan on trying to move it. Finally, redbpf-macros is used heavily by redbpf-probes, and since redbpf-probes will be our main focus for this post we will also touch the proc-macros to some extent.

There is another tool, bpf-linker which is not part of the RedBPF collection, but was written recently by the primary author of the RedBPF crates. Although I haven't started using it heavily yet as it's so new, I have no doubt it will become key to this space.

Example Current Solution to Part 1 Problem

We'll first create a solution the problem we spoke about in Part 1 using RedBPF as it stands (kind of). This will allow us to contrast the solution with future iterations, and discover/discuss design changes and improvements.

To write BPF code in Rust, it's easiest to use
cargo-bpf (part of the redbpf suite) which handles setting up the project and can even function as a development loader.

I recommend installing it via cargo install from the git repository, but first you must make sure you have all the required development files such as LLVM 11, kernel headers, etc. (see the repository for details as it's quite explicit and outside the scope of this post)

Once you have all the required packages, install cargo-bpf with:

$ cargo install cargo-bpf --git https://github.com/redsift/redbpf

We'll create our example project called mplex:

$ cargo new mplex
$ cd mplex/

Our project will most likely contain a custom loader in the future, and at least one BPF object. mplex will be the place holder for our "outer" userspace application and loader. We'll use cargo-bpf to add BPF objects to this
project, which will be in the form of stand alone binaries.

The way cargo-bpf does this is by creating additional binaries using cargo's [[bin]] tables, and requiring specific cargo features to be passed in order to compile the BPF code. When you run cargo bpf build it will search through the project and find the binaries listed, and build them with the appropriate cargo features enabled. In this way we can split out the dependencies required by our BPF program(s) and our userspace application/loader.

At first though we will be using cargo-bpf as the loader, so we can focus on the important parts of this post.

XDP, TC, or Socket?

Ok, we're at our first decision point; where to hook in our BPF program? We know we want a networking hook, and the socket layer is too high as we want to affect the incoming port. We could use TC, but we only need the ingress side, and since we'll be re-writing part of the packet it'd be nice to have direct access to the packet memory.

XDP

XDP checks all of our boxes, and is the earliest possible point to observe or mutate a packet.

The argument against XDP is that if we'll most likely be passing the packet up the networking stack anyways (after mutating) we don't really save anything by not allocating a socket buffer in the TC layer.

XDP is normally best when you're just trying to drop/redirect packets out (firewalling or routing), however, we are validating packets and potentially dropping a fair percentage so using XDP is fine. Perhaps in a later post I will come back and show a TC variant as well so we can contrast the two solutions. In fact if I do my job well, the two solutions should not be too different by the time we're done improving these crates.

We can then tell cargo bpf to create a BPF executable for us which we'll call mplex_xdp:

$ cargo bpf add mplex_xdp

Our project structure now has two binaries, mplex (at src/main.rs) will be the eventual userspace loader, and a new src/mplex_xdp/main.rs has been added which will be the BPF object.

It's going to feel like we just jumped from level 0 to 100 skipping all the steps in-between. But the purpose of this post is to discuss the current state of RedBPF networking, vs future improvements we can make. Not the specifics of how
we can implement a program for the problem in part 1.

Current RedBPF XDP Example

Removing the generated example from src/mplex_xdp/main.rs we accomplish our set out task with the following simplified code (see caveat below):

#![no_std]
#![no_main]
use redbpf_probes::{
    bindings::tcphdr,
    net::Transport,
    xdp::prelude::*,
};

program!(0xFFFFFFFE, "GPL");

#[xdp]
pub fn mplex(ctx: XdpContext) -> XdpResult {
    // only match TCP
    if let Ok(Transport::TCP(tcp)) = ctx.transport() {
        unsafe {
            // Only match destination port 5000
            if u16::from_be((*tcp).dest) == 5000 {
                let d = ctx.data()?; // get the payload
                let ds = d.slice(d.len())?; // turn the payload into a byte slice

                // "Validation" ensure payload length is within the narrow window
                let payload_len = ds.len();
                if payload_len < 290 || payload_len > 294 {
                    // Drop packet if not
                    return Ok(XdpAction::Drop);
                }

                // Multiplex based on entity tag which is 3 bytes, at an offset of 
                // 20 bytes into the payload
                //
                // Double slice means "skip 20 bytes, then return the next 3"
                match &ds[20..][..3] {
                    b"600" => {
                        // re-write destination port
                        (*(tcp as *mut tcphdr)).dest = u16::to_be(5001); 
                    }
                    b"601" => {
                        (*(tcp as *mut tcphdr)).dest = u16::to_be(5002);
                    }
                    b"602" => {
                        (*(tcp as *mut tcphdr)).dest = u16::to_be(5003);
                    }
                    _ => return Ok(XdpAction::Drop);, // no matching tag means invalid; drop
                }
            }
        }
    }

    // Allow to pass up the stack
    Ok(XdpAction::Pass)
}

CAVEAT: The caveat is that the above code will fail the BPF verifier incorrectly because it thinks we're not doing proper bounds checking. This is actually one of the things that lead me to look at improving the RedBPF networking. So this example is roughly what a solution would look like sans a few changes that aren't super important right now.

But generally, we can already see that this is leaps and bounds better in terms of ergonomics and readability (IMO) than the equivalent C.

If we were to run this on our incoming interface, (after calming the verifier) we'd see that in-fact all the problems described in part 1 are magically gone. The server application now believes multiple ports are being utilized, so it's happy, and the external source system is only sending data across a single port so they're happy too. Meanwhile this tiny XDP shim is churning along, with no performance hit.

Can we do Better?

But we're not done! We can make the verifier happy with standard Rust idioms, we shouldn't have to jump through hoops just to make the verifier happy when it appear like we're already doing those things by telling Rust to do it. There
is also a little too much unsafe for me, and the abstractions fall a little short since it's a little too coupled to the 4 protocols currently supported.

After speaking with the authors of redbpf-probes, they stated networking/XDP wasn't their main focus, as they instead they utilize the tracing aspects far more frequently so the networking API is not as solid or had as much thought put into it. It was also some of the first code written as merely a proof of concept.

It's open source, so let's see if we can help with that!

I reached out to the original author, and we've been going back and forth about potential changes. He's on board, and we're both super excited to see where we can take this!

Wrap Up

In the part of this series we'll do that deep dive into the current implementation and note areas that we want to mark for improvement or change.