TheLazy.Dev
Portfolio GitHub

Blog

rust |webassembly |devjournal

#![no_std] with WASI is more complicated than I thought it would be

thepuzzlemaker

James [Undefined]

Posted on June 3, 2021

#![no_std] with WASI is more complicated than I thought it would be

This post is meant to document my experience using #![no_std] in Rust with WASI, which was way more complicated than I thought it would. Perhaps this can help to make the documentation better, who knows? :P

So I was wanting to learn more about WASI (WebAssembly System Interface), which is basically like POSIX interfaces for WASI, with capabilities for better security. I decided that perhaps making a "Hello, world!" example in Rust with #[no_std] would be a good idea.

It was... difficult. There were a ton of roadblocks.

So, let's start this.

I started with a basic shell of a #![no_std] program:

#![no_std]
#![no_main]

use core::panic::PanicInfo;

#[panic_handler]
fn panic_handler(_info: &PanicInfo) -> ! {
    loop {}
}

#[no_mangle]
pub extern "C" fn _start() -> ! {
    loop {}
}
Enter fullscreen mode Exit fullscreen mode

In the Cargo.toml I set panic = "abort":

[profile.dev]
panic = "abort"

[profile.release]
panic = "abort"
Enter fullscreen mode Exit fullscreen mode

I use cargo-wasi to build it:

$ cargo wasi build
   Compiling wasi-testing v0.1.0 (/home/james/Code/Rust/wasi-testing)
error: linking with `rust-lld` failed: exit status: 1
  |
  = note: ... really long command line invocation ...
  = note: rust-lld: error: duplicate symbol: _start
          >>> defined in /home/james/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-wasi/lib/self-contained/crt1-command.o
          >>> defined in /home/james/Code/Rust/wasi-testing/target/wasm32-wasi/debug/deps/wasi_testing.5awycwmt43f1nxf9.rcgu.o


error: aborting due to previous error

error: could not compile `wasi-testing`

To learn more, run the command again with --verbose.
Enter fullscreen mode Exit fullscreen mode

Uh... what?

So what happened is that _start is already defined by this mysterious crt1-command.o. I didn't learn until much after that this is from wasi-libc:

#include <wasi/api.h>
#include <stdlib.h>
extern void __wasm_call_ctors(void);
extern int __original_main(void);
extern void __wasm_call_dtors(void);

__attribute__((export_name("_start")))
void _start(void) {
    // Call `__original_main` which will either be the application's zero-argument
    // `__original_main` function or a libc routine which calls `__main_void`.
    // TODO: Call `main` directly once we no longer have to support old compilers.
    int r = __original_main();

    // If main exited successfully, just return, otherwise call `exit`.
    if (r != 0) {
        exit(r);
    }
}
Enter fullscreen mode Exit fullscreen mode 
// TODO: Call `main` directly once we no longer have to support old compilers.
Enter fullscreen mode Exit fullscreen mode

This one comment is what caused 99% of my pain with starting the actual program.

At the time, I didn't know about this, so I just tried changing the name of _start to __original_main:

#[no_mangle]
pub extern "C" fn __original_main() -> ! {
    loop {}
}
Enter fullscreen mode Exit fullscreen mode

I build it, and I get this error:

error: linking with `rust-lld` failed: exit status: 1
  |
  = note: ... really long command line invocation ...
  = note: rust-lld: error: function signature mismatch: __original_main
          >>> defined as () -> i32 in /home/james/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-wasi/lib/self-contained/crt1-command.o
          >>> defined as () -> void in /home/james/Code/Rust/wasi-testing/target/wasm32-wasi/debug/deps/wasi_testing.5awycwmt43f1nxf9.rcgu.o
Enter fullscreen mode Exit fullscreen mode

Ok, looks like this function must return an i32. That's fine, I guess. I assumed that this function's return value is the exit code, which I eventually learn is true.

#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    0
}
Enter fullscreen mode Exit fullscreen mode

I compile it, and it works.

Now I run it:

Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/wasi-testing.wasm"
    1: unknown import: `env::exit` has not been defined
Enter fullscreen mode Exit fullscreen mode

Uh... ok then.
At this point I decide that I should probably use WASI to define this exit function, instead of just endlessly looping. So I link in proc_exit. According to the WASI docs, proc_exit takes in an exitcode (a u32) and returns nothing. Perfect!

So I link it in. Notice the deadly mistake? I didn't, yet.

extern "C" {
    fn proc_exit(code: u32);
}
Enter fullscreen mode Exit fullscreen mode

Then I define fn exit() -> !:

#[no_mangle]
pub extern "C" fn exit(code: u32) -> ! {
    unsafe { proc_exit(code); }
    loop {} // (hopefully) never reached
}
Enter fullscreen mode Exit fullscreen mode

It builds, but it doesn't work:

Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/wasi-testing.wasm"
    1: unknown import: `env::proc_exit` has not been defined
Enter fullscreen mode Exit fullscreen mode

Hm.

So I try defining proc_exit as such:

pub mod internal {
    #[no_mangle]
    #[export_name = "proc_exit"]
    pub unsafe extern "C" fn proc_exit(code: u32) {
        super::proc_exit(code);
    }
}
Enter fullscreen mode Exit fullscreen mode

I run it, and... uh what?

Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to invoke command default
    1: wasm trap: call stack exhausted
       wasm backtrace:
           0: 0xffffffff - <unknown>!proc_exit
           1:   0xd8 - <unknown>!proc_exit
           2:   0xd8 - <unknown>!proc_exit
           3:   0xd8 - <unknown>!proc_exit
           4:   0xd8 - <unknown>!proc_exit
           5:   0xd8 - <unknown>!proc_exit
           6:   0xd8 - <unknown>!proc_exit
           7:   0xd8 - <unknown>!proc_exit
           ... a TON of lines cut ...
           32682:   0xd8 - <unknown>!proc_exit
           32683:  0x10b - <unknown>!exit
           32684:  0x122 - <unknown>!_start
         note: run with `WASMTIME_BACKTRACE_DETAILS=1` environment variable to display more information
Enter fullscreen mode Exit fullscreen mode

Uh ok. At this point I realized I was probably making it call into itself endlessly.

At this point, I look into Bytecode Alliance's WASI rust bindings, used by Rust's std.

I look multiple times, and only about the third time do I notice this fatal line:

    #[link(wasm_import_module = "wasi_snapshot_preview1")]
    extern "C" {
        // ... //
    }
Enter fullscreen mode Exit fullscreen mode

I add that to my extern "C" block and remove the weird internal::proc_exit function. Here's the full code at this point:

#![no_std]
#![no_main]

use core::panic::PanicInfo;

#[panic_handler]
fn panic_handler(_info: &PanicInfo) -> ! {
    loop {}
}

#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    0
}

#[no_mangle]
pub extern "C" fn exit(code: u32) -> ! {
    unsafe {
        proc_exit(code);
    }
    loop {} // (hopefully) never reached
}

#[link(wasm_import_module = "wasi_snapshot_preview1")]
extern "C" {
    fn proc_exit(code: u32);
}
Enter fullscreen mode Exit fullscreen mode

I build and run it, and it works. Since it prints nothing to the screen, I change the 0 in __original_main to a different value that I can observe from the terminal. The specific number doesn't matter, just that it's not zero.

$ wasmtime target/wasm32-wasi/debug/wasi-testing.wasm
$ echo $status # I use fish, with POSIX-compliant shells it's `$!`.
42
Enter fullscreen mode Exit fullscreen mode

And it works! Ok, time to do something a bit more complex. I'm going to work on a print function.

I use fd_write, and since I don't want to look into how interface types work, I just look at the wasi crate again, with some help from the wasmtime WASI WAT tutorial.

I link it in (adding some helper structs, aliases, and enums which I've omitted for brevity, but can be seen in the full code at the end of the post):

// extern "C" block from before
extern "C" {
    pub fn fd_write(
        fd: Fd,
        ciovecs: *const Ciovec,
        n_ciovecs: Size,
        n_written: *mut Size,
    ) -> Errno;
}
Enter fullscreen mode Exit fullscreen mode

I then create a print function:

fn print(s: &str) -> Result<Size, Errno> {
    assert!(
        s.len() <= u32::MAX as usize,
        "please don't store >4GB of text in a string, thanks"
    );
    let ciovec = Ciovec {
        buf: s.as_ptr(),
        buf_len: s.len() as u32,
    };
    let ciovec_ptr = &ciovec as *const Ciovec;
    let mut n_written: Size = 0;
    let errno = unsafe { fd_write(1, ciovec_ptr, 1, &mut n_written as *mut Size) };
    if errno == Errno::Esuccess {
        return Ok(n_written);
    }
    Err(errno)
Enter fullscreen mode Exit fullscreen mode

Here's how this works:

  • I first assert that the string is not larger than the 32-bit limit, which would be logically incorrect and potentially even unsound to try and print.
  • I then construct the ciovec (const IO vector) with the buffer (pointer to the string as a *const u8), and the length as a u32.
  • I then get a pointer to that ciovec on the stack.
  • I create a variable on the stack, n_written.
  • I call fd_write with the file descriptor (1 = stdout), the pointer to the ciovecs, the length of the ciovec array (1), as you can print multiple ciovec arrays at once as an optimization for buffered writes, I think, and finally a mutable reference to the n_written variable on the stack.
  • If the errno returned did not indicate an error (0 = success), then I return Ok(n_written).
  • Otherwise I return the errno.

I call the function within the __original_main function (changing the return value to 0, again):

#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    print("Hello, world!\n").unwrap();
    0
}
Enter fullscreen mode Exit fullscreen mode

At this point, since I'm calling .unwrap(), I'm now realizing I should probably have a better #[panic_handler]. So I change it now:

#[panic_handler]
fn panic_handler(info: &PanicInfo) -> ! {
    print(&format!("{}\n", info)).unwrap();

    loop {}
}
Enter fullscreen mode Exit fullscreen mode

I then again realize that format! can't be used in #![no_std] contexts without alloc.

I add this to the top of the file to indicate that it should use alloc:

extern crate alloc;
Enter fullscreen mode Exit fullscreen mode

Then I import format! from alloc:

use alloc::format;
Enter fullscreen mode Exit fullscreen mode

I build it:

error: no global memory allocator found but one is required; link to std or add `#[global_allocator]` to a static item that implements the GlobalAlloc trait.

error: `#[alloc_error_handler]` function required, but not found.

note: Use `#![feature(default_alloc_error_handler)]` for a default error handler.
Enter fullscreen mode Exit fullscreen mode

Ok. Now I need to add in a global allocator. I use wee_alloc since I know it works pretty well with WASM:
Cargo.toml

[dependencies]
wee_alloc = "0.4.5"
Enter fullscreen mode Exit fullscreen mode

main.rs

use wee_alloc::WeeAlloc;

#[global_allocator]
pub static ALLOC: WeeAlloc = WeeAlloc::INIT;
Enter fullscreen mode Exit fullscreen mode

And then I need to create an allocator error handler:

#[alloc_error_handler]
pub fn alloc_error_handler(_: core::alloc::Layout) -> ! {
    loop {}
}
Enter fullscreen mode Exit fullscreen mode

This means now that I have to add the feature flag for #[alloc_error_handler], meaning that I now have to use the nightly branch:

#![feature(alloc_error_handler)]
Enter fullscreen mode Exit fullscreen mode

I add a panic!() in the main function to test it.

I run it:

Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/wasi-testing.wasm"
    1: unknown import: `env::memcpy` has not been defined
Enter fullscreen mode Exit fullscreen mode

Um. Ok, I remember this kind of thing. I do some searching and then rediscover compiler_builtins.

I add it to the program:
Cargo.toml

[dependencies]
# ...
compiler_builtins = "0.1.44"
Enter fullscreen mode Exit fullscreen mode

main.rs

extern crate compiler_builtins;
Enter fullscreen mode Exit fullscreen mode

It still doesn't work, however. It still complains that memcpy is not defined. I then realize that I have to enable the mem feature:

[dependencies]
# ...
compiler_builtins = { version = "0.1.44", features = ["mem"] }
Enter fullscreen mode Exit fullscreen mode

And it works! Well, kind of:

Hello, world!
panicked at 'explicit panic', src/main.rs:31:5
Enter fullscreen mode Exit fullscreen mode

It unfortunately, however, infinitely hangs thanks to the infinite loop {}. I decide to use proc_raise to send an abort signal:

extern "C" {
    // ...
    fn proc_raise(sig: Signal) -> Errno;
}
Enter fullscreen mode Exit fullscreen mode

You can see the definition of Signal and Errno in the full code at the end of this post.

I then define a function, abort(), because I'll be using this same logic within the alloc_error_handler so that when allocation fails, it doesn't silently hang.

fn abort() -> ! {
    unsafe { proc_raise(Signal::Abrt); }
    loop {} // should not ever be reached, but required for the `!` return type
}
Enter fullscreen mode Exit fullscreen mode

I then also change the two error/panic handlers to use this now:

#[alloc_error_handler]
pub fn alloc_error_handler(_: core::alloc::Layout) -> ! {
    abort()
}

#[panic_handler]
fn panic_handler(info: &PanicInfo) -> ! {
    print(&format!("{}\n", info)).unwrap();

    abort()
}
Enter fullscreen mode Exit fullscreen mode

And it works! Well, kind of.

Hello, world!
panicked at 'explicit panic', src/main.rs:31:5
Error: failed to run main module `target/wasm32-wasi/debug/wasi-testing.wasm`

Caused by:
    0: failed to invoke command default
    1: proc_raise unsupported
       wasm backtrace:
           0: 0xa692 - <unknown>!wasi_testing::abort::h7ce8efa81892bb51
           1: 0x4b6a - <unknown>!rust_begin_unwind
           2: 0xacf8 - <unknown>!core::panicking::panic_fmt::h3e7a769e079b4878
           3: 0xa3d6 - <unknown>!core::panicking::panic::h0c30bb50c6755362
           4: 0x8431 - <unknown>!__original_main
           5: 0xb136 - <unknown>!_start
       note: run with `WASMTIME_BACKTRACE_DETAILS=1` environment variable to display more information
Enter fullscreen mode Exit fullscreen mode

proc_raise isn't supported by wasmtime yet, but it is part of the WASI API, so I should probably use it as it's the best method at the moment, and may be supported at some point.

And that is the hellish journey that I took to get WASI working with #![no_std] in Rust.

Here is the full code at this point:

#![feature(alloc_error_handler)]
#![allow(dead_code)]
#![no_std]
#![no_main]

extern crate alloc;
extern crate compiler_builtins;

use alloc::format;
use core::panic::PanicInfo;
use wee_alloc::WeeAlloc;

#[global_allocator]
pub static ALLOC: WeeAlloc = WeeAlloc::INIT;

#[alloc_error_handler]
pub fn alloc_error_handler(_: core::alloc::Layout) -> ! {
    abort()
}

#[panic_handler]
fn panic_handler(info: &PanicInfo) -> ! {
    print(&format!("{}\n", info)).unwrap();

    abort()
}

#[no_mangle]
pub extern "C" fn __original_main() -> i32 {
    print("Hello, world!\n").unwrap();
    0
}

#[no_mangle]
pub extern "C" fn exit(code: u32) -> ! {
    unsafe {
        proc_exit(code);
    }
    loop {} // (hopefully) never reached
}

fn abort() -> ! {
    unsafe {
        proc_raise(Signal::Abrt);
    }
    loop {} // should not ever be reached, but required for the `!` return type
}

#[repr(u16)]
#[derive(Copy, Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub enum Errno {
    Esuccess,
    E2big,
    Eacces,
    Eaddrinuse,
    Eaddrnotavail,
    Eafnosupport,
    Eagain,
    Ealready,
    Ebadf,
    Ebadmsg,
    Ebusy,
    Ecanceled,
    Echild,
    Econnaborted,
    Econnrefused,
    Econnreset,
    Edeadlk,
    Edestaddrreq,
    Edom,
    Edquot,
    Eexist,
    Efault,
    Efbig,
    Ehostunreach,
    Eidrm,
    Eilseq,
    Einprogress,
    Eintr,
    Einval,
    Eio,
    Eisconn,
    Eisdir,
    Eloop,
    Emfile,
    Emlink,
    Emsgsize,
    Emultihop,
    Enametoolong,
    Enetdown,
    Enetreset,
    Enetunreach,
    Enfile,
    Enobufs,
    Enodev,
    Enoent,
    Enoexec,
    Enolck,
    Enolink,
    Enomem,
    Enomsg,
    Enoprotoopt,
    Enospc,
    Enosys,
    Enotconn,
    Enotdir,
    Enotempty,
    Enotrecoverable,
    Enotsock,
    Enotsup,
    Enotty,
    Enxio,
    Eoverflow,
    Eownerdead,
    Eperm,
    Epipe,
    Eproto,
    Eprotonosupport,
    Eprototype,
    Erange,
    Erofs,
    Espipe,
    Esrch,
    Estale,
    Etimedout,
    Etxtbsy,
    Exdev,
    Enotcapable,
}

pub type Size = u32;
pub type Fd = u32;
pub type ExitCode = u32;

#[repr(C)]
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub struct Ciovec {
    pub buf: *const u8,
    pub buf_len: Size,
}

#[repr(u8)]
pub enum Signal {
    None,
    Hup,
    Int,
    Quit,
    Ill,
    Trap,
    Abrt,
    Bus,
    Fpe,
    Kill,
    Usr1,
    Segv,
    Usr2,
    Pipe,
    Alrm,
    Term,
    Chld,
    Cont,
    Stop,
    Tstp,
    Ttin,
    Ttou,
    Urg,
    Xcpu,
    Xfsz,
    Vtalrm,
    Prof,
    Winch,
    Poll,
    Pwr,
    Sys,
}

#[link(wasm_import_module = "wasi_snapshot_preview1")]
extern "C" {
    pub fn fd_write(fd: Fd, ciovecs: *const Ciovec, n_ciovecs: Size, n_written: *mut Size)
        -> Errno;

    pub fn proc_raise(sig: Signal) -> Errno;

    pub fn proc_exit(rval: ExitCode);
}

fn print(s: &str) -> Result<Size, Errno> {
    assert!(
        s.len() <= u32::MAX as usize,
        "please don't store >4GB of text in a string, thanks"
    );
    let ciovec = Ciovec {
        buf: s.as_ptr(),
        buf_len: s.len() as u32,
    };
    let ciovec_ptr = &ciovec as *const Ciovec;
    let mut n_written: Size = 0;
    let errno = unsafe { fd_write(1, ciovec_ptr, 1, &mut n_written as *mut Size) };
    if errno == Errno::Esuccess {
        return Ok(n_written);
    }
    Err(errno)
}
Enter fullscreen mode Exit fullscreen mode

And the Cargo.toml:

[package]
name = "wasi-testing"
version = "0.1.0"
authors = ["ThePuzzlemaker <tpzker@thepuzzlemaker.info>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
wee_alloc = "0.4.5"
compiler_builtins = { version = "=0.1.43", features = ["mem"] }

[profile.dev]
panic = "abort"

[profile.release]
panic = "abort"
Enter fullscreen mode Exit fullscreen mode
💖 💪 🙅 🚩
thepuzzlemaker
James [Undefined]

Posted on June 3, 2021

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

#![no_std] with WASI is more complicated than I thought it would be
rust #![no_std] with WASI is more complicated than I thought it would be

June 3, 2021