(also published on my blog)

Does it run DOOM?

It's becoming the defacto question for any piece of technology with a display on it.. fridges, printers, all sorts of things can run DOOM.

So why not this shiny new ESP32S3 based electronic badge I have just received from ElectroMagnetic Field 2022? It has a decent dual core 32bit Xtensa MCU, 8MB of RAM, 8MB of flash storage (partitioned up - see below) and an ST7789 colour pixel display.

Oh, and a joystick :)

Similar work & self-imposed constraints

There is a port of DOOM for an ESP32, proving that it can be done, however I have added additional constraints on myself as follows:

• Co-exist with the Micropython environment that comes pre-installed on the badge.
• Create an app that can be distributed from the TiDAL Hatchery

So what does the micropython runtime environment look like?

• It's based on the Espressif IoT Development Framework (IDF), which defines a storage partitioning scheme that I cannot replace (but I can use to my advantage..)
• ESP-IDF, and thus Micropython are not designed as an OS with runtime application loading/unloading in mind..
• Micropython is extensible at runtime, via pre-built .mpy format modules, which can be compiled from C.

..and the Hatchery format?

• A folder named after the app (eg: /apps/Doom), that must be a python module (ie: contain an __init__.py file), plus any other files required to run.
• The __init__.py must define an instance of an App class (or subclass) that the badge shell can locate / load.

Cunning plan #1, and failure.

How hard can it be to simply compile up DOOM as an .mpy extension, patch in the I/O (display, files, user-input) back to Micropython and load it up?

It turns out the Micropython extension format is very homebrew (for perfectly good reasons), and comes with significant limitations compared to an off-the-shelf application format such as ELF or COFF, in particular:

• No declared data, there is no support for loading a .data segment as part of a module.
• No static global variables (you know, the bedrock of most C programs!) as it cannot re-locate references when loading into memory, only visible globals can be used.

I also had to choose a starting version of DOOM, with an emphasis on portability. This turned out to be the easy bit, enter DOOM generic, which happily compiled and ran first time on my Linux desktop ;)

Oh, and of course I had to write a new scaler for the video to reduce 320x200 => 240x135.

The grind..

So, first remove all declared data from the DOOM sources, by moving initialisation into suitable xx_Init() functions if they are available, or adding them if not. ~2000 lines of code (LoC) changed, a number of ugly corner cases simply removed (like being able to change key bindings) and it runs on the desktop again - phew!

Now remove that static keyword from all the global variables, and prefix them all with a local module abbreviation to avoid collisions. Another ~2000 LoC changed, and it still runs on the desktop - double phew since this took several days!

Chuck all those changes into a large git patch file, and save it away. Done.

Tooling fail.

Compile everything according to the Micropython supplied tooling. Discover it cannot link certain object files - grrr :(

Fix Multiple, bugs in Micropython.

Chuck all the Micropython bug fixes into a git patch file and save that away. Done now?

Runtime fail.

Errr nope. It turns out Micropython makes more assumptions about how code is compiled,
in particular that all things are relocatable in memory via a single Global Offset Table (GOT). DOOM makes different assumptions, and repeatedly crashes in unpredictable locations due to memory corruption.

After a few days trying to debug this fiasco (with printf of course since I cannot use a debugger due to proprietary module formatting and a special loader), I give up.

Cunning plan #2, and success (eventually!)

Back to basics

Having dismally failed to run anything so far, I decide to start at the other end, building up from a Hello Mum! program that relies only on the built-in ESP-ROM (and is compiled without the Micropython "special tooling" that is causing me problems), designed to run as the first user-provided code on the device, in ESP-IDF parlance, as the "secondary boot loader". Thus is born phlashboot, the shortest program I can write that says 'Hi' without crashing.

After some trial & error with watchdog timers, this works - relief!

A new plan forms

Remember I mentioned a Partition table that ESP-IDF uses? This puts application code in a specific location within flash^ (one of two 'over-the-air' or OTA update partitions), along with a header that defines where in memory it runs, including if it's memory mapped. The normal boot flow of an ESP-IDF application is that the secondary loader applies the defined memory mapping or copies code/data into RAM (hello .data!), then jumps to a defined entry point. Neat.

This is how Micropython itself get's going, can I re-use the same flow to load/run my application at known memory locations outside Micropython, without having to use the broken .mpy module format? Baby steps ensue..

^ for those of you wondering why not memory map from the filesystem part of flash? The MMU is not granular enough for this, hence having to map from a specific 'OTA' partition offset.

Gently does it..

I adapt phlashboot so it doesn't expect to be the bootloader, instead it looks like an OTA application, which the ESP-IDF secondary loader is happy to memory map (mmap) and run, next step..

After a great deal of reading through technical manuals and rummaging in the code, creating the romread test application and generally poking at (and crashing) my device, I understand how the memory manager works, what I can and cannot touch and in theory how I can memory map and execute another app within micropython. There is also a fair amount of investigation into the Micropython memory layout, see memstuff folder! Thus is born doomloader.mpy a short and separate Micropython module that pokes at the MMU to load an OTA app into Micropython-safe address space. It works :D, next step..

DOOM is back baby!

I can now use standard tooling, to build a standard application and run it on device "underneath" Micropython, so back to DOOM. There is some fiddling with compiler options and load addresses to get a sensible OTA application out, which fits easily into a flash (~460k, OTA partitions are 2MiB), the existing plan for connecting in I/O is still intact(!) so in theory it should work... and it doesn't.

It turns out that DOOM expects a little more POSIX compliance from it's C library than Micropython provides, in particular the printf implementation is lacking many features. The good part is that because I'm building an entirely separate binary application, I can use someone elses printf, and finally, after a lot of pain, it runs!

The colours are awful. I can't actually control anything yet. I'm deliriously happy :D

Final fixes

The colour being mangled turns out to be an endian difference between the MPU and the display device, easily determined by looking at the code for the display driver, which byte-swaps everything when drawing into the device, unless it's blitting a buffer, when it assumes it's already swapped. Fixed.

Providing user input is a case of finding the right DOOM keys to fake when pressing joystick buttons or others on the badge.

Show me the code!

To horribly mis-quote another of my favourite games come get some baby!

..OR..

Grab the built objects from the hatchery

The required demo video..