60FPS Eink GameBoy Emulator on M5PaperS3

Overview Video

I have made a video about this project. It contains the most information I have below, so if you prefer watching a video than reading, here is it:

Introduction

Recently I got myself a M5PaperS3. It's an ESP32-S3 dev kit shaped like a pocketable e-reader, featuring a 4.7" 960x540 Eink touchscreen. What makes it special compared to other small Eink devkits is that it has got a screen with raw row/column driver interface (also commonly referred to as parallel interface when talking about Eink screens, but it's not the same parallel interface on the LCD screen). This enabled me to drive this screen while bypassing the normal waveform method. This would in turn, making high refresh rate possible, up to 60fps. More on that later. That got me thinking, what kind of fun stuff I cab do with it other than writing an e-reader application? Several years ago, I made a GameBoy FPGA re-implementation, so it's fair to say that I have at least some understanding of the GameBoy architecture. How about combining both?

I searched online, and found no-one have ever attempted building such as thing. As a result, I am really tempted to create the first GameBoy emulator on the PaperS3, and make it actually playable!

Eink Driver

The first thing I started working on is the high-refresh-rate Eink driver. Because this is the component that decides if this project would ever work or not.

Eink are generally perceived as slow: low refresh rate and slow response. This is generally true, because the colored particles inside the Eink panel are just rather slow. It usually take around 100ms to change color. So that's 10Hz. The slow response/ long latency comes from this as a side-effect: the user may need to wait up till 100ms before the image being taken by the screen. If the OS uses triple buffering then it introduces another 100ms of delay.

Many existing fast refresh strategies are basically reducing the driving time. For example, if we only drive the screen for 50ms, that improves the frame rate to 20fps at the expense of lower contrast (because the pixels weren't driven fully). But this method can only gets you so far: even with the latest and greatest Carta 1300 screen with IGZO backplane at +/-24V driving voltage, you can only push it up to 30Hz before the image becomes too washed out to be useable. M5PaperS3 uses an older generation screen (Carta 1000 with +/-15V driving voltage) that tops out at around 15fps.

But there is still way to make it work. You see, Eink is slow, 100ms. But that doesn't mean you need to cap the frame rate to 10Hz to get 100ms driving time. It could also be 6 frames at 60Hz. Actually this is already how it the screen driver works under the hood for various reasons. But the controller would only take a new image every 6 frames, like a global lock, making it behaving like 10Hz. But what if we remove that lock? Making it take a new image every frame so we can unlock 60Hz.

That's totally possible. The tradeoff is that now you will need to keep of track of the timings of each pixel, to make sure they are still being driven for 6 frames in total. This requires adding a new state buffer, that's read from and write to for each pixel. While the global lock method only requires a single counter that's updated once each frame. So it's definitely not free. Depending on the platform, this cost can be prohibitively high (like on a lower end microcontroller, or high end microcontroller but with limited SRAM capacity and limited PSRAM bandwidth)

But we get back is butter smooth 60fps on Eink screens:

That was a demo I did back in 2022 on FPGA. After that, I turned that demo into an FPGA-based Eink monitor, called the Modos Flow. Now we just need to see if we can do the same on ESP32.

Luckily for this project I don't really need to process at full screen pixel rate. GameBoy only has 160x144 pixels. I need 3 pixels to represent 4 shades of grey, duplicating dithered image by 3 times I get 3X pixel perfect scaling + dithered image. This means I only need to process 160x3x144 pixels at 60Hz, a much lower target than 960x540 at 60Hz. This also means all my buffers fit in the SRAM, without having to spilling things to the slower PSRAM. I did some benchmark, current code at this 480x144 resolution is using about 50% of the CPU time. Surely it wasn't particularly optimized, but that also gives a rough estimation of how far we are from full resolution.

I am storing 4 bits states per pixel, counting the state of the pixel. It gets compared with the incoming buffer to decide what to send to the screen. The LCDIF + DMA is used to actually pumping out the pixels to the screen, so the core can calculate next batch of pixels to display The whole thing runs on the 2nd core on the ESP32, each iteration is set to run for around 1/60s, providing a 60Hz refresh rate. Double buffering is also implemented, so the emulator on the first core also have a 60Hz time base, allowing it to render image with vertical sync.

GameBoy Emulator

While I have implemented a GameBoy emulator in the past, that was also on FPGA. GB emulator on microcontroller has been done many many times over, so I decided to use existing emulator instead of reinventing the wheel myself.

I tried 3 existing emulators as the core:

• Peanut GB (https://github.com/deltabeard/peanut-gb)
• Walnut CGB (https://github.com/Mr-PauI/Walnut-CGB)
• CrankBoy (https://github.com/CrankBoyHQ/crankboy-app)

Peanut GB seems to be the most popular emulator option, with decent performance on microcontrollers.

Walnut CGB is a fork of the Peanut GB. It modified the emulator to do 16-bit instruction fetch instead of 8-bit fetch, potentially saving time on memory access, mostly saving time on the expensive emulated memory mapping path by issuing less memory accesses.

CrankBoy is a fork of the Peanut GB + frontend for Playdate project, offering a complete optimized GB emulator for Playdate. They added many optimizations for the port, especially around the memory mapping path, allowing much better performance without the rather unusual dual-fetch architecture from the Walnut CGB.

I tried them out on the ESP32-S3. I tested the emulation speed in various titles. Here are the results:

Performance measured as relative to GameBoy. 1 meaning it's running at (roughly) the same speed as the gameboy

So I used the most performant CrankBoy as the emulation core. With dynamic frameskip (which means, if the emulator is lagging behind, it would only do the emulation but not rendering to the screen, saving a few ms to catch up), it can hit 100% speed in many titles, and rendering at 30-60fps. The actual frame rate depends a lot on runtime situations.

With the current performance, it's rather unrealistic to enable GBC support for now though (which allows the CPU to run at double clock speed). We will need something far better to allow GBC to run at a reasonable speed or letting the GB titles hitting 60fps more consistently.

Sound

All three emulators uses minigb-apu as the sound emulation engine. The GB integrates a 4-channel PSG (programable sound generator). 2 channels of square wave, 1 channel of sample, 1 channel of noise. 4 channels are then mixed together to produce the sound output. In a real GB, some part of the mixing happens in the analog domain, but generally for emulators it's easiest to just do everything as PCM samples, at 32.768kHz sample rate. All modern sound cards playback PCM samples anyways.

This is however not the case on the PaperS3. The PaperS3 only has a buzzer, capable of generating a single tone when driven by the PWM. One well known technique is to use the PWM DAC to output PCM samples. Basically just send PCM samples right into the PWM module as the duty cycle (with waveform shifted by 128 to be DC-biased). However, on ESP32-S3, PWM module couldn't be fed by the DMA. It needs to be either polled, or generating an interrupt at the sample rate to update the duty cycle register. Both would require me to reserve some CPU cycles just for outputting the sound. (Note that by default, interrupt handling on ESP32 is rather slow, due to the context switching/ interrupt dispatching. You could interrupt the core at 32kHz but it's going to be rather expensive). Instead I used the PDM output capability of the I2S peripheral. It was intended for connecting to DAC/ amplifiers accepting PDM, but you could also just add a RC filter to it and itself becomes a DAC. I implemented this first, and it worked right away. But the issue is... it's just way too quiet. If you are in a very quiet environment then it's kinda audible, but otherwise it may as well just be muted. I thought it might be because the modulation frequency was too high (4MHz), driving the BJT into linear region or something, so I also tried just 32kHz PWM, but it doesn't make any meaningful difference, so I am sticking with the PDM.

But I felt like that was not enough. Yes it does have sound, but if it's not audible in many situations... it's not that useful. To make loud enough buzz, it pretty much has to be a single tone, something it was actually designed to do. So, how about actually just use a single tone? You see, on GameBoy's PSG, first 2 channels are straight square wave. The sample channel could also be approximated with a square wave (assuming the game is using it to output something like a sine wave, triangle wave, etc.) If we also use square wave in place of noise (which honestly is a bit far fetched), then it's just 4 channels square waves. While the buzzer can only do one tone, it's possible to switch between multiple sound channels rapidly to create a pseudo polyphony effect. This is a well known technique on platforms that also only has a single buzzer, such as an IBM compatible PC without any soundcard. Regarding the results, I think it worked surprising well and bad at the same time. The good part, it's loud and clear, game music are still very recognizable, and it's overall interesting to listen to. The bad part, well the GameBoy is never supposed to be sound like that.

To ensure all these don't eat into the limited CPU resources for emulation, I put the entire PSG emulation on the 2nd core. Based on profiling on selected games, PSG emulation generally runs for up to 2ms per frame depending on the sound complexity.

Input

The input is rather straight-forward. The M5PaperS3 has a touch screen, so it's natural to just show a touchscreen joypad on the screen.

For serious gamers (???), I have also added experimental Bluetooth LE joypad support. Though it doesn't actually fully implement the HID protocol, it only guesses about the report format. As a result, the compatibility is quite poor. I think as a demo this is fine, but this definitely can see some improvements.

JIT 2 HOLIDAY

CrankBoy already provides rather good performance. But it's by no means solid & stable 60fps. During game play, it frequently dips below 60fps due to the CPU emulation being too slow. While it might sound crazy that a 240MHz Xtensa LX7 cannot emulate 4MHz GameBoy CPU at full speed, but this is what we have right now. To push this further, some serious optimizations are necessary.

One option is to use JIT (just-in-time) dynamic recompiler. A JIT recompiler recompiles instructions for GameBoy CPU into Xtensa LX7's instruction set. So the ESP32 can execute them natively, without having to run through an interpreter.

Developing a real stable and performant JIT recompiler is a big project on its own. So what I am have here is more or less just a proof-of-concept, built by my friend Yukidama using Claude Code: https://github.com/JIT2Holidays/gb-jit-xtensa. While the game compatibility is largely untested, with the games that boots, it already shows some decent speed ups. This partially proves that, combined with the 60Hz screen driver, it should be possible to produce a firmware that can more run GameBoy games at close to 100% speed.

Save Data

The game boy doesn't have any internal non-volatile storage, save state is stored on the cartridge. So, to allow save data to persist across reboot cycles, it just needs to be saved to the SD card. Simple, right? Not quite.

Saving the data to the SD card is the easy part, the question is when. Some games have an explicit save menu, but this may not always be the case. The hardware on the cartridge is usually just a battery backed SRAM. It may not even be battery backed, in case the game just wants some extra RAM. The emulator might be able to guess based on the heuristics of cartridge RAM write patterns and decide when to flush the data to the SD card (for example, in the simplest form, if there hasn't been updates for the last 2 seconds, flush it to the SD card). But that doesn't sound reliable to me: What if I turned it off before it finishes flushing? What if the game keeps writing to the cartridge RAM every second so it never triggers a flush? As a player, save data lost is among the things I least want to see. Of course I am not saying it's impossible to do auto-save. I think it's possible to do it, and when done properly it could work very well. Though I haven't attempted it yet.

Another option is to have some explicit event that would trigger a save. The most common one being: closing the emulator. However this is not possible on the M5PaperS3, because the power button on the S3 directly cuts power rather than trying to do a graceful shutdown. The S3 doesn't even know it would be shut down, let alone flushing the save state to the SD card. So I had to put a save button there, explicitly asking the user to press it turning the machine off. It's ugly but it works.

To make the button less useless, I added another feature: quick save and quick load. This basically stores and loads the state of the entire emulated GameBoy, allowing quickly creating checkpoint and return to the checkpoint. Save and quick save are the same button, pressing it does both. So when next time you pick up the PaperS3, you can just use quick load to get back to the game immediately.

Conclusion

In conclusion, I think the end result is a pretty decent/ usable GameBoy emulator for the M5PaperS3, pushing the Eink to its limit as well. It's sad that PaperS3 has already reached EOL, hopefully we get a proper replacement for it soon!