Chen Liang Experiments with Arduino_GFX on ESP32 to Find the Fastest, Smoothest Video Format

Chen Liang has published a guide to playing back video, with audio, on an ESP32-based microcontroller using the Arduino_GFX library — and has included some handy benchmarks showing which format gets you the best result.

Chen starts by walking through the various advantages and disadvantages of the Espressif ESP32 for video playback — including the SPI buses theoretically capable of driving a 320x240 16-bit display at 60 frames per second, an SD card bus, I2S for audio, and over 100kB of RAM for buffering, but without the processing power that would be required to decode MP4 format video files.

The solution: Converting the files into something the ESP32 can decode, but what?

Chen looks at three formats in the tutorial: RGB565, a raw format for 16-bit colour data; the Graphics Interchange Format (GIF); and Motion JPEG (MJPEG). For audio, raw pulse-code modulation (PCM) and MPEG Layer 3 (MP3) are investigated — and examples are given for converting existing video and audio into each format using FFMPEG.

The most interesting part of Chen's experiment lies in the benchmarking: Running the sample code on a LilyGo TTGO ESP32-based development board, Chen found that MJPEG proved the fastest format with a stable 30 frames per second playback rate with PCM audio - a rate that could be increased, he notes, but is already at the upper limit of what the LCD panel can display.

Switching to compressed MP3 audio dropped things somewhat to a still-acceptable 24 frames per second, but moving away from MJPEG saw higher losses: GIF with PCM audio could play at only 15 frames per second, while RGB565 with PCM audio performed the worst at nine frames per second — which, Chen notes, is likely due to the amount of data it needs to stream from SD Card storage.

The full guide, with wiring instructions and source code, can be found on Instructables.