TinytinyLM: Running a Transformer Language Model on a ESP32

This project is a fork/port of TinyLM by Viacheslav Sierbov, adapted to run on a plain ESP32 without PSRAM.

Large language models live in datacenters with gigabytes of memory. I wanted to see how far that could be pushed down — so I ported an existing ESP32-S3 transformer LLM project onto a plain ESP32 with no external PSRAM at all.

The Hardware

My board: an ESP32-D0WD-V3, dual-core, 240MHz, 4MB flash, and only ~323KB of usable heap. No PSRAM. That's a massive step down from the original project's target — an ESP32-S3 with 8MB of external RAM to play with. Every design decision here comes back to that one constraint: fit inside on-chip SRAM, or don't fit at all.

A 0.96" SSD1306 OLED over I2C handles output — model stats and generated text streamed live as it runs.

The Model

I built a tiny decoder-only transformer, sized as small as I could make it while still functioning:

• 64-dim embeddings
• 4 layers, 4 attention heads
• RoPE (rotary position embeddings) instead of learned position embeddings
• 64-token context window
• Byte-level tokenizer — 256 possible byte values as tokens, no BPE, no merge tables

Total: 180, 800 parameters. Tiny by any modern standard, but every one of them had to justify its place in the memory budget.

I trained it on CPU — no GPU needed at this scale — for 4, 000 steps (~6 minutes), landing at a validation perplexity of ~2.98 on 8.2M tokens seen.

Getting It Small Enough

Trained weights got quantized down to 4-bit integers, each block carrying its own scale factor for dequantizing at inference time. That took the model from several hundred KB in full precision down to about 88KB of pure parameter data.

The real surprise was discovering where the memory actually goes at runtime. It's not the weights — it's the KV cache. Kept in full fp32 precision for accuracy during generation, it alone eats 128KB of my ~149KB total SRAM budget. Quantizing the weights harder barely moves the needle on total memory use; the real lever is context length, since the KV cache scales directly with it.

Porting the Firmware

The original firmware assumed PSRAM everywhere. Porting it meant:

• Swapping every heap_caps_malloc(MALLOC_CAP_SPIRAM) call for plain malloc()
• Rebuilding the flash partition table for a 4MB chip (model binary lives in a 2.6MB partition at offset 0x140000)
• Reading the model straight from its raw flash partition — no filesystem layer, to avoid overhead on a chip with no PSRAM to cache reads
• Rewiring the display driver for my specific OLED (I2C address 0x3C, SDA=21, SCL=22)
• Regenerating every prompt token ID for the new byte-level vocabulary (the old ones were built for a 32, 768-entry BPE vocab and were just invalid here)

Results

It works. The board loads the quantized model from flash, runs the full transformer forward pass in on-chip SRAM, and streams generated text to the OLED at roughly 19–42 tokens/sec.

Given a prompt like "Once upon a time, " the output is repetitive — expected at 181K parameters, and honestly part of the fun. This was never about getting great text out. It's about proving a fully trained, quantized transformer can run start to finish on a $3 microcontroller with zero external memory.