Klingon Meets TinyML

Qapla'! If you don't speak Klingon, that's roughly equivalent to saying "success" — an appropriate greeting for a project that has accomplished something few would have thought possible. Rather than simply running a pre-trained language model on an ESP32-S3 microcontroller, this open source project teaches the chip to train its own transformer from scratch. And, because ordinary languages apparently weren't nerdy enough, it learns Klingon along the way.

Most microcontroller-based AI projects focus specifically on inference. A model is trained on a powerful GPU, compressed, copied onto an embedded device, and used to generate predictions. But this project doesn’t offload the training process to a more powerful machine. The ESP32-S3 starts with no knowledge whatsoever, initializes its weights randomly, tokenizes its training data, performs forward passes, computes cross-entropy loss, runs backpropagation, updates weights using stochastic gradient descent with momentum, and periodically saves checkpoints to flash memory.

That’s a lot to ask of an ESP32, so the developer had to do a lot of optimization and forgo nearly all modern conveniences. There is no PyTorch, no TensorFlow, and no automatic differentiation library to lean on. Every derivative used during backpropagation is written out manually in C, then verified against finite-difference gradient checks to ensure correctness before being deployed to real hardware.

That hardware is an $8 ESP32-S3 N16R8 development board with PSRAM, paired with a small SH1106 OLED display that shows the training loss as the model slowly improves over time. The transformer consists of a single attention block with one attention head, roughly 319,000 parameters, a vocabulary of just 31 characters, and a context window of 32 characters. Training for 5,000 optimization steps takes roughly two days while powered from nothing more than a USB phone charger.

Klingon wasn’t chosen purely because the project’s creator is a fan of Star Trek. The language's compact vocabulary and well-defined grammar make it a good candidate for a memory-constrained experiment like this. The model isn't expected to produce flawless conversational Klingon, but it successfully learns many of the language's characteristic prefixes and suffixes, generating text that often looks structurally correct even if its meaning is questionable.

No one is claiming this tiny transformer competes with modern large language models, but there are still scenarios where on-device training can come in handy. When an embedded device cannot arrive pre-trained because the data it needs simply doesn't exist until after deployment, for instance, techniques like this one could save the day. For more details on the project, check out the GitHub repository.