Hardware components | ||||||
| × | 1 | ||||
| × | 2 | ||||
Software apps and online services | ||||||
 |
| |||||
| ||||||
If you use voice assistants - Siri, Google Assistant, Alexa - your voice travels to the cloud every single time. It gets recognized there, and the response comes back. Works well enough, but you need internet, there's latency, and essentially you're handing your voice over to someone else's servers. And what about on-device processing? The options are pretty limited right now: either cheap chips that recognize a handful of hardcoded commands like "turn on light, " or beefy processors that drain batteries fast and don't fit into compact devices.
Between these two extremes, there's a gap: recognizing free-form speech right on the device, without the cloud, at low power. Recently, microcontrollers (MCUs) with built-in neural processing units (NPUs) started appearing on the market. I decided to dig in - could a chip like that bridge the gap? I went through available papers and projects - and couldn't find any existing solution that actually does this.
System ArchitectureI implemented the core part of a speech recognition system on an STM32N6 microcontroller with an integrated NPU.
The approach divides recognition into three stages - this helps save RAM and improve overall accuracy:
- Acoustic model — the most resource-intensive part. It takes raw audio from the microphone and converts it into a stream of phonemes (the smallest individual sounds that make up human speech).
- Decoder — assembles phonemes into words using a dictionary and language rules.
- Rescoring — double-checks the result and picks the most likely phrase based on context.
So far I've built and tested the acoustic model — that's the foundation; without it the rest doesn't make sense. I'm working on the decoder and rescoring stages now.
Speech Recognition DemoIn the video, the device runs in real time — I speak arbitrary phrases and the system recognizes them. For demonstration purposes I wrote a simple interpreter that maps phonemes to words. At the bottom of the screen you can see the raw phonemes as the model outputs them. At the top — the words and numbers derived from those phonemes.
An important caveat: the demo currently uses hard-coded phoneme-to-word mapping - basically a lookup table. This significantly limits what it can do. Once a proper phoneme decoder with a language model is in place, accuracy will improve noticeably and the system will be able to recognize any words, not just the ones in the table.
Power ConsumptionThis is probably the most exciting result - the power numbers. The entire device during active speech recognition draws only ~0.215 W:
And this is with zero optimization. I haven't done any power tuning yet - just ran it as-is. The Cortex-M55 core currently only computes the mel spectrogram and doesn't participate in neural network inference, so it could be put to sleep or clocked down. The NPU is only at 10.4% utilization - plenty of room to optimize further.
In a real-world scenario (wake on keyword → recognize → sleep), idle power draw would be minimal. On a battery, a device like this could run for a long time.
Input Signal: Dataset vs. MicrophoneFigure 3 shows a mel spectrogram from LibriSpeech (the dataset the model was trained on), and Figure 4 shows the same clip recorded by the dev board's microphone. The audio was played back through a phone speaker. Even so, the key speech structures are preserved and the model gets a workable signal. The real recording is noisier - no filtering is applied before feeding data into the neural network yet. Noise suppression is one of the next steps on the list.
Accuracy ResultsWhere: dev_clean is the cleaner LibriSpeech validation subset, dev_other is the harder, noisier subset with accents. The 2.5× ratio between them is expected and consistent with published results from other models.
Acoustic model quality is measured by PER (Phone Error Rate) - the proportion of errors in recognized phonemes. Lower is better - it shows how well the model "hears" the sounds that the decoder will later assemble into words.
Training was done on the full LibriSpeech dataset, then the model was quantized to int8 - the accuracy loss is minimal. The numbers in the table aren't from a PC - they're from the actual device: the entire validation set was run through the NPU, results were pulled over UART, and metrics were calculated on a computer. These are real numbers from real hardware.
About that latency: 985 ms is the total end-to-end latency, of which 485 ms is the "lookahead window" (the model peeks ahead in time to predict phonemes more accurately).
Comparison with Other ModelsDirect comparison is tricky: I'm reporting PER (phoneme-level errors), while most papers report WER (Word Error Rate). WER depends not just on the acoustic model but also on the decoder and language model - so these are different things. But for a rough picture, here are results from models of similar and larger sizes:
Worth noting: at 8.5M parameters, this model achieves a PER of 5.51% on test_clean. That's comparable to wav2vec 2.0 Base (5.74%) and HuBERT Base (5.41%) - which are 11× larger. And none of them run on a microcontroller. This model is the only one in the table that runs entirely on an NPU in real time.
Accuracy was measured using an extended phoneme set of 72 symbols, which is a harder task compared to the standard 39-44 phoneme sets.
Projected results with decoder: once a phoneme decoder is added, the expected WER is ~16-25% on the challenging LibriSpeech dev_other subset, with a total system size of around 8-16M parameters.Headroom and Growth Potential
Right now I'm only using a fraction of this chip's capabilities.
RAM is at 18% utilization. It only holds activation parameters — meaning the model could be scaled up roughly 4× without changing the hardware.
NPU is at 10.4% utilization. There's a huge compute margin — room for a more complex model or parallel tasks.
Power hasn't been optimized yet. The current ~0.2 W is with the core running at full clock and no sleep modes. There's significant room to bring that down.
I'm currently working on a phoneme decoder and language model. The decoder will see the probability distribution over all phonemes at each time step and pick the most likely words - that should noticeably boost accuracy.
Also on the roadmap: noise filtering before the neural network. Right now there's none, and in noisy conditions the model has to work with a dirty signal.
Where MCU + NPU Can Replace the CloudLet me set expectations right away: this isn't a replacement for Siri or Alexa. You won't be dictating long texts into a microcontroller - that requires 100+ MB models that simply don't fit on this platform.
But here's what the system can already do that classic keyword spotting (KWS) can't: it recognizes free-form words and phrases. No need to memorize commands. You can say "make it warmer, " "add about five degrees, " or "temperature up" - and all of these get interpreted as the same action. Traditional KWS can't do that: it waits for an exact pattern match.
The sweet spot is voice commands, short messages, and voice data entry -on-device, offline. Here's where I see this being useful:
Smart home without the cloud. Control lights, climate, appliances - not with memorized commands, but with natural speech. "Make the living room cozier, " "too bright in the kitchen, " "a bit warmer in the kid's room" — the system parses the phrase and figures out what you want, which room, and what parameter to adjust. And not a single word leaves your home.
Voice data entry - meter readings, measurement results, codes. You can speak naturally: "pressure one thirty-two, " "gauge reads one thirty-two" — the system extracts the number. When you're wearing gloves or your hands are busy, this makes a real difference.
Industrial equipment. Not just "start/stop" — but commands with parameters: "feed workpiece to position three, " "speed twenty percent." The operator doesn't walk to the control panel — they give voice commands right at the machine.
Medical devices. Blood pressure monitors, glucose meters, wearable bands. A patient can speak context: "after a meal, " "morning, fasting." Easier than pressing tiny buttons, and no smartphone needed nearby.
Warehouse and logistics. A worker with both hands full dictates into a headset: "shelf B-12 — three boxes, " "this one's defective, pull it." The device parses the phrase and extracts the location, quantity, and action.
Vehicles and heavy equipment — where there's no internet: out in the field, on a construction site, underground. The driver speaks naturally: "head back to base, " "switch to eco mode." No need to remember the exact command from the manual.
Children's toys and educational devices. A toy robot that understands not just ten phrases but free-form children's speech — names, numbers, colors. No subscriptions, no Wi-Fi, no sending a child's voice off to some server.
Assistive devices. Wheelchair or prosthetic control via voice. "Go forward, " "a little to the left, " "stop" — works just as well as formal "move forward, " "halt." Here it's especially important that the device is autonomous and can run on battery power for extended periods.
Conclusion: So, Can They Replace It?As of now, the acoustic model works — it recognizes phonemes from free-form speech on a microcontroller, in real time, offline, drawing ~0.2 W. It won't handle text dictation, but for voice commands, data entry, and device control - it's sufficient. Where the cloud is overkill and KWS is too primitive, a microcontroller also wins on cost: the chip is cheaper than a microprocessor, power draw is several times lower, and development is simpler - no heavy hardware platform or complex software stack needed.
Next up, I'm planning to implement a phoneme decoder and language model - to turn phonemes into actual words and phrases. Plus, optimize power consumption and add noise suppression. With each of these steps, MCUs with NPUs will take on more and more tasks from the cloud - wherever autonomy, privacy, and offline operation matter.
Resources
Comments