Big Advancement in Tiny Machine Learning

Thanks to their low cost and minimal power consumption, microcontrollers are ubiquitous in IoT devices to handle device control and data processing tasks. Those benefits come at a cost, however—microcontrollers have far less processing power and memory as compared with processors found in desktop computers or even smartphones. A typical microcontroller will have tens to a few hundreds of kilobytes of memory, which pales in comparison to the gigabytes of memory available to your phone. How then, with these constraints, are we to leverage machine learning, and all of the benefits it offers, on our pint-sized IoT devices?

A new proposal for supercharging tiny machine learning has recently surfaced in a collaboration between MIT and National Taiwan University. Called MCUNet, the method can produce lightweight neural networks tuned for deployment on microcontrollers. The models provide accurate results while simultaneously keeping power consumption and resource utilization low.

The team developed a two-pronged approach to the problem. They blended an efficient neural architecture search (TinyNAS) with a lightweight inference engine (TinyEngine). TinyNAS automatically handles constraints (e.g. device, latency, energy, memory) while maintaining low search costs to minimize model parameters and generate compact neural networks. Developed to complement TinyNAS, TinyEngine is an inference engine optimized to run neural networks on tiny devices. The engine removes all code that is not absolutely essential and has been shown to reduce memory usage by 2.7 times and improve inference speed by 22% as compared with existing tools.

Applying the new technique to image classification with an ImageNet model on a microcontroller, the team was able to achieve an accuracy of 70.2%. This represents an improvement over the current state-of-the-art accuracy, while also reducing memory usage by 3.6 times.

A common task for IoT devices involves the recognition of wake words. Accordingly, the team evaluated the performance of MCUNet in this area. They found their new method to have greater accuracy, with 2.8 times faster inference speed and 2.6 times less memory usage, than existing methods.

MCUNet has advanced the state-of-the-art in tiny machine learning and looks poised to play a role in making ubiquitous machine learning possible in the future.