TinyML: Magic wand

We built a digital “magic wand” that can be waved to cast a variety of letters. It takes multidimensional sensor data as input data and its output will be a simple classification to indicate the class to which the movements belong. We use the magic of deep learning to transform plain numerical data into meaningful information!

The wand reacts to each spell by lighting an LED. Moreover, it will also output information to its serial port, which can be observed on the terminal.

Prologue

Gesture recognition is a topic in computer vision, which is aimed at interpreting human gestures via mathematical algorithms. Gestures can originate from any bodily motion or state but commonly originate from the face or hand.

In computer interfaces, two types of gestures are distinguished. We first consider online gestures, which can also be regarded as direct manipulations like scaling and rotating. In contrast, offline gestures are usually processed after the user interaction with the object. In this project, we focus on offline gestures.

Our Magical Model

Our gesture-detecting model is a convolutional neural network, weighing in at around 20 KB, that accepts raw accelerometer values as its input. Each value is a 32-bit floating-point number that indicates the amount of acceleration in that direction.

The model was trained on four gestures performed by numerous people. It outputs probability scores for four classes: one representing each gesture as shown in the figure below.

Magical Factory

Our magic wand is based on the magical factory of Nano 33 BLE Sense (without headers). The Nano 33 BLE Sense (without headers) is Arduino’s 3.3V AI-enabled board in the smallest available form factor: 45x18mm!

Magic Show

We built the application basically by following the procedure in Chapter 12 and Chapter 13 of the book "Tinyml: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers". We eventually made the magic happen on an RHEL 7 Platform!

Our experiment was executed on a desktop with an Intel(R) Xeon(R) CPU E3-1245 v5 clocked at 3.50GHz and 32 GB of memory.

Epilogue

One interesting direction is to experiment with the threshold values to make the gestures easier or more difficult to perform (at the cost of more false positives or negatives). We are also going to play with models for different gestures, such as the "star", "circle", "triangle" and even "umbrella" in the Squid Game series!