COMP 554 Fall 19 TinyML Final Project: Wake Word Detection

Team members

Yucheng He (yh43)

Quan Zhou (qz19)

Bowen Yang (by11)

Kena Du (kd39)

Architecture

We followed the same setup as Chapter 7: Wake word detection: Building an application in the TinyML book. The device microphone on Nano 33 gets audio input from the user constantly, and the feature provider converts the raw audio data into spectrograms for our model. Below attached the output of the feature provider the illustration for "yes" and "no" separately.

After we got the spectrogram, which is a two-dimensional array, we feed it into the model as a 2D tensor. Our pre-trained model uses CNN to help make the prediction. The final output is displayed on the onboard LED and the serial output.

Deployment

We’ll be deploying the code to Arduino Nano 33 BLE Sense (https://oreil.ly/ztU5E). Arduino Nano 33 BLE is the only Arduino board that has an onboard mic. The Arduino Nano 33 BLE Sense also has a built-in LED, which is what we’ll be using to indicate that a word has been recognized.

To deploy this example, here’s what we’ll need:

• An Arduino Nano 33 BLE Sense board

• A micro-USB cable

• The Arduino IDE

We downloaded the Tensorflow Lite library to the Arduino IDE, then we fetch the example "micro_speech" from the File-Examples tab to load. We then select our connected Arduino Nano 33 BLE port. Then we click the "upload" button to load the pre-trained model to our board. To test the program, try saying “yes”! When it detects a “yes”, the LED will remain solid for around 3 seconds.

Project Demo

Future work

We see that from the demo it successfully recognized most of the commands except for "Hello". When the device heard "Hello", instead of recognizing it as "unknown word", it recognized as "no". We think that the reason is that the "llo" syllable is very similar to "no". In the future, we could potentially only feed in 1-second spectrograms that have a clear start and clear end so that no partial words are going to pass into the model.

Reference

WARDEN, PETE.TINYML: Machine Learning with Tensorflow on Arduino, and Ultra-Low Power Micro-Controllers. OREILLY MEDIA, 2020.

https://www.amazon.com/TinyML-Learning-TensorFlow-Ultra-Low-Power-Microcontrollers/dp/1492052043/ref=sr_1_1?keywords=tinyml&qid=1575689679&sr=8-1