BABL: A Baby Monitor Powered by tinyML and Edge Impulse!

Idea 🤔 💡

With Edge Impulse's recent announcement of support for the popular Arduino Nano 33 BLE Sense, the exciting world of Machine Learning is now easily accessible to even novice Arduino developers. To demonstrate how quick and easy it is to get started and create results with Edge Impulse Studio, I wanted to target a conventional application that could be dramatically improved with machine learning. Baby monitors work by transmitting audio wirelessly from a transmitter in the baby's room to a receiver which the parent can monitor. The problem is that any sound - whether it's a baby crying out with hunger - or a baby playfully babbling as it falls back asleep - produces the same results for the overtired parent: disruption of their precious few moments of sleep. By leveraging machine learning, the system can recognize crying from other noises, and behave accordingly. In this first project, we will see just how quickly such advanced functionality can be achieved with tinyML and Edge Impulse.

Getting Started 🔰 👩‍💻

Edge Impulse has fantastic documentation, which I highly recommend you go back and review in detail as time allows, but for now, our goal is to get up and running and monitoring our babies as quickly as possible, so here is my CliffsNotes version; I'm using Ubuntu 20.04 but if you run into problems on a different version or platform, let me know in the comments and I'll be more than happy to help you out.

First, you must install the Arduino CLI - if you are not familiar, this glorious piece of software brings the functionality of the Arduino IDE that we all know and love to the command line - meaning you can automate things, build and deploy from other IDEs and editors, and so on. And the tools from Edge Impulse require it in order to work properly, so follow the installation guide, or (on Linux):

curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sh
echo 'export PATH=~/bin:$PATH' >> ~/.bashrc
. ~/.bashrc
arduino-cli help

If, after this, you are presented with a list of available commands, then you are all set. If not, feel free to drop your error into the comments below and I'll gladly help you get it sorted.

OK, onto the Edge Impulse CLI! These tools are distributed as an npm package, and require Node.js 10 or higher. If you're not familiar with Node, npm packages, and Node version management, I highly recommend you check out nvm, which lets you easily switch between the Node version required for your project. Here's how I used nvm to install the EI CLI with the latest stable version of Node:

node --version
#v8.16.0
nvm ls
nvm use stable
node --version
#v12.16.1
npm install -g edge-impulse-cli
edge-impulse-uploader --help

If this results in usage instructions, you are all set! If you run into problems, check out the troubleshooting section in the docs, or drop a comment below and I'll be more than happy to help figure things out with you!

With all of the CLI tools sorted, it's time to get the Arduino itself set up! On Linux, connect your Nano via micro-USB, press reset twice to enter bootloader mode, and grab and install the latest firmware as follows:

cd ~/Downloads
wget https://cdn.edgeimpulse.com/firmware/arduino-nano-33-ble-sense.zip
mkdir arduino-nano-33-ble-sense
unzip arduino-nano-33-ble-sense.zip -d arduino-nano-33-ble-sense
cd arduino-nano-33-ble-sense
./flash_linux.sh

I ran into a problem with flash_linux.sh and had to hand-edit it for it to work successfully, but that was a beta version of the software, and I'm told it's been fixed since. If all went well, you should receive a message saying the firmware was flashed and it's time to start the daemon. Press reset one last time, but don't actually start the daemon yet. One thing that the docs never seem to mention anywhere is the (admittedly obvious, but worthy of mention) requirement to create and Edge Impulse account and project. If you haven't already done this, then the config steps of the daemon won't actually make sense. Once you've created an account and project, go ahead and start the daemon:

edge-impulse-daemon

and follow the on-screen prompts to configure your device. After you see:

[WS ] Connecting to wss://remote-mgmt.edgeimpulse.com
[WS ] Connected to wss://remote-mgmt.edgeimpulse.com
[WS ] Authenticated

go back to your project in the Studio, click Devices, and you should see your device listed and connected. At this point, setup is complete, and it's time to move onto the fun stuff! 🚀

Development Process 💪 💻

With your device connected, you can use Edge Impulse Studio'sData acquisition functionality to interactively gather audio data in real time. In the Record new data section of this page, ensure your device is selected, enter noise for Label, set Sample length to 16000ms (16 seconds - the maximum possible with the Arduino Nano 33 BLE Sense's available memory), select Built-in microphone for Sensor, and set Frequency to 16000Hz. We need around 5 minutes of background noise data, so take your Nano and laptop to your baby's room, press Start sampling, and repeat around 20 times until you have ~5 minutes of data for your noise class. There's a handy Data collected tally at the top of the page to help you keep track, and you can click on an individual sample to see its waveform or play back its audio. By definition, our background noise samples aren't going to be very interesting:

Now repeat this process for your crying class - just update the Label, and keep everything else the same. My baby was asleep while I was working on this project, so I found some baby crying sound effects. In order to ensure I had some unique data to test with, I made sure to not use all of the available samples when building my dataset. Once you're up to 10 minutes total, (5 for each class) it's time to create what EI call an "Impulse".

On the left nav, select Impulse design, and you will be presented with the "blocks" interface, the first of which is pre-filled for us with Time series data. The audio samples will be analyzed in Windows, and we want to determine if our baby is crying within one second of data, so we set Window size to 1000ms. Instead of only looking at each one second sample once, we will set the Window increase to 300ms in order to extract several overlapping blocks from each second of audio. For our Processing block, click and add Audio (MFCC), and then Neural Network (Keras) for our Learning block, and click Save Impulse.

The Impulse design section on the left nav will now display some new options, with grey bullets next to them, indicating that they are still to do. Click on MFCC and configure your parameters as follows (should be the default):

Next, from the Generate features tab, click the Generate features button. When the process is complete, you'll get a nice graph on the Feature explorer which allows you to visualize your dataset; there's actually quite a lot of overlap in ours - perhaps because the audio between actual cries is extremely similar to our noise audio.

With our features generated, you'll see NN Classifier is the last grey bullet in our process, so click on that in the right nav, and set:

• Number of training cycles (epochs): 300
• Learning rate: 0.00005
• Minimum confidence: 0.7

and leave the Neural network architecture as-is. Click Start training and wait for the job to complete.

Once the job has completed, you'll see some very helpful summary data:

86.3% accuracy sounds good, but let's just give it a try for ourselves with the Live classification tool in the left nav. With your device still connected, select it from the Device dropdown, set Sample length to 5000ms, and select Built-in microphone for Sensor, and set Frequency to 16000Hz as before. Click Start sampling, and play one of the unused baby crying samples. Review the results to confirm that the classification is accurate. You can also use the Classify existing test sample functionality to work with existing data. Our results are looking pretty good here:

For more extensive testing, you can record more test samples via the Test data tab of Data acquisition, and test them using the Model testing feature.

Further analysis of individual samples can be done using the same tool as during Live classification above, here labeled Show classification under the three vertical dots (⋮) next to a sample.

OK, everything is looking good, so let's get it deployed to our Arduino! From the Deployment tab, select Arduino library under Create library. You could also choose Arduino Nano 33 BLE Sense under Build firmware for a pre-built image you can just flash to the device, but we want to customize our code, so we'll use the library. Click the Build button, which will result in a .zip file being generated and downloaded.

Open the Arduino IDE, and import the library (for example, ei-baby-arduino-1.0.1.zip) using Arduino's Library import feature, under Sketch > Include Library > Add.ZIP Library...

Then select nano_ble33_sense_microphone under File > Examples > baby Inferencing (Edge Impulse) (your library name will differ - this was based on the project being called baby). This will provide you with a basic sketch that will work with your trained audio data. Hit Ctrl+U to Compile and Upload the sketch to your attached Arduino Nano 33 BLE Sense. Open Arduino's Serial Monitor under Tools and ensure that your connection speed is set to 115200 baud. As the sketch repeats the loop() function, you will see output like:

Recording...
Recording done
Predictions (DSP: 501 ms., Classification: 18 ms., Anomaly: 0 ms.):
crying: 0.07422
noise: 0.92578
Starting inferencing in 2 seconds...
Recording...
Recording done
Predictions (DSP: 502 ms., Classification: 17 ms., Anomaly: 0 ms.):
crying: 0.15234
noise: 0.84766
Starting inferencing in 2 seconds...
Recording...
Recording done
Predictions (DSP: 502 ms., Classification: 17 ms., Anomaly: 0 ms.):
crying: 0.08203
noise: 0.91797
.
.
.
Starting inferencing in 2 seconds...
Recording...
Recording done
Predictions (DSP: 501 ms., Classification: 18 ms., Anomaly: 0 ms.):
crying: 0.84766
noise: 0.15234
Starting inferencing in 2 seconds...
Recording...
Recording done
Predictions (DSP: 501 ms., Classification: 17 ms., Anomaly: 0 ms.):
crying: 0.86328
noise: 0.13672
Starting inferencing in 2 seconds...
Recording...
Recording done
Predictions (DSP: 501 ms., Classification: 17 ms., Anomaly: 0 ms.):
crying: 0.85156
noise: 0.14844

As you can probably guess, the above output corresponds to an initial period of silence/noise, followed by the playing of an audio sample of a baby crying. Note the "Starting inferencing in 2 seconds..." - this is just to give you a chance to see what's going on - it can be eliminated in production projects by removing delay(2000); in loop().

Neat! But...what if you don't want to stay continually tethered to your laptop and serial monitor? The Arduino Nano 33 BLE Sense has a nice RGB LED onboard, so let's use that to indicate our state without having to monitor serial output! The color of the RGB is determined by the values written to 3 pins - one each for red, green, and blue, so let's establish some easier-to-remember names for those:

static signed short redPin = 22;
static signed short greenPin = 23;
static signed short bluePin = 24;

Somewhat counterintuitively, the values range from 255 (off) to 0 (on) - so let's create a helper function to flip those around to the more typical 0-255 intensity used to define RGB colors:

void setLed(signed short red, signed short green, signed short blue) {
// values are inverted i.e. 255 == off
analogWrite(redPin, 255 - red);
analogWrite(greenPin, 255 - green);
analogWrite(bluePin, 255 - blue);
}

Let's add some arrays to keep track of our colors for each class (this could probably be made into a 2d-array, and extended for arbitrary numbers of classifications, but that might make things harder to understand):

static signed short classificationRed[] = {255,0,0};
static signed short classificationGreen[] = {0,255,0};
static signed short classificationBlue[] = {0,0,255};
static signed short classificationNone = EI_CLASSIFIER_LABEL_COUNT;

So, our 0th class, if we take the 0th index of each color (read top-to-bottom above, even though the values happen to end up being the same from left-to-right), is 255,0,0, or bright red. The 1st and 2nd classes are bright green and bright blue. In setup(), let's go ahead and start the LED off in our "none" state (blue):

setLed(classificationRed[classificationNone],
classificationGreen[classificationNone],
classificationBlue[classificationNone]);

Let's add that to the loop() too, before the classification loop (for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {). Now, inside the loop, let's set our LED based on classification, provided we have greater than 70% confidence:

// overwrite LED if confident
if (result.classification[ix].value > 0.7) {
setLed(classificationRed[ix], classificationGreen[ix], classificationBlue[ix]);
}

This will result in the LED turning red for our 0th class, crying, or green for our 1st class, noise - or remaining blue if neither classification has more than 70% confidence.

The source for this project can be found on GitHub, and you can review the diff to see the precise code required to add the RGB functionality to the original example.

Field Testing 👨‍🔬📐

Here's a quick demo of the LED status changing from green to red with the detection of crying - as well as some indeterminate (blue) sounds, which were actually in the 50-60s, so maybe we should drop our threshold a bit...

Results and Conclusions ⚗️ 📊

The ability to incorporate machine learning into even modest microcontrollers will transform our world over the next decade. Even something as simple as a baby monitor can be revolutionized - from a blunt tool which wakes parents on any sound, to a precise instrument for detecting actual events which need responding to. And this is just the first evolution in a multi-project series... 😀💭🔮

Next Steps/Future Enhancements 🔜 🔮

Transforming the ability to recognize whether a baby actually needs assistance is just the beginning. In subsequent projects, we'll show how to retrain the model to eliminate further false positives, such as innocuous baby babbling, as well as how to create a mobile app that connects to our device via BLE and displays notifications when the child is in need of attention (and critically, doesn't, when she does not, so parents can get their precious sleep). If you have other ideas, feel free to drop them in the comments below, and they might make it into an upcoming evolution of this project! 🤔🚀