Running Faucets Custodian

Water conservation is the most cost-effective and environmentally sound way to reduce our water demand. Although water is used all over the world to generate electricity, using less water conserves a vast amount of energy annually. It also helps to alleviate pressures on our sewage and drainage systems. Most important, however, water conservation helps prolong the lifespan of lakes and rivers which are crucial to the health of ecosystems around the world.

This project tries to alleviate water wastage in households by reminding you but it also helps in seniors' safety. Poor memory is only one of the many unpleasant experiences that accompany old age and these problems can have far-reaching implications on the comfort and security of seniors. Dementia is one of the most common neurological problems associated with the elderly. Imagine a case of seniors leaving the faucet on. The kind of water damage that might ensue is simply unimaginable. Not to mention lots of safety concerns such as electrocution and drowning. Also, sometimes kids or even adults forget to turn off the faucet after use. It also adds up to your monthly water usage bills.

In this project, I have built a proof-of-concept of an IoT-enabled non-Intrusive low-powered edge device that can monitor running faucets using an inbuilt microphone and alert by beeping a buzzer and optionally can send you an email.

Hardware Selection

We are using a Nordic Thingy:53 which is built around the nRF5340 SoC. The application core processing power (128 MHz) and memory size (512 KB RAM) of its dual Arm Cortex-M33 processors enables it to run embedded machine learning models directly on the device. The slim-profile case and inbuilt microphone and battery make it portable and a good fit for our purpose.

It has many onboard sensors (see the internal layout in the image below) but we will be using the PDM microphone, Buzzer, and RGB LED in this project.

For sending notifications over the BLE connection and sending an alert email using WiFi, we are using an ESP32 development board (M5StickC PLUS) with the Thingy:53.

Setup Development Environment

For the development, I am using a macOS but all other supported platforms should have a similar setup process. First, we need to download nRF connect for Desktop from here: https://www.nordicsemi.com/Software-and-tools/Development-Tools/nRF-Connect-for-desktop/Download.

The nRF Connect for Desktop is a cross-platform tool that enables testing and development with nRF5340. Please follow the installation guide in the link above. When the installation is completed, open the app and click on the Toolchain Manager and choose nRF Connect SDK v2.0.0.

By default, the SDK is installed in the /opt/nordic/ncs directory in MacOS. After installation, click on the Open Terminal which opens up a command line terminal with all environment variables initialized to get started quickly with the development.

Model creation and training

We will use Edge Impulse Studio to train and build a TensorFlow Lite model. We need to create an account and create a new project at https://studio.edgeimpulse.com. We are using a prebuilt dataset for detecting whether a faucet is running based on audio. It contains 15 minutes of data sampled from a microphone at 16KHz over the following two classes:

• Faucet - faucet is running, with a variety of background activities.
• Noise - just background activities.

We can import this dataset to the Edge Impulse Studio project using the Edge Impulse CLI Uploader. Please follow the instructions here to install Edge Impulse CLI: https://docs.edgeimpulse.com/docs/cli-installation. The datasets can be downloaded from here: https://cdn.edgeimpulse.com/datasets/faucet.zip.

$ unzip faucet.zip
$ cd faucet
$ edge-impulse-uploader --clean
$ edge-impulse-uploader --category training faucet/training/*.cbor
$ edge-impulse-uploader --category testing faucet/testing/*.cbor

You will be prompted for your username, password, and the project where you want to add the dataset. After uploading is finished we can see the data on the Data Acquisition page.

In the Impulse Design > Create Impulse page, we can add a processing block and learning block. We have chosen MFE for the processing block which extracts a spectrogram from audio signals using Mel-filterbank energy features, great for non-voice audio, and for the learning block, we have chosen Neural Network (Keras) which learns patterns from data and can apply these to new data for recognizing audio.

Now we need to generate features in the Impulse Design > MFE page. We can go with the default parameters.

After clicking on the Save Parameters button the page will redirect to the Generate Features page where we can start generating features which would take a few minutes. After feature generation, we can see the output in the 3D Feature Explorer.

Now we can go to the Impulse Design > NN Classifier page where we can define the Neural Network architecture. We are using a 1-D convolutional network which is suitable for audio classification.

After finalizing the architecture, we can start training which will take a couple of minutes to finish. We can see the accuracy and confusion matrix below.

For such a small dataset 99.5% accuracy is pretty good so we will use this model.

Deployment

Since we will be deploying the model at Nordic Thingy:53, on the Deployment page we will choose the Create Library > C++ library option. For theSelect optimization option, we will choose Enable EON Compiler which reduces the memory usage of the model. Also, we will opt for the Quantized (Int8) model. Now click on the Build button and in a few seconds the library bundle will be downloaded to the local computer. For inferencing, we have created a new Zephyr project which borrows most of the code from the Edge Impulse Thingy:53 firmware project and can be cloned from the GitHub repository.

$ git clone https://github.com/metanav/thingy53_running_faucet_detector.git

Now unzip the downloaded library bundle and copy the following directories to the ei-model directory under the project root directory.

• edge-impulse-SDK
• model-parameters
• tflite-model

Build and flash the firmware

Execute the command below in the terminal by opening it using the Toolchain manager as described in the Setup Development Environment section.

$ cd /opt/nordic/ncs/v2.0.0/thingy53_running_faucet_detector
$ west build -b thingy53_nrf5340_cpuapp

There are many ways to flash firmware to the Nordic Thingy:53. We are using a J-Link Edu Mini, an external debug probe, to flash the firmware by following the steps below.

• Open the connector cover on the side of the Nordic Thingy:53 (see the image below).
• Use a JTAG cable to connect the Nordic Thingy:53 to the debug out port on a 10-pin external debug probe.
• Power on the Nordic Thingy:53; move the power switch SW1 to the ON position.
• Connect the external debug probe to the computer with a micro-USB cable.

After setting up the hardware, execute the command below.

$ west flash

Build ESP32 firmware

We also need to build and flash ESP32 (M5StickC PLUS) to provide WiFi functionality. Please download and install Arduino IDE by following the instructions here: https://www.arduino.cc/en/software. Also, we need to install M5StickC-Plus Library for Arduino. Once the installation is finished, download the sketch file (Central_Device_Faucet_Running.ino) from the GitHub repository (https://github.com/metanav/thingy53_running_faucet_detector) and upload it to the M5StickC PLUS using the Arduino IDE.

Application workflow

On the startup, M5StickC PLUS connects to the Thingy:53 over a BLE connection. The M5StickC PLUS acts as a central device and the Thingy:53 acts as a peripheral device. The application samples 1000 ms of audio data at a 16KHz sample rate from the onboard PDM microphone and runs the inferencing continuously. When it detected the running faucet sound, the onboard RGB LED turns green. If the faucet is continuously running for 10 seconds the LED turns red and the buzzer starts beeping. We can see the inferencing status as Normal or Running Faucet on the M5StickC PLUS display screen. The M5StickC PLUS sends an email using Wi-Fi if the faucet is running continuously for another 30 seconds. To avoid false positives, the last 10 predictions are stored in a ring buffer to make a decision. The inferencing logs can be monitored over USB serial connection at a 115200 baud rate. The inferencing rate is below 20ms per sample which is blazingly fast for such a low-powered microcontroller.

Inferencing Demo

Fun fact

We use about 5 gallons of water if we leave the water running while brushing our teeth. Please save water and save our planet Earth! Watch the video below to see how Thingy:53 reminds Manasi to not waste water.

Conclusion

This is an easy-to-use and convenient device that respects users' privacy by running the inferencing at the edge. This project showcases that a simple neural network can be used to solve complex problems with signal processing done correctly and run on low-powered resource-constrained devices, like Nordic Thingy:53.