Audio ML Platform for Raspberry Pi Pico with Arduino Mbed

Audio ML Platform for Raspberry Pi Pico with Arduino Mbed

Integrating audio machine learning on Arduino platforms running Pico Mbed OS presents significant challenges, primarily due to the reliance on external audio libraries such as CMSIS-DSP, and TensorFlow Lite for Microcontrollers (TFLM). These libraries, while powerful, introduce complexities in compatibility and resource management, as they are not natively aligned with Arduino’s lightweight abstraction layers or the constrained runtime environment of Mbed OS.

The integrated platform offers a powerful advantage by seamlessly connecting advanced audio and IoT capabilities with the vast ecosystem of the Arduino community, enabling developers to extend thousands of features with minimal effort.

Main features:
- Pi Pico (RP2040) running with Arduino Pico Mbed OS
- Integrate TensorFlow Lite micro
- Integrate CMSIS-DSP + CMSIS_6 Core (note: existing Arduino_CMSIS-DSP does NOT support RP2040 CM0+ MCU)
- Integrate Kalman filter on ML post-processing
- Support I2S 24-bit microphone
- Support Wiznet W5500 Ethernet connectivity
- Detect fire alarm sound and texting user via WhatsApp

This project demonstrates how a Raspberry Pi Pico, paired with a Wiznet W5500 Ethernet Controller and an INMP441 microphone, can detect fire alarm sounds and send alert messages to the user via WhatsApp using an Ethernet connection.

System diagram

Hardware
The following components are required for this project:
- Raspberry Pi Pico
- Wiznet W5500
- TDK InvenSense INMP441
- PC with Arduino v2.3.6+ IDE and Arduino libraries installed (see Software setup section)
- mobile to generate fire alarm sound and receive WhatsApp message

PCBA photo

Wiring
Wrap wiring between "Pi Pico - W5500" and "Pi Pico - INMP441" as below

Software setup
- Install Arduino IDE v2.3.6+ for Arduino
- Install ArduTFLite, by Spazio Chirale
- Install UrlEncode, by Masayuki Sugahara
- Install SimpleKalmanFilter, by Denys Sene
- Install ArduProf v2.2.2+, by teamprof
- Install arduino-eventethernet, by teamprof
- Install ArduCMSIS_DSP, by teamprof
- Clone this github code by "git clone https://github.com/teamprof/arduino-pico-audio-ml"

Build firmware
- Follow the instruction on https://www.callmebot.com/blog/free-api-whatsapp-messages/ to get the APIKey
- Install Arduino IDE and libraries according to section "Software setup"
- Clone this github code by "git clone https://github.com/teamprof/arduino-pico-audio-ml"
- Open the pico-audio-ml.ino in Arduino IDE
- Open the secret.h file and replace the placeholder values with your mobile number and the API key provided by CallMeBot

#define MOBILE_NUMBER "<MobileNumber>"
#define APIKEY "<ApiKey>"

- On Arduino IDE, click menu "Tools" -> "Board: Rasberry Pi Pico" -> "Arduino Mbed OS RP2040 boards" -> "Rasberry Pi Pico "
- On Arduino IDE, click menu "Sketch" -> "Verify/Compile"
If everything goes smoothly, you should see the following screen.

- Connect Pi Pico to PC, upload firmware by clicking menu "Sketch" -> "Upload"
If everything goes smoothly, you should see the following screen.

- Launch "Serial Monitor" on Arduino IDE, press "RESET" button on the board, you should see the following screen.

- Connect an Ethernet cable between the board and router, you should see the following screen.

- Generate an alarm sound with an audio output device, the on-board green LED should be turned on and you should see the following screen.

- You should receive a WhatsApp message on your mobile phone a few seconds later. See video demo for reference.

Software flow

When the device boots, it launches ThreadNet to initialize the Wiznet W5500 Ethernet controller. Once the Ethernet network is established, ThreadNet signals ThreadApp with an EthUp event. Upon receiving this event, ThreadApp blinks the on-board LED five times and then launches ThreadAudio.

ThreadAudio initializes the INMP441 I²S microphone and starts the audio inference engine. The inference results are continuously sent from ThreadAudio to ThreadApp.

ThreadApp evaluates these results, and if an alarm condition is detected, it instructs ThreadNet to send an alert message via WhatsApp.

Video demo
Video demo is available on

00:01 power up device => red LED turns on
00:03 Ethernet network is up => green LED blinks 5 times
00:08 generate fire alarm sound
00:10 device detects fire alarm sound => green LED turns on, send WhatsApp message "alarm sound detected"
00:14 mobile received WhatsApp message "alarm sound detected"
00:19 stop alarm sound
00:22 device detects no alarm sound => green LED turns off, send WhatsApp message "no alarm"
00:31 mobile received WhatsApp message "no alarm"
00:37 generate fire alarm sound
00:40 device detects fire alarm sound => green LED turns on, send WhatsApp message "alarm sound detected"
00:48 mobile received WhatsApp message "alarm sound detected"
00:48 stop alarm sound
00:50 device detects no alarm sound => green LED turns off, send WhatsApp message "no alarm"
00:56 mobile received WhatsApp message "no alarm"

License
- The project is licensed under GNU GENERAL PUBLIC LICENSE Version 3