SoundSense

What it does

SoundSense provides directional audio and speech awareness to users with hearing impairments.

• The acoustic sensor array detects noise and voice activity relative to the user.
• The Arduino Uno Q uses a speech to text model to process captions from the sounds.
• Captions and direction of sound amplitude is displayed to the user.

How we built it

• Build an acoustic array of sound sensors, preferably microphones. We used the KY-037 Sound Sensor.
• Connect sensors to Arduino Uno Q.
• Arduino Uno Q publishes sound sensor data over serial to the Arduino Uno Q's Debian Linux MPU, using Rami's DirectBridge to open a socket instead of using Arduino App Lab.
• Build the Whisper.cpp model on the Debian machine. We used one of the tiny models. As a backup we had an API to ElevenLabs to provide higher quality speech to text.
• If you chose to build the acoustic array out of microphones, pass the microphone data over serial to the Debian side. We had an external microphone connected to the Debian machine to pick up vocals since the sound sensors are cheap and output only binary (noise or no noise).
• Build.wav files out of microphone data, periodically sending them to the Whisper Model.
• Take the output captions from Whisper and build tcp messages to send to the AR Unity Project.
• Build an AR Unity project that shows users the direction of the highest sound amplitude (using smooth gradient indicators on the edges of user's vision).
• Build caption display boxes that appear in the AR view.

Challenges we ran into

• We want to run a local AI model on the Arduino Uno Q’s debian MPU, but if the model is not already supported by Arduino App Lab then it becomes challenging. App Lab creates a docker container when users run their apps. The docker compose file used to define the container is not exposed to the user, leaving us unable to set up our local model when running through App Lab.
• Our solution is to use Rami’s DirectBridge to create a socket between the Arduino’s MCU and MPU. This allows us to pass data from the sensors into the model running on the debian linux side.