Arpan Mondal's Space Drums Combine Espressif's ESP32 with IMU Sensors and Computer Vision

Maker Arpan Mondal has designed a pair of drumsticks for the minimalist — using Espressif ESP32 microcontrollers and a computer vision model to have them work mid-air, no drums required.

"As a kid, I was into stuff that made zero sense to normal people. While other kids wanted action figures, I was asking my dad for biometric security systems," Mondal says by way of background to the project. "Since my parents weren't going to fund my mad scientist projects, I developed a survival skill: I just built things myself. And this habit stayed with me as I grew into the adult I am today. When I saw a video of air drums online, the kid inside me screamed, 'you have to make one of those!' Naturally, I didn't think twice before deciding to build my own pair. I only start big projects because I severely underestimate how hard they'll be, and I had no clue how many headaches were waiting for me."

The Air Drums themselves are contained in a pair of oversized drumsticks, each housing compact lithium-ion batteries, an Espressif ESP32-S3 wireless microcontroller module, and an six-axis inertial measurement unit (IMU) sensor with accelerometer and gyroscope for tracing movement. A third set of all the components is housed in a kick-pedal — or, rather, mounted to the user's shoe to simulate the same.

What Mondal has designed, over a three-month period, isn't just a device that makes a noise when you wave it around: it tracks the position of the sticks in space in order to map out what the maker calls "a full, invisible, 3D drum kit": a hi-hat to the left, a snare in the right-middle, a floor tom at the right, crash and ride cymbals at the top-left and -right, and the kick drum linked to the user's shoe. "Because the AI [Artificial Intelligence] tracks your shoulders and joints in real-time, the virtual kit actually adjusts to your position. You get a massive, responsive percussion setup without sacrificing bedroom floor space."

Initially, the Space Drums worked entirely based on data from the IMU — but drift meant that the virtual drums would move over time, and with no visual feedback become impossible to play. Research showed that adding a magnetic digital compass probably wouldn't fix the problem well enough, and attempts to use infrared sensors as anchors proved too sensitive to light. The fix: a smartphone feeding its camera feed to a cloud-based posture estimation model, tracking shoulders, elbows, and wrists. "I didn't need the sticks to know where they were anymore," Mondal explains. "The AI would watch my arms, anchor the invisible drum kit directly to my body, and completely eliminate drift forever."

The project is documented in full on Instructables, with source code available on GitHub under the permissive Apache 2.0 license.