Inspired by Futurama's Holophoner, the Soundweaver Aims to Turn Mood Music in Visualizations

Cornell Digital Systems Design Using Microcontroller (ECE4760) students Rafael Gottlieb and Eric Zhang have built a Raspberry Pi Pico-powered live music visualizer — using as their inspiration the Holophoner from sci-fi series Futurama.

"In the show, there is an instrument known as the Holophoner," the pair explain. "As the instrument is played, the musician's thoughts are displayed via a hologram from the instrument, creating different scenes. In the episode 'The Devil's Hands Are Idle Playthings,' Fry (the main character) creates an opera using the instrument, with the scenes being portrayed through his playing of the Holophoner. This project aims to create an instrument similar to the Holophoner, named the Soundweaver."

The Soundweaver, it must be noted, does not create holograms based on the player's thoughts — but it does analyze music, received through a microphone, and build visualizations accordingly. The analysis is relatively simple: a fast Fourier transform (FFT), which calculates frequencies and amplitudes of incoming audio, with a historical comparison which aims to capture the "mood" of the music. This then drives a visualization which takes inspiration from nature: "the animation was completed using an algorithm imitating a starling murmuration," the students explain.

Both analysis and visualization through the starling algorithm, known as Boids, take place on a Raspberry Pi Pico microcontroller board — making full use of the RP2040's dual Arm Cortex-M0+ cores by running the FFT analysis on one core and the animation, which is output to a VGA display, on the second. "We found that slow piano music worked the best," the pair say of their creation, "since the frequencies of the music are very distinct, and it was clear when notes are being played versus not."

While admitting that the Soundweaver "did not meet expectations," primarily due to an inability to visualize amplitude changes and a very blocky image, the students declared the project "a great step in connecting sight and sound, [which] connects what we hear to what we see, connecting our senses in harmony together."

More information is available on the project homepage, along with a wiring diagram and full source code.