M5Stack-Based Musical Biofeedback System Could Tune-up Stroke Rehabilitation

A trio of researchers from Aalborg University in Denmark have put together a technical framework designed to aid with stroke rehabilitation — by tying the feedback from wearable inertial sensors to synthesized music.

"Given the impact that this had on me and my family," first author Prithvi Ravi Kantan tells IEEE Spectrum of his experience with rehabilitation following his father's brush with viral encephalitis and subsequent partial paralyzation, "it was a revelation when I discovered music therapy, as it gave me a way to channel my professional skills towards a cause that held some personal meaning to me."

Kantan's work, with colleagues Erika G. Spaich and Sofia Dahl, centers around auditory biofeedback, or ABF: The real-time conversion of bodily measurements into sound. Using wearable motion sensors, the proposed system turns movement into music generated through adjustable layered synthesis — providing immediate feedback on the patient's progress during rehabilitation.

The sensors used in the feedback system are off-the-shelf M5Stack Grey microcontrollers, running a sketch written in the Arduino IDE to stream data from their internal inertial measurement units (IMU). These data are converted to Open Sound Control (OSC) message and fed to a JUCE environment while a FAUST script generates a JUCE-compatible digital signal processor.

Exactly where the sensors are worn and in what quantity depends on the patient: Some were fitted with a single sensor to their lower back, others had one sensor on each ankle or one sensor above and another sensor below the knee. The FAUST-based DSP handles performance generation, providing an eight-track stereo ensemble with both melodic and percussive elements — turning movement into "percussion, melody, and harmony in a simplified pop music style," the researchers explain, with options for Dance, Reggaeton, and Slow Rock variants.

In testing with stroke patients in a clinical setting — though without, the team admits, "a rigid protocol" — the devices appeared to be of assistance: Patients responded positively to the music and were able to match its changes to their own movements, though some reported that the devices detected jerky movements where none were believed to be present.

"Our future studies will include rigorous usability studies and clinical trials, where we systematically compare MBF [Musical Biofeedback] strategies and assess ratings of enjoyment, arousal, and perceived agency," the team concludes. "We will also focus on adding DMIs [Digital Musical Interactions] and MBF strategies to the framework, aimed at providing positive feedback that reinforces task-intrinsic perceptual information critical to motor learning (e.g., proprioception, and/or using embodied schemata.)"

"Additionally, we shall focus on DMIs that provide feedback to enhance compensatory mechanisms and strategies to overcome loss of motor function, as opposed to only sonifying deviations from 'desirable' performance."

"It would be extremely rewarding to contribute directly or indirectly to improving rehabilitation outcomes for patients," Kantan says in his IEEE Spectrum interview. "Although there is a lot of research to do before we see the widespread adoption of musical feedback in clinical practice, we at Aalborg University are determined to bring present efforts in the field to fruition in the coming years."

The team's work has been published under open-access terms in the journal IEEE Transactions on Human-Machine Systems, with more information available from IEEE Spectrum's coverage.