Engineers Turn to Ultrasound to Build a Wrist-Worn High-Precision Gesture-Tracking Sensor
New wearable platform demonstrated for everything from controlling virtual reality environments to directing the motion of a robot arm.
Engineers at the Massachusetts Institute of Technology (MIT) have developed a wrist-based wearable capable of tracking the muscles, tendons, and ligaments responsible for hand movement in real-time — delivering virtual reality (VR) and robot control with impressive precision.
"We believe this is the most advanced way to track dexterous hand motion, through wearable imaging of the wrist," says corresponding author Xuanhe Zhao of the team's work. "We think these wearable ultrasound bands can provide intuitive and versatile controls for virtual reality and robotic hands."We think this work has immediate impact in potentially replacing hand tracking techniques with wearable ultrasound bands in virtual and augmented reality. It could also provide huge amounts of training data for dexterous humanoid robots."
Gesture-tracking is not a new concept, but existing approaches have always been somewhat lacking: camera-based systems can't track when your hand moves out of sight; inertial measurement can only track the simplest of gesture types; gloves provide per-finger tracking but can be cumbersome and bulky; electromyographic sensors measure muscle impulses directly, but the signal is often noisy and difficult to tie to particular gestures.
The team's solution: a wristband that uses ultrasound to peer through the skin and observe the moment of the muscles and tendons visually. "The tendons and muscles in your wrist are like strings pulling on puppets, which are your fingers," explains co-author Gengxi Lu. "So the idea is: each time you take a picture of the state of the strings, you’ll know the state of the hand."
The current sensor prototype is roughly the size of a smartwatch, with a supporting electronics package the size of a cellphone, and capable of detecting 22 degrees of freedom — recognizing gestures from American Sign Language through to grasping balls, scissors, and pencils. These gestures were then used to control a virtual reality interface, and later a physical robotic hand — with volunteers using gesture control to have the robot pick out a simple tune on a piano.
The team's work has been published in the journal Nature Electronics under closed-access terms.
Main article image courtesy of Melanie Gonick.