A Sound Solution to Hand Tracking

Most of the activities that we are engaged in in our daily lives involve our hands in one way or another. Whether one is picking up a cup, typing on a keyboard, performing a delicate surgery, or playing a grand piano, the hands play a crucial role. It should come as little surprise then that there is considerable interest in developing a practical means of tracking the position of the hands in three-dimensional space. This information has many applications in the area of human-computer interaction, for example, where it can be utilized for contextual awareness or as an input method.

Today’s hand tracking systems have their share of issues, however. These issues largely stem from the fact that they most commonly rely on cameras to glean information about the hands. Images are prone to have missing data due to either partial or full occlusions of the hands. These can result from the presence of any number of objects that get in the way, or even from one portion of the hand blocking another from view. Given the high degree of flexibility of the hand, these occlusions can render existing systems inaccurate in some cases.

Aside from accuracy problems, camera-based systems also generally require a significant amount of computational horsepower for processing, and the requisite energy for their operation, which is impractical for mobile devices. Moreover, many privacy-related issues arise from the use of an always-on camera. Accordingly, there is a need for alternative solutions that can overcome these issues with present hand tracking technologies.

An interesting solution was recently put forth by a team of researchers at Cornell University. They have developed a wrist-worn device that they call EchoWrist. Rather than relying on a camera, EchoWrist instead uses acoustic sensing to detect hand poses, and also to recognize objects near the hand. This sensing modality requires little power and computational resources, yet it was demonstrated to be quite accurate. The use of sound waves also serves to preserve the user’s privacy, which may help to make it a suitable replacement for camera-based systems in the future.

EchoWrist consists of a silicone band worn on the wrist that is equipped with two pairs of speakers and microphones — one pair is positioned above the wrist, and the other below, to get a full view of the area. These are wired to a custom PCB containing a Nordic Semiconductor nRF52840 microcontroller, an audio amplifier, and a power management module. The device is powered by a small LiPo battery, which can keep it running all day long before a recharge is needed. An onboard Bluetooth Low Energy transceiver allows the wristband to wirelessly communicate with other devices.

The speakers emit frequency-modulated continuous waves, which are inaudible, toward the hand. As the sound waves strike the hand, they are reflected and diffracted and travel back in the direction of the microphone, which captures the sound pattern. This information is then forwarded into a convolutional neural network, which predicts the three-dimensional positions of twenty finger joints, which allows the hand pose to be determined. This algorithm can also recognize many objects that are being held in the hand, as well as other types of hand-based interactions.

EchoWrist was evaluated in a series of trials involving twelve participants. These experiments revealed that the system could track finger positions with a mean joint Euclidean distance error of 4.81 millimeters and a mean joint angular error of 3.79 degrees. The recognition capabilities of the device were also evaluated, and it was found that it could detect a set of a dozen common hand-object interactions, like holding a cup, with 97.6% accuracy.

In the future, the team hopes to integrate EchoWrist with an off-the-shelf smartwatch. Such an integration could extend the benefits of hand tracking to a much wider audience.