Radio Killed the Video Stream

Keep your hands where I can see them! That is the unofficial mantra of traditional, optical-based hand pose tracking systems. Such systems use cameras to capture a stream of images of the hand, then process the images, usually with a machine learning algorithm, to determine the position of joints and other body parts in three-dimensional space. And some of these systems have been very successful, consistently producing highly accurate predictions and enabling cutting edge applications in virtual reality, gesture recognition, and beyond. But as was alluded to earlier, optical tracking technologies require a clear, direct view of the hand. However, under normal circumstances, that view of the hands is frequently occluded by clothing, held objects, or even by other parts of the hands themselves.

This consideration has been a major factor in limiting the adoption of hand tracking technology. Additionally, cameras raise privacy concerns, and since few people want a camera pointed at themselves all day long, the list of acceptable applications, and duration of their use, is limited. A pair of engineers at Carnegie Mellon University have been exploring an alternative approach to the problem of continual hand pose tracking. Using RF signals, they have developed a wearable device called EtherPose that is capable of high-precision hand tracking without intruding on the wearer’s privacy. These signals are also capable of seeing through clothing, darkness, or other objects that would prevent an optical system from getting a good view of the hand.

The wrist-worn EtherPose device has a pair of antennas (front and back-right) that transmit a swept-frequency RF signal. This signal interacts with the hand, then is reflected back to the unit, where the reflected signal’s magnitude and phase shift are measured with vector network analyzers (VNA). There is one VNA per antenna, with each connected to a Raspberry Pi Zero 2 W. The Raspberry Pis run a machine learning model, SciPy’s ExtraTreesRegressor, to predict the 3D position of 21 hand key points to continuously track 3D hand pose position. Similar processing pipelines were created to continuously predict wrist angle position, and also micro-gestures that track the thumb’s position relative to the other four fingers. The entire device is self-contained, with a 3.5 hour runtime using a 16 Wh LiPo battery. Key point predictions can be transmitted wirelessly via WiFi to other devices for integration.

Sounds great, but does it work? To answer this question, the researchers conducted a small user study with a total of nine participants. Each user was first fitted with the EtherPose device, then instructed to sit in front of a computer monitor that directed them to make specific hand poses or gestures. Ground truth measurements were captured with MediaPipe Pose and a webcam for comparison with the predictions made by the EtherPose system. When assessing continuous hand pose estimation, a mean per-joint position error of 11.57 millimeters was observed. When testing out the continuous wrist rotation capability, a mean wrist angular error of 5.87 degrees was found.

EtherPose was proven to be a capable system, but as it currently stands, it has a few areas that need additional work to make it practical outside of the research lab. To be accurate, the wristband needs to be calibrated between different users, and also across sessions of wearing the device. It has also been noted that the system works better when hands are held out from the body, as holding them in close causes interference with the RF signals. This is also a problem when in proximity to any conductive surface, like a steel door. A possible solution to this problem might be pointing additional antennas in the direction of the hands. These issues will take some time to work through, but if acceptable solutions can be found, EtherPose may find its way into future commercial products.