A Dynamic Duo Enhances Hand Tracking

While there are still some areas that need further work, current hand pose recognition systems are exceedingly good. They can accurately locate the position of the hand and finger joints in three-dimensional space even under difficult conditions. The detected hand poses can be leveraged in building natural and intuitive user interfaces for all manner of electronic devices.

But how many commercial devices are controlled by hand gestures? You likely have to give it some serious thought to name any — that is, if you can name any at all. So if hand pose recognition systems are so good, and they have the potential to be so useful, then why are they so rarely included in real-world products? The issue is one of practicality. The systems that perform the best tend to rely either on cameras or bulky and cumbersome wearable sensing equipment.

Whether it is because they are too clumsy, or because they are seen as an unacceptable intrusion of privacy, people just do not want to use existing hand pose recognition systems. A trio of researchers at the East China Normal University in Shanghai has just proposed a solution to this problem. By collecting data from multiple locations — both the wrist and the finger — they have shown that they can estimate hand poses without cumbersome hardware or privacy concerns. And they can do it with a high degree of accuracy.

The system they developed fuses signals from two types of wearable sensors — an inertial measurement unit (IMU) embedded in a smart ring worn on the thumb, and a single-channel surface electromyography (EMG) sensor integrated into a smartwatch worn on the wrist. Unlike existing camera-based systems, this configuration preserves user privacy and avoids the limitations of line-of-sight technologies. It also sidesteps the discomfort and complexity of gloves or multi-ring setups.

The ring prototype includes a Nordic NRF52832 microcontroller, which has a 2.4 GHz Bluetooth Low Energy transceiver and a 32-bit ARM Cortex-M4 processor. There is also an onboard 9-axis IMU that captures accelerometer, gyroscope, and magnetometer data via I2C. The ring is powered by a 15 mAh arc-shaped lithium-ion battery. With this hardware, the ring delivers real-time motion tracking for over an hour on a single charge — all while consuming just 56 milliwatts of power.

The EMG sensor embedded in the watch captures electrical activity from muscles located on the back of the forearm. These signals are collected using gold-plated copper electrodes and processed through an onboard amplification and filtering module. This module increases signal strength by a factor of 1,000 and filters out environmental noise. The resulting analog signal — ranging from 0 to 3 volts — reflects muscle activity levels and is digitized using the ADC input of an Arduino Uno, which then sends the data for analysis.

A transformer-based machine learning model with time encoding and cross-modal attention mechanisms was developed to estimate the 3D pose of the hand, given IMU and EMG measurements. A weighted loss function ensures the resulting pose estimates are spatially accurate, biomechanically plausible, and kinematically smooth.

Testing the system with a dataset of 19 hand gestures from 10 participants yielded some promising results. A mean per-joint position error of just 0.75 cm and an average joint angle difference of 6.815° was observed in cross-user evaluations. This combination of performance and practicality may finally help hand gesture control to go mainstream.