Researchers from the Tokyo Institute of Technology, Carnegie Mellon, the University of St. Andrews, and the University of New South Wales have developed a wrist-worn device that accurately identifies the 3D hand pose of the wearer. To put that another way, they created a device that would provide precision hand tracking and gestures in AR and VR applications, without the need to wear bulky gloves or use controllers. According to the team, this is the first vision-based, real-time 3D hand pose estimator that garners data from the dorsal hand region, or rather the movement of bones, muscle, and tendons.
“Our system achieves a mean joint-angle error of 8.81° for user-specific models and 9.77° for a general model. Further evaluation shows that our system outperforms previous work with an average of 20% higher accuracy in recognizing dynamic gestures, and achieves a 75% accuracy of detecting 11 different grasp types,” the researchers write in their recently paper. “We also demonstrate three applications that employ our system as a control device, an input device, and a grasped object recognizer.”
The Back-Hand-Pose system was designed using a wrist-worn, wide-angle RGB camera to capture movement in the dorsal hand region. The system is linked with a Leap Motion camera for collecting ground truth hand pose, and the data is piped into a computer outfitted with an NVIDIA RTX 2080 Ti video card and an Intel i7-9700 CPU, where the researchers' DorsalNet neural network can accurately recognize dynamic gestures by identifying movement on the back of the wear’s hand.
The team demonstrated their Back-Hand-Pose system using three applications, with the first offering smart device control. The wearer can interact with a VR/MR controller using a single finger. They also showed it would be possible to use it as a virtual mouse or keyboard, as well as classify different grasp-types just by imaging the back of the hand, meaning the system should be able to be used for VR object grasp recognition.
The project is still in the early stages of development, but it does look promising as the next AR/VR/MR object interaction and gestural tool to do away with controllers and gloves finally.