Hand Gesture Recognition Face-off

Developers are always on the lookout for more intuitive and efficient ways to interact with our electronic devices. As technology has advanced and enabled the development of more practical alternative interfaces, one area that has received a lot of attention is hand-to-face gestures. These gestures, such as touching the face, scratching the head, or rubbing the chin, can be integrated into devices to provide commands to a computer to control various applications.

Unlike traditional input methods like keyboards or mice, which require explicit actions and can be less intuitive, hand-to-face gestures capitalize on actions people already perform regularly and subconsciously. This familiarity reduces the learning curve and can enhance user comfort and engagement. These types of gestures can also make virtual reality experiences feel more immersive.

Existing hand-to-face gesture recognition systems suffer from a variety of limitations. Many are camera-based, which makes them bulky and expensive. These solutions also fail to operate correctly under challenging lighting conditions or in the presence of obstructions. And of course there are always privacy concerns when working with an always-on camera. There are also radar-based solutions to the problem, but they have issues with obstructions as well, and are very sensitive to variations in distance which limits their accuracy.

A new option has just been introduced by a team of engineers at the University of Pittsburgh. Called Mobile Acoustic Field (MAF), the system leverages audio signals generated by bone conduction earphones to create an acoustic field on and near the head. Disturbances in this acoustic field, caused by hand gestures in close proximity to the face, can be measured and decoded with a custom machine learning pipeline. It was shown that a variety of on- and over-face gestures could be recognized by MAF with a high degree of accuracy.

MAF uses commercial bone conduction earphones to transfer mechanical energy to the tissues of the human head. This generates inaudible surface acoustic waves in ultrasound frequencies which also disperse into the air as they travel through the user’s face. Both the surface waves and the leaked waves persist as long as the earphones are playing audio. This, in turn, creates an acoustic field that surrounds the wearer’s head, even as they move about.

A microphone is also included in the system to measure the sound waves produced by the bone conduction earphones. When a hand either touches the face or interacts with the acoustic field surrounding the face, the sound waves are altered before reaching the microphone. A custom algorithm was developed to preprocess the captured audio data before feeding it into a deep neural network. This neural network was trained to classify a variety of hand gestures based on the way that they alter the acoustic field.

To validate their approach, the team tested MAF on a cohort of 22 participants. These individuals were asked to perform four on-face, and an additional six over-face, hand gestures while wearing the device. MAF proved to be pretty accurate, with an average classification accuracy rate of 92 percent being observed in the trial.

As a next step, the team plans to investigate using bone conduction earphones with a built-in microphone, rather than requiring a separate microphone to be taped to the face. They also intend to work toward supporting micro-gestures, as the present set of available gestures is relatively coarse. With refinements such as these, MAF could one day find itself powering any number of device interfaces.