Gesture Recognition at Your Fingertips

Gesture inputs for electronic devices have become increasingly popular as a means of enhancing user experience and interaction with technology. These intuitive interfaces offer a hands-free and natural way to navigate through digital content, particularly in situations where physically touching the device may not be feasible or desirable, such as when hands are occupied or when the user is focused on another task, like driving. By enabling users to control devices through simple movements or gestures, these systems provide a more seamless and convenient user experience, making them particularly valuable in various settings, including automotive, gaming, and virtual reality applications.

However, despite their numerous benefits, gesture input systems also present several challenges, especially when integrated into portable devices used on the go. Privacy concerns arise because third parties can easily observe the movements associated with gestures, raising questions about data security and the unauthorized collection of sensitive information. Additionally, the high power consumption associated with some gesture recognition technologies, especially those that leverage cameras, can significantly impact the battery life of portable devices, limiting their usability and convenience, particularly in situations where access to power sources may be limited. Furthermore, the use of gesture inputs in public spaces can be socially awkward, and users may feel self-conscious or uncomfortable performing gestures in front of others, potentially hindering the widespread adoption of these systems in certain settings.

As a general rule of thumb, we want our portable electronics to seamlessly integrate into our daily routines, functioning unobtrusively and without imposing unnecessary distractions or complications. A trio of researchers at Keio University in Japan believe that gesture recognition systems should also provide their services in an equally transparent manner. Towards that end, they have developed a fingertip-mounted input device that they call TouchLog. By using the thumb to trace out a gesture on the tip of the index finger, one can interact with devices while preserving their privacy, and in a socially acceptable manner. Further, the technologies leveraged by the team allow TouchLog to operate at low levels of energy consumption.

A number of devices have been developed in the past that are either in the form of a ring or a sensing pad that covers the fingertip. While they have proved their worth in many respects, they also have some nagging issues. Namely, they lack the ability to collect information about contact pressures, or they hinder the ability of the finger to register natural tactile sensations. TouchLog circumvents both of these issues by instrumenting the top of the index finger with an array of photo-reflective sensors. These sensors can detect deformations of the skin, and with the help of a machine learning algorithm, recognize when specific gestures have been performed, and can also determine the amount of pressure that was applied. The finger pad itself is not instrumented, so normal sensations can be felt.

A random forest classifier was trained to recognize eleven different types of finger microgestures, like double taps and swipes in various directions. Then a small user study, consisting of ten participants, was conducted to assess the utility and accuracy of the system. A brief training session was provided for each individual, after which they were asked to try each gesture 20 times while measurements were collected from the TouchLog device. On average, the classifier was found to be able to identify the correct gesture in 91.1% of trials. By analyzing the results, the team discovered that the misclassifications occurred because the algorithm had a difficult time distinguishing the difference between gestures involving straight and curved lines. Additional experiments were conducted to assess the system’s ability to measure contact forces. The TouchLog measurements were found to correspond well with ground-truth measurements obtained from a load cell.

It is known that the photo-reflective sensors can be impacted by the presence of sunlight, which may cause issues using the device outdoors. Differences in lighting, or movement of the hand while performing gestures, may also cause problems. These factors were not assessed in this study, so the performance of TouchLog under real-world conditions is not well understood. The researchers intend to explore these issues in the near future.