Tapping Outside the Box

Google researchers have advanced the state of the art in off-screen tap recognition with their machine learning-based approach named TapNet.

Nick Bild
3 years ago β€’ Machine Learning & AI
TapNet (πŸ“·: M. Huang et al.)

Touchscreens provide a simple and intuitive interface for interacting with our smartphones and tablets, but are not without their drawbacks. For example, touchscreen-only interfaces limit users to single-hand operation of the device, and the operating finger itself blocks portions of the display. With these problems in mind, a team of Google engineers is thinking beyond the touchscreen with a new method for user input that can be implemented on most devices, without the need for additional hardware.

The new method, named TapNet, makes use of the inertial measurement unit (IMU) already present in most smartphones, and with the help of a machine learning model, detects finger-triggered inputs on the sensorless back and sides of a phone. TapNet can determine tap direction (whether the front, back, or side was tapped), tap location (which area was tapped), and finger part (fingernail or finger pad).

Previous research efforts have leveraged smartphone IMUs to detect off-screen tap events, however, the reported accuracies have been far from what would be needed for a practical, real world user interface. TapNet seeks to advance the state of the art by introducing a multi-input and multi-output neural network to estimate multiple properties of taps. The neutral network was designed with a multi-branch architecture in which each branch tackles a different recognition task β€” this architecture requires less computation and memory than using separate models for each task. TapNet also uses additional information about the form factor of the phone, which was shown to increase prediction accuracy.

In order to minimize power consumption on mobile devices, the z-axis from the IMU was used as a gating signal. Only when a peak or valley was detected in this signal would the neural network be used to classify a tap.

After generating a dataset of 135,000 samples and training the model, TapNet performance was evaluated under experimental conditions. The new technique was found to be a significant improvement over existing off-screen tap detection methods with a 51% improvement in classification of tap direction, 161.5% improvement in location classification, and a 30% improvement in tap location regression. The trained model was also found to be highly generalized even after being trained by data collected from a single user. This finding means that the model may ultimately be deployed with little or no additional data collection required of end users.

The advancements presented in TapNet represent a significant improvement in the state of the art for off-screen tap detection and bring the technology much closer to being ready for real world applications.

Nick Bild
R&D, creativity, and building the next big thing you never knew you wanted are my specialties.
Latest articles
Sponsored articles
Related articles
Latest articles
Read more
Related articles