A Touching Use of ML

The most intuitive user interface to come on the scene since the rise in popularity of point-and-click interfaces in the 1990s is the touchscreen. Removing the cursor and simply touching what one wants to interact with is very natural and requires virtually no instruction to get started. While touchscreens are very easy to use, they do have some drawbacks yet to be addressed. One of the primary problems with touchscreens is the latency between a finger contacting the screen and the requested action being seen on the display. Typical tracking latencies for modern devices are around 80 milliseconds — that may not sound like much, but for certain applications, it is nearly unbearable, which limits the use of touchscreens in some cases.

Consider hand drawing graphics on a touchscreen, for example. Finger movements can be quite swift while drawing, which leads to a rubber-banding effect that breaks the realism and responsiveness of the interface, and makes precision difficult. A group of researchers at Carnegie Mellon University have taken a multipronged approach to improve touchscreen response times. By adding an additional layer to the screen that indirectly measures frictional forces across the screen, in combination with the traditional 2D position-sensing layer, they were able to develop a machine learning algorithm that can make accurate predictions of the precise touch location, with much less latency than is possible with current methods.

TriboTouch, as the prototype device is called, employs a thin film embossed with a 2D pattern of small bumps that are spaced five microns apart. As a finger or stylus glides across this surface an acoustic vibration is induced with a frequency that directly encodes velocity in the x- and y-axes. This ultrasonic signal is fed into a conventional audio processing pipeline, with a typical latency of about 28 milliseconds. This vibroacoustic signal is combined with the high-accuracy 2D location data from standard capacitive touchscreen sensors. This data is then fed into a machine learning model for touch location prediction.

After preprocessing the sensor data and extracting the most important features, the data is fed into an extra-trees regressor. This algorithm outputs x and y coordinates of the current predicted finger location, and is able to provide 375 touch predictions per second, which is over six times faster than the refresh rate of a typical display. Running inferences faster than the display can refresh offers advantages, like allowing a smoothing algorithm to run on the output without adding latency.

To train the model, nine minutes of touch data was recorded from four participants that were asked to make a wide variety of strokes on the screen. TriboTouch was then validated in a user study with eleven participants that were asked to complete a series of touch input trials that involved drawing, writing, and dragging. They found that their model had velocity prediction power, but when it came to slow movements (below 50 millimeters per second) and fast movements (above 400 millimeters per second), the acoustic model had difficulties in making accurate predictions. Also notable is that the present implementation is computationally intensive, which makes it impractical for mobile devices, where touchscreens are most desirable. With further work, perhaps TriboTouch will become more practical and open up touchscreens to a whole new set of use cases that are not currently feasible.