Tapping Out

There is no question that mobile devices like smartphones and tablets offer us a tremendous amount of convenience and utility in our everyday lives. But these mobile devices have one major Achilles’ heel — user input. Entering data by tapping the screen with our thumbs is fine when only a small amount of input is needed, but it drastically limits the useful range of applications for these devices. Modern smartphones have significant computational resources available, so running, for example, a word processing application on the go is no problem. But tapping with two thumbs just is not going to cut it if you are trying to write the next great American novel, now is it?

When you want rapid input, it is hard to beat the tried and true keyboard. However, lugging around a full-size keyboard is impractical and defeats the purpose of mobile devices. A team of engineers at ETH Zürich may have a solution in their device called TapType that provides much of the speed of input of a keyboard with none of the bulk. Through the use of specially-instrumented wristbands and some machine learning algorithms, TapType can turn any surface into a virtual keyboard.

To create TapType, a pair of elastic silicone wristbands, one for each wrist, are outfitted with ultra low-power Bosch Sensortec BMA456 three-axis accelerometers. The wristbands also have a Dialog Semiconductor DA14695 system on a chip, with an ​​Arm Cortex-M33 processor and a Bluetooth Low Energy (BLE) transceiver. A coin cell battery can power a wristband for about 30 hours of continuous use.

The wearer of these wristbands can “type” on any surface (e.g.: a table, wall, or their leg) as if it were a keyboard. The accelerometers in the wristbands record the motions that are made while typing, and forward them to a processing backend wirelessly over BLE. The processing pipeline begins with an event detection system that looks for sudden changes in the accelerometer signal, which is indicative of a tap event. Data from these events is classified by a Bayesian deep neural network that has been trained to recognize the signature of a tap by each finger. Knowing which finger tapped the surface narrows down which “key” has been pressed, but is not sufficient to assign a specific character.

The sequence of classified finger taps is fed into an n-gram language model that determines the most likely character sequence that was typed, given the set of possible options. After a short 30 minute training period, participants in an evaluation were found to be capable of typing 19 words per minute with a 0.6% character error rate. With a bit more practice, users of TapType were able to type around 25 words per minute, with a similar error rate. With participants reaching up to 44 words per minute during some tests, it is possible that with greater familiarity, typing speed would continue to increase and perhaps even rival that of a physical keyboard.

As it stands, TapType needs a bit of work before it can be rolled out to a larger audience. For real-time typing, the processing backend requires a high-end CPU or a GPU, which is costly and reduces the mobility that TapType is seeking to offer. Also, since the system cannot determine exactly which key was intended to be pressed, but rather must infer it from context, TapType cannot recognize words that are not in the vocabulary that is understood by the language model. Should the team find solutions to these issues, TapType may be able to offer a fast and accurate means of getting user input into a wide range of mobile devices.