Tapping Into Vibrations

Portable computing devices offer a lot of convenience on the go, but any serious user input is a big pain. Have you ever heard of anyone writing a novel on their smartphone? Exactly.

For this reason, much effort has gone into extending input surfaces beyond the confines of the device, and we have covered several of these. The latest entrant comes to us in the form of a tap detecting and localizing wristband from Facebook Reality Labs and Dartmouth College named Acoustico.

The wristband is equipped with two highly sensitive accelerometers, two SparkFun MEMS microphone breakouts, and an ICP signal conditioner. A PicoScope 2406B oscilloscope was used to sample signals from the accelerometers and microphones and stream them to a desktop computer via USB.

The desktop computer analyzes the sensor data in Matlab using a two stage pipeline. The first stage detects surface taps, and the second stage localizes the taps. Accelerometer data detects surface waves — these are the vibrations that propagate through the surface of the tapped object. The microphones detect sound waves, which are vibrations that propagate through the air as a result of the surface tap. By fusing data from the different sensors, the team was able to improve performance of the system.

A tap is only registered when both surface and sound wave signals rise above a predefined threshold. This avoids many false detections that would occur with a single sensor type. For tap localization, the researchers used the time differences of arrival (TDOA) between each pair of sensors. As the device is built into a wristband and the sensors are necessarily located near to one another, this presented difficulties in implementing TDOA because the measurements were extremely small. To solve this issue, time differences were calculated between different sensors (i.e. between accelerometers and microphones). Signal propagation time differs between surface materials and air, which adds extra time differences for the tap localization calculation.

In a study of twenty participants designed to evaluate Acoustico, tapping detection accuracy was found to have an average F1 score of 0.9987. For tap localization, a mean error of 7.6mm was observed on the x-axis, and a mean error of 4.6mm was observed on the y-axis.

There is still some room for future improvements before Acoustico is ready for everyday use. The team did not evaluate the device in situations where the user’s hand is not on a surface, but vibration and sound signals are detected. This would presumably lead to many false tap detections in the real world. Moreover, Acoustico needs to be specifically trained for any surface it has not already been trained on, which could also impact the operation of the device in unexpected ways.