The Toe Commandments

Alternative methods for interacting with digital devices, like voice and gesture recognition, have been rising in popularity in recent years. These interfaces are particularly appealing when it comes to mobile applications, where directly interacting with the device may be either inconvenient or impossible. But these alternatives are not without their shortcomings, either. Gesturing in public spaces is often socially awkward, and furthermore, it also telegraphs exactly what one is doing to everyone around them, compromising their privacy.

A team at the University of Salzburg and Monash University recently got together and proposed a possible solution called GestureSock. It is exactly what it sounds like — a sock-based user interface that detects gestures made with the toes. The toes may not have the ability to be as expressive as fingers, but since they are hidden away inside shoes, the gestures can be made discreetly. And aside from social and privacy concerns, the toes are also available when the hands are occupied with other tasks.

Initial attempts to capture movements of the toes involved the use of flex sensors. However, toe movements proved to be too subtle for these sensors to capture reliably. Next they tried Velostat, a pressure-sensitive material that changes its resistance based on applied force. Velostat proved to be sensitive, flexible, and customizable, allowing the researchers to tailor sensor shapes and placements precisely to the contours of the foot.

Using this approach, sensors were strategically placed on the toe tips and joints of the big toe, the three central toes, and the small toe to capture presses and flexes. Additional sensors were sewn between toes to detect toe-splitting gestures. These sensors were wired to a custom circuit board housed at the top of the sock, which handled signal processing and interfaced with the central microcontroller, an ESP32.

First, the system processed raw sensor data using a moving average filter to ensure stable readings. Then, using a machine learning model trained on 470 gesture samples, it classified gestures in real time. Two models were tested: Random Forest and XGBoost. The latter achieved an accuracy of 97.5%, making it the final choice for deployment.

The XGBoost model was compiled and deployed directly onto the ESP32. The microcontroller also leveraged Bluetooth Low Energy to emulate a wireless keyboard, allowing it to communicate with devices like Android and Apple smartphones, or even AR/VR headsets such as the Apple Vision Pro. With a few toe movements, users could play or pause media, skip tracks, or trigger other predefined commands — all without touching the device.

The team’s prototype demonstrates how a low-profile wearable can enable hands-free, eyes-free, and socially discreet interaction with electronic devices. By tapping into an underexplored input method — toe gestures — GestureSock broadens the vocabulary of human-computer interaction and opens new doors for accessibility, multitasking, and privacy-preserving design.