Researchers Build a Plug-and-Play System for Gesture Control Using Muscle and Motion Sensing

Researchers from the MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a plug-and-play gesture control system, called “Conduct-A-Bot," which reads both motion and muscle signals — filtered through a neural network.

"From personal assistants to remote exploration, pervasive robots could extend the capabilities and productivity of casual users. Yet a core question is how to enable effective communication; rather than adapting to the robot, a person should convey intentions as if they were interacting with another person," researchers Joseph DelPreto and Daniela Rus argue. "A variety of techniques have worked towards this goal including speech detection, natural language processing, computer vision, haptics, and gestures.

"Gestures in particular could help robots interpret non-verbal cues that people naturally use while communicating, to supplement other modalities or to create a stand-alone interface. However, deploying gestural interfaces for everyday applications with casual users yields a variety of challenges. Interfaces should be plug-and-play, not requiring users to provide extensive training data or perform dedicated calibration routines, and should minimise sensing infrastructure. Despite limited training data, which is generally costly and time-consuming to collect, classifiers must accommodate variations in gestures between users and over time."

Conduct-A-Bot relies on input from a couple of sensors, an intertial measurement unit (IMU) and an electromyographic sensor for reading muscle signals. Using these two inputs the researchers were able to detect a range of gestures including arm stiffening, fist clenching, rotation, and forearm activation, and use that to control an off-the-shelf Parrot Bebop 2 drone in closed-loop trials and an experiment controller with two LED towers driven by an Arduino Mega board for open-loop trials.

"Experiments with six subjects evaluate classifier performance and interface efficacy," the pair continue. "Classifiers correctly identified 97.6% of 1,200 cued gestures, and a drone correctly responded to 81.6% of 1,535 unstructured gestures as subjects remotely controlled it through target hoops during 119 minutes of total flight time."

The researchers' work has been published as part of the proceedings of the ACM/IEEE International Conference on Human-Robot Interaction 2020 (HRI'20), and is available from the ACM under open-access terms.