Do You See What I’m Saying?

Smart speakers offer a lot of convenience to their users (when they can understand us, that is), from managing a smart home to looking up a recipe or calling a friend. While these speakers offer a very natural user interface to many, such devices are not usable by the deaf and hard-of-hearing that communicate via sign language. This impacts almost thirty million people with bilateral hearing loss in the United States alone.

A team from George Mason University has taken on the task of solving this problem with a device they have designed called mmASL. This device takes advantage of 60 GHz millimeter-wave radio signals to detect the presence of American Sign Language (ASL) gestures, while preserving privacy of users in a way that RGB and infrared cameras (the common means of detecting ASL gestures) cannot.

mmASL is implemented using a 60 GHz reprogrammable software radio platform and a phased antenna array. Antennas are partitioned into separate transmit and receive modules, and are capable of beamforming, which is used to focus the signal on particular subjects as needed. Reflected signals are converted into Doppler spread spectrograms for further processing.

When running, mmASL scans through a set of beam sectors and creates spectrograms of received signals. The spectrograms are fed into a Convolutional Neural Network (CNN) to check for the presence of a wake-word that activates the system. Upon detecting a wake-word, mmASL then uses another CNN classifier to determine the location of the user. Signals are then directed at that location to capture ASL gestures. Finally, these gestures are classified against a third CNN classifier that was trained to recognize fifty words commonly used to interact with home assistants.

In a large-scale study, mmASL was found to have an average accuracy of 87% in sign recognition. This accuracy is comparable to well-studied camera-based sign recognition systems.

A significant drawback to using millimeter-wave radio signals is that it is unable to detect hand shape, and rather relies on hand movements. A number of ASL gestures have similar hand movements and are differentiated by hand shape, which prevents mmASL from recognizing them. The researchers suggest that augmenting the device with radar to estimate handshapes may improve classification performance. Also problematic is the hardware required is both expensive and bulky. Advancements in the related fields are needed to miniaturize the device and reduce costs sufficiently to make mmASL accessible in the way that voice-based home assistants currently are.