I Read You Loud and Clear

Over 500,000 deaf people in the US and Canada alone communicate through American Sign Language (ASL). While this provides these people with an effective form of communication, consider that there are hundreds of millions of people in these countries. Accordingly, there are frequent communication breakdowns between those that know ASL, and those that do not.

A team led by Zhanpeng Jin at the University of Buffalo has developed a novel device that seeks to bridge this communication barrier with a nonintrusive technology. Called SonicASL, the device is built into an off-the-shelf pair of headphones. It adds an additional speaker and microphone — the speaker sends out inaudible sound waves, which bounce off of objects in the environment, and the echoes are received by the microphone.

This application of Doppler technology can sense tiny fluctuations in sound waves that are created by the hand movements of someone signing in ASL. These signals are transferred to a companion smartphone app running a custom algorithm, which can translate the echoes into words. These words are then spoken through the headphones to allow the observer to hear an audible representation of what is being signed.

A small study, consisting of eight subjects, was performed to validate the device. In the study, 42 frequently used words, and 30 meaningful sentences were signed and interpreted by SonicASL. The system achieved an accuracy of 93.8% and 90.6% for word-level and sentence-level recognition, respectively. Tests were conducted in real-world scenarios, both indoors and outdoors, to assess SonicASL’s robustness against noise. The system was found to be highly effective even in the presence of ambient noises and nearby pedestrians.

While these results are very encouraging, SonicASL is still in the proof-of-concept stage. In addition to further enhancing the accuracy of the device, it will also be necessary to incorporate facial expressions into the translation algorithm, as they are a major component of ASL. The team also has their sights set on expanding the platform to be compatible with other sign languages beyond ASL. With additional work in these areas, SonicASL has the potential to break down many long persisting barriers to communication.