MSU Engineers Create a Small, Portable Virtual Sign Language Interpreter

For people who are deaf or hard of hearing, communication with people who don’t understand American Sign Language (ASL) is a challenge that stretches across all aspects of life. Simply ordering a coffee from the barista is difficult, and more critical interactions, like in a job interview or with a doctor, are nearly impossible without an interpreter. That’s why engineers from Michigan State University have developed a small, portable, and affordable virtual sign language interpreter.

This virtual interpreter is based on a Leap Motion device, which retails for less than $80. The Leap Motion device is small — about the size of a Bic lighter — but it’s able to track and analyze hand movements with its built-in cameras. It’s like a small Microsoft Kinect Sensor, but for your hands. By using the Leap Motion device in conjunction with their specially-designed DeepASL deep learning system, the MSU engineering team was able to create a virtual sign language interpreter that people who are hard of hearing can carry with them to communicate with people who don’t understand ASL.

While virtual interpreters do already exist, the current commercial options have some severe limitations. Many require the user to wear a sensor glove that detects their hand movements, which is obtrusive and can make the user feel demeaned. Others can only sense one hand movement at a time, which slows down conversation to a crawl.

The DeepASL deep learning algorithm is able to seamlessly interpret hand signs in real time, so users can have a natural conversation. Aside from facilitating communication, this system could also be used as a training aid for people who are trying to learn sign language, as it can give immediate feedback on their accuracy. The engineers behind DeepASL say that the technology is ready for commercialization, so it may actually be on the market relatively soon.