What’s Your Sign?

According to the World Health Organization, 5.3% of the world’s population experiences a disabling hearing loss. While exact figures are difficult to come by, it is generally thought that less than 1% of individuals know a sign language. This gap between hearing abilities and fluency in sign language is leaving many with communication difficulties.

A recent Google initiative, called Project Shuwa after the Japanese word for “sign language," is seeking to use technology to benefit Deaf culture and increase understanding of sign language. Goals of the project include improving Google searches for sign languages, and building a more comprehensive dictionary across more sign and written languages.

Another part of this initiative, an interactive web game named SignTown, seeks to teach sign language with the help of machine learning and computer vision. SignTown uses the camera on a laptop or smartphone to watch the user. In the course of the game, the user will be asked questions that they need to respond to with sign language. Feedback is given when signs are not performed correctly.

To make this work, the MediaPipe Holistic model was used to extract body poses and facial expressions from the image data. By going beyond just poses, and also including facial expressions, the model is better able to capture the speaker’s true intent. A machine learning classifier then uses this data to determine the closest matching sign. Through the use of TensorFlow.js, the machine learning inferences all take place in the user’s web browser.

Trying out SignTown for myself, I was surprised to see how well it worked, considering that it was all running locally in my own browser. I found that it started up quickly, was highly responsive, and appeared to be quite accurate. When my signs were off the mark, SignTown was able to give specific feedback that helped me to fix my mistakes. It is worth noting, however, that at present, SignTown is limited to Japanese and Hong Kong sign languages.

Project Shuwa looks to be a significant step towards Google’s stated goal of building a more accessible world for people with disabilities through technology. For anyone that wants to experiment with Google’s models or tools, they have been open sourced.