More Than Words Can Say

Human communication is a complex and multi-faceted phenomenon that extends far beyond the boundaries of spoken language. While speech is undoubtedly a vital tool for conveying thoughts and ideas, our ability to express ourselves is enriched by an intricate array of nonverbal cues. Among these, facial expressions stand out as powerful messengers of emotions. A simple smile can express happiness, while a furrowed brow might signify concern. These nonverbal cues often work in harmony with spoken language, reinforcing, contradicting, or adding depth to the words we utter.

However, for individuals who have lost the ability to speak, expressing themselves can become a formidable challenge. Conditions such as severe speech disorders or paralysis due to various neurological conditions can significantly restrict one's ability to communicate effectively. These individuals often encounter frustrating barriers as their internal thoughts and emotions remain locked, struggling to find an outlet. The loss of facial muscle control means that the subtleties of emotion that are effortlessly conveyed through expressions are now inaccessible, leaving them to rely solely on alternative means of communication.

Assistive technologies have made remarkable strides in aiding those who face these challenges, but tend to be plagued with frustrating limitations. For example, they tend to be very slow when it comes to recreating speech, and in most cases, they completely ignore the valuable information contained in facial expressions. A team at the University of California, San Francisco has taken a novel approach that addresses many of the concerns with existing systems, offering the promise of restoring rich communication capabilities to those that have lost their natural ability to do so.

To do this, the team developed a technique that involves surgically implanting a grid of 253 electrodes on the surface of an individual’s brain. This brain-computer interface intercepts neural signals that would stimulate the muscles of the face, lips, tongue, and jaw under normal circumstances to produce speech and facial expressions. Interpreting these signals is by no means straightforward, so a machine learning algorithm was trained to recognize how the brain signals translate into muscle movements and vocalizations.

Rather than take the common approach of classifying a limited set of words, the researchers instead taught their model to recognize phonemes. Phonemes are perceptually distinct units of sound that can be arranged to compose any conceivable word. Using this approach made the system very fast — it only needed to recognize 39 phonemes, rather than a large set of possible words. Moreover, the vocabulary of the device is unlimited, allowing the individual to express themselves with great depth.

In addition to recognizing speech, the algorithm was also trained to understand how neural signals translate into facial movements. This information was leveraged to control a digital avatar that mirrors the intended facial expressions of the user of the system, such that a person conversing with them can get a read of their emotions for additional context.

The system was installed in an individual with locked-in syndrome. It was also highly customized for this specific person by training a speech synthesizer to speak in a way that resembles her natural voice by supplying it with videos of her talking before the medical emergency that gave rise to her locked-in syndrome. The on-screen avatar was also developed to resemble her facial features.

Previously, this individual had utilized a system that allowed her to communicate at a rate of 14 words per minute by using subtle head movements. The new technique allows this same person to communicate in a much more natural way, and at a much faster rate of 80 words per minute.

As it currently stands, the brain-computer interface requires a physical connection to a nearby computer for processing. The researchers are presently working to make the connection wireless, to make it less cumbersome for the user. That should go a long way towards making users of the system feel more connected to the world. The participant in this study was inspired by the experience, and now aspires to become a counselor in a physical rehabilitation facility, to help others experiencing problems like her own.