Sensor Translates Sign Language Into Audio

Tsinghua University researchers have developed a self-powering sensor with multiple environmental stimuli detection and monitoring capabilities, allowing it to translate sign language into audio. The team found inspiration from human skin that detects temperature, humidity, pressure, and light stimuli changes. It doesn't rely on an external power supply, making it unique compared to artificial sensors.

The new graphene oxide sensor relies on a moist electric generator (MEG) for power. It has a membrane that collects water in the air. Once the water sticks to the surface, the membrane's top will have a higher concentration of hydrogen ions. Simulating the device via humidity, pressure, temperature, and light generates electrical charges that cause a potential variation, producing response signals to those stimuli.

Machine learning also helps with this process, combining the external responses into individual signals for the device to learn, store, and interpret. The engineers placed the sensor on a volunteer's wrist and taught the machine learning algorithm to read and translate varying finger and hand movements, associating them with each word and phrase. The unique pressure stimulus sequences produced by finger movements and hand gestures can be collected via Bluetooth. Then, the gestures undergo classification thanks to the machine learning model decoding and analyzing those signals. These gestures and audio are then displayed on the team's smartphone app.

The team believes this new device can be integrated into IoT and serve applications for health monitoring, human-computer interactions, and the metaverse. However, the tech is still in the early stages and needs improvement before being used for everyday purposes. For instance, they expect environmental disturbances in complex applications to generate response signal noise. That can be solved via electrical signal processing or device performance optimization.