Researchers at Cornell University have developed an inexpensive method, called ShadowSense, for soft robots to detect interactions without relying on touch. Some interactions include pats, punches, and hugs. The robot has a USB camera in its body that records the shadow of a moving hand’s gesture on the robot and classifies them using machine-learning software. This robot could be used to lead humans to safety during an emergency evacuation. However, it needs to communicate with humans in a hostile condition and environment to perform its task.
Instead of integrating contact sensors, which adds weight and wiring, the team used sight to measure touch. The robot contains a small inflatable bladder of nylon skin that covers a four-foot cylindrical skeleton attached to a mobile base. A USB camera, which is under the robot’s skin, connects to a laptop. The team created a neural-network-based algorithm that uses training data to differentiate between six touch gestures, including palm touches, two hands touching, pointing, punching, and no touches. It had an accuracy score of 87.5 to 96%, depending on the lighting.
It can also react to specific touches and gestures, such as rolling away or delivering a message through a loudspeaker. Even cooler, the robot’s skin could be used as an interactive screen. ShadowSense can be integrated into different materials, including balloons, transforming them into a touch-sensitive device.
If sufficient data is collected, a robot could be trained to identify more interactions, which could be customized to fit the robot’s task. ShadowSense also offers a sense of privacy.
“If the robot can only see you in the form of your shadow, it can detect what you’re doing without taking high fidelity images of your appearance,” said Yuhan Hu, researcher and the paper's lead author. “That gives you a physical filter, and protection, and provides psychological comfort.”
Interacting and understanding human movements and moods could be equally important to the person as it is to the robot. “Touch interaction is a very important channel in terms of human-human interaction. It is an intimate modality of communication,” Hu notes. “And that’s not easily replaceable.”