ReMotion Tracks a User's Face and Body to Drive a More Realistic Telepresence Robot Proxy
Using a Kinect camera and a "NeckFace" wearable, ReMotion renders its user as an attentive avatar capable of moving around a room.
Researchers from Cornell University have developed a new robot platform designed for those who can't be physically presence in a space — transferring bodily movement and facial expressions to a robotic proxy.
“Pointing gestures, the perception of another’s gaze, intuitively knowing where someone's attention is," says first author and doctoral student in information science Mose Sakashita. "In remote settings, we lose these non-verbal, implicit cues that are very important for carrying out design activities. With ReMotion, we show that we can enable rapid, dynamic interactions through the help of a mobile, automated robot."
ReMotion is the combination of two central design concepts. The first is the telepresence robot itself, standing almost six feet tall and with a monitor as its head and omnidirectional wheels for locomotion — powered by the Unity 3D game engine with Python scripts to interface it with the electronics. So far, ReMotion sounds a lot like any other telepresence robot — but the monitor doesn't just simply host a video feed from the remote participant's camera.
Instead, the remote part of the system combines an Azure Kinect depth-sensing camera to track the user with a wearable camera system dubbed NeckFace to capture facial expressions — expressions which are then used to render a lifelike avatar, displayed on the proxy robot's monitor "face." Between the face, which can "look" where the remote participant is looking, and a pair of pointing arms which mimic the user's hand motions, the robot not only mimics the user's spatial position and movements but also indicates their attention — crucial for teaching.
In tests, most participants indicated a feeling of increased connection with their remote participants compared to a standard telepresence robot with a particular boost to feelings of shared attention. There's work still to be done, though: the current ReMotion system only works with two users, one local and one remote, who are present in rooms of identical size and layout.
The team's work has been published in the Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23), under open-access terms.