Google's PaLM-SayCan Gives Everyday Robots a Better Understanding of Natural Language Commands

A team at Google's research arm is looking to make robots more understanding, offering a batter grasp of natural language instructions and statements — boosting performance, it claims, and giving it the ability to handle more abstract tasks.

"If a person tells you, 'I am running out of time,' you don't immediately worry they are jogging on a street where the space-time continuum ceases to exist," explains Google Research's head of robotics Vincent Vanhoucke. "You understand that they're probably coming up against a deadline. And if they hurriedly walk toward a closed door, you don't brace for a collision, because you trust this person can open the door, whether by turning a knob or pulling a handle.

"A robot doesn’t innately have that understanding. And that’s the inherent challenge of programming helpful robots that can interact with humans. We know it as 'Moravec's paradox' — the idea that in robotics, it’s the easiest things that are the most difficult to program a robot to do."

Part of that difficulty is communicating a task to a robot, which traditionally takes the form of painstaking programming. While robots capable of responding to natural-language commands exist, they're typically limited to simple statements detailing correspondingly simple tasks — which is where PaLM-SayCan, developed in partnership with Everyday Robots, comes in.

"This joint research uses PaLM — or Pathways Language Model — in a robot learning model running on an Everyday Robots helper robot," Vanhoucke explains. "This effort is the first implementation that uses a large-scale language model to plan for a real robot. It not only makes it possible for people to communicate with helper robots via text or speech, but also improves the robot's overall performance and ability to execute more complex and abstract tasks by tapping into the world knowledge encoded in the language model."

In testing, equipping the robot with PaLM-SayCan delivered a 14 percent boost to planning success rate, a 13 percent improvement to execution success rate, and an impressive 26 percent enhancement to the planning of so-called "long horizon" tasks with eight or more steps: "I left out a soda, an apple and water. Can you throw them away and then bring me a sponge to wipe the table?"

Full details on the effort are available alongside a copy of the paper on the PaLM-SayCan project page; the group has also published an open source simulation of a PaLM-SayCan-equipped desktop robot on GitHub with which the research community can experiment.