Giving Robots the Ability to "Hallucinate" Could Prove Key to Better Natural Language Comprehension

Researchers at the University of Freiburg have developed a new method for teaching robots how to accurately respond to natural language commands — by granting them the ability to "hallucinate" potential outcomes.

"In our work, we concentrate on relational object placing instructions, such as 'place the mug on the right of the box' or 'put the yellow toy on top of the box,'" co-author Oier Mees explains in an interview with TechXplore. "To do so, the robot needs to reason about where to place the mug relative to the box or any other reference object in order to reproduce the spatial relation described by a user."

The key difficulty: Getting a robot, which operates on very precise instructions, to understand the ambiguity in such human-centric instructions. The solution: Granting the robot the ability to "hallucinate" objects within the scene, giving it a view of what would happen in a range of potential outcomes — and in a way that makes the hallucinated objects indistinguishable from the real thing, from the perspective of the robot.

The results are impressive: Using the hallucination technique, a robot equipped with two arms was able to correctly respond to natural language commands — "right, left, on top of," and the like without needing to use traditional three-dimensional imaging tools — and, Mees claims, could prove "a promising first step towards enabling a shared understanding between humans and robots."

More information is available in the TechXplore article, or from the pre-publication paper available on arXiv.org.