Do Androids Dream of Tactile Data?

Robots have become synonymous with stiff, jerky, awkward movements for good reason; that is how most of them operate. To reach for a cup on a table, a robot might first lift its arm straight up, then move it forward until the cup is centered within its hand, then finally tighten its fingers around the cup. Mechanical movement of this sort gets the job done in controlled environments, but in the real world, things happen. These unexpected circumstances are more than standard control algorithms can handle and lead to spectacular failures.

People, on the other hand, take a very different approach. We move naturally and with incredible dexterity, and we intuitively understand, and adapt to, hand-object interaction dynamics. If, for instance, we bump a cup and nearly spill it while reaching for it, we instantly know exactly how to react to keep it from tipping over.

A team led by researchers at Carnegie Mellon University is attempting to give robots human-like dexterity and understanding so that they can stop being so, well… robotic. Their new system, called Humanoid Transformer with Touch Dreaming (HTD), combines whole-body control, tactile sensing, and AI-driven prediction to help humanoid robots manipulate objects more naturally in complex real-world environments. The researchers say the approach could eventually allow humanoid robots to perform household chores, assist workers in industrial settings, and collaborate more safely with humans.

One of the biggest challenges in humanoid robotics is that manipulation is not just about moving a hand or arm. Every action affects the entire body. Reaching too far can throw off balance, while a poorly timed grip can shift an object unexpectedly and destabilize the robot. Existing systems often struggle because they treat locomotion and manipulation as separate problems.

To address this, the researchers developed a layered control system that divides responsibilities between different subsystems. A reinforcement learning-based lower-body controller keeps the robot balanced by managing torso orientation, walking, and posture. At the same time, upper-body inverse kinematics handle arm positioning, while dexterous hand-retargeting software manages finger movements.

The researchers also created a VR teleoperation platform that allows human operators to control the humanoid robot while collecting demonstration data. The setup integrates distributed tactile sensors throughout the robot’s hands so that the machine not only sees what it is touching, but also feels it.

That tactile information became the foundation for HTD’s most unusual capability: “touch dreaming.” Instead of only predicting future actions, the AI model also predicts how touch and force interactions will evolve during manipulation tasks. This includes forecasting future hand-joint forces and tactile information.

The system was evaluated across five real-world tasks, including towel folding, tea serving, cat litter scooping, object insertion, and organizing books. These tasks required precise alignment, sustained contact, deformable-object handling, and coordinated whole-body movement. Across all five tasks, HTD achieved a 90.9% relative improvement in average success rate over a strong baseline model known as ACT.

Future work will investigate scaling the framework using larger datasets derived from both robots and human demonstrations, with the long-term goal of building humanoid systems that can adapt to many different embodiments, environments, and forms of physical interaction.