May the Mixed Reality Be with You

Just admit it — at one time or another after binge-watching Star Wars, you have tested out your own powers in the Force. We have all done it. But whether the goal is to make your drink fly into your hand or bring the popcorn bowl within reach, even the greatest levels of concentration never make them budge. The Force is all well and good in a galaxy far, far away, but none of us ever have any luck right here in the Milky Way.

Fortunately for us, a pair of Princeton University researchers has taken the time to build a technology that simulates the powers of the Force for those of us not infected by midi-chlorians. Their system, which they call Reality Promises, leverages virtual reality (VR) and robotics to allow us to make objects appear to float across the room on demand. If you were to remove your VR headset you would see a robot grabbing the item and bringing it to you, but in the virtual world, it could be a cartoon character, or nothing at all, that makes the delivery.

The team’s core idea is to decouple what the user sees from what is physically happening, and then secretly recouple it once reality has caught up. Take the example of ordering a drink, for instance. In Reality Promises, you can pinch your fingers together to open a virtual menu, scroll through the options, and then pull a digital bottle from the air and place it on your desk. At first, it is only pixels. But within moments, you hear a chime, and suddenly a real bottle has appeared in that very spot, courtesy of a mobile robot that has been silently and invisibly doing the heavy lifting.

The illusion works because the system renders the robot invisible to the headset wearer using a sophisticated technique called 3D Gaussian splatting. This involves creating a highly detailed, three-dimensional scan of the room and then digitally painting over the robot so that the user never sees it in their passthrough view. Instead, they might see an animated bee fluttering across the room carrying their snack.

Behind the scenes, however, a mobile robot nicknamed Skynet is carefully navigating the room, tracking its own location relative to the user, and executing precise movements with its manipulator arm. A second component, dubbed SplatiMate, is responsible for cloaking the robot and animating digital effects such as objects materializing, vanishing, or teleporting from one spot to another.

The researchers have already demonstrated four distinct user experiences: MagicMake, where objects materialize from thin air; Seamless MagicMove, where distant items seem to leap across the room instantly; De- and Rematerializing MagicMove, where objects disappear from one place and reappear elsewhere; and Character MagicMove, where a virtual companion snatches items away or delivers them.

Mixed reality is normally limited to visual overlays and virtual effects, but Reality Promises pushes the envelope by tying those illusions directly to real-world physical changes. The end result is an experience that feels magical, intuitive, and futuristic all at once, bringing us one step closer to making our Force-fueled daydreams a reality.