PatternTrack Enables Instant Multi-User AR Experiences

With all the hype around the metaverse (even if it has not fully arrived just yet), we are growing to expect that virtual experiences will be opportunities to interact, work, play, and socialize with others. And that may be true when it comes to virtual reality, but augmented reality (AR) is a much less social experience at present. When the entire world around you is digital, it is not terribly challenging to enable interactions with others in a shared space. But when digital content is layered on top of the real world, synchronizing the experience between multiple users is quite difficult.

Current approaches often require the use of fiducial markers to provide common reference points. This is effective in terms of performance, but setup is necessary, and the markers can distract from the task at hand. Other systems use UWB, Bluetooth, or RF signals for proximity sensing, but they require external hardware and do not provide the vector or positional information that is necessary for high levels of accuracy.

A better approach may be on the horizon, thanks to the work of a group of engineers at Carnegie Mellon University and the University of British Columbia. They have developed what they call PatternTrack, which is a new multi-device localization approach. It repurposes hardware that is already present in many commercial AR-enabled hardware platforms, such as the Apple Vision Pro, iPhone, iPad, and Meta Quest 3. No external hardware infrastructure is needed for operation.

To build their system, the team repurposed the depth‑sensing cameras already found in many devices, using them to project an infrared dot pattern (144 points in the case of Apple’s LiDAR array) so they can measure distance. The perspective distortion of that grid, when it lands on a wall or tabletop, encodes the precise 3D position and orientation of the projector itself. Any nearby device that can see the dots instantly knows where the projecting device is, without scanning an entire room or exchanging large spatial maps.

Since device manufacturers often keep the raw infrared data from the depth camera hidden, the team built a proof‑of‑concept rig from a Raspberry Pi Zero 2 W that is paired with a 940‑nanometer‑filtered camera. This hardware was velcroed to the back of an iPhone for testing. The Raspberry Pi streams infrared frames over Wi‑Fi, while the phone supplies aligned RGB and depth data. All of the data sources are then reassembled on a laptop, recreating the missing pieces.

Despite the ad‑hoc hardware solution, the results of the validation tests were promising. Across six surfaces, including smooth, feature‑free drywall, the system averaged just 11.02 cm of positional error and 6.81° of angular error at separations up to 2.5 m. It was also found that it often needs only a single frame to lock on, meaning a shared AR session can spin up in under a tenth of a second.

Right now, the team’s approach requires a bit of hacking, but if hardware makers expose their infrared sensors in the future (or simply add PatternTrack‑style math into their firmware), shared, marker‑free AR could become a tap‑and‑go feature.