Meta's Light-Field Pass-Through System Aims to Make VR Headsets Turn Transparent On-Demand

Facebook owner Meta's virtual reality division is working on a headset with a neat trick: a "light-field pass-through" system which lets you see the real world exactly as you would without a headset on at all.

"Virtual reality pass-through uses external cameras on the front of a VR headset to show images of the real world to a user in VR," the company explains of the current state-of-the-art. "However, the pass-through camera captures a different perspective of the world than what the user would see without the headset. To account for this difference, computational methods can be used to synthesize the target view through image reprojection — but computational pass-through can lead to visual distortions."

The solution, according to Meta, is something it's calling Flamera: rethinking things from the ground up to avoid reprojection, instead recreating the light rays exactly as they would have reached the user's eyes — providing, when required, a distortion-free view of the real world which looks exactly the same as if you weren't wearing a VR headset at all.

"Our key design insight is to add a set of apertures that physically block all the [light] rays that shouldn't enter the eye," Meta explains of the new approach, which uses an array of microlenses designed to capture light fields. "With carefully designed apertures we can perfectly capture the target rays over the whole field of view. If we designed the aperture correctly, it will look like there's a single opening that exists somewhere behind the [microlens] array."

The company's prototype is capable of working in real-time, providing the wearer with a pass-through image which looks convincingly like the headset has been turned transparent — even when the headset is in motion. Meta has not, however, indicated when — or even if — the technology will be brought to market.

More details are available in the video above.