This "Lensless" Camera Captures 3D Images in a Single Shot — and Can Even See "Through" Obstacles
Clever camera system and its neural network reconstruction system could come to a future smartphone, if the team has its way.
A pair of researchers from the University of California, Davis have created a "lensless" camera capable of capturing three-dimensional images in a single shot, complete with full refocus capabilities post-capture — using a neural network to boost reconstruction performance to real-time without complex calibration.
"We consider our camera lensless because it replaces the bulk lenses used in conventional cameras with a thin, lightweight microlens array made of flexible polymer," explains team lead Weijian Yang. “Because each microlens can observe objects from different viewing angles, it can accomplish complex imaging tasks such as acquiring 3D information from objects partially obscured by objects closer to the camera."
Based on technology the team originally developed for a microscope, the microlens array captures multiple views of a single scene in one pass. While each sub-image contains only two dimensional data, the difference in angle between each microlens means that 3D data can be calculated — with a newly-developed neural network doing so in real-time.
"Many existing neural networks can perform designated tasks, but the underlying mechanism is difficult to explain and understand," claims Yang. "Our neural network is based on a physical model of image reconstruction. This makes the learning process much easier and results in high quality reconstructions."
Images captured by the camera system can be refocused post-capture, while the neural network also outputs a depth map, which can be used for navigation tasks or 3D modeling. The camera can also see "through" opaque objects, rendering them transparent providing at least one microlens could see around them — a world-first for lensless cameras, Yang says.
“This 3D camera could be used to give robots 3D vision, which could help them navigate 3D space or enable complex tasks such as manipulation of fine objects," Yang claims. "It could also be used to acquire rich 3D information that could provide content for 3D displays used in gaming, entertainment or many other applications. With the recent development of low-cost, advanced micro-optics manufacturing techniques as well as advancements in machine learning and computational resources, computational imaging will enable many new imaging systems with advanced functionality."
The team is now working on improving the image quality by lowering the error rate and miniaturizing the hardware with the goal of being able to fit the technology into future smartphones.
The pair's work is published in the journal Optics Express under open-access terms.