NVIDIA's DIB-R Creates Convincing 3D Models from 2D Image Inputs, Adds Depth Perception to Cameras
Once trained on a data set, DIB-R can create a 3D model from a single 2D image in under 100 milliseconds.
Researchers from NVIDIA have released a new rendering framework dubbed DIB-R, which they say is capable of creating high-quality three-dimensional object models from two-dimensional images through a process dubbed "differentiable interpolation."
"In traditional computer graphics, a pipeline renders a 3D model to a 2D screen. But there’s information to be gained from doing the opposite — a model that could infer a 3D object from a 2D image would be able to perform better object tracking, for example," explains NVIDIA's Lauren Finkle of the project. "NVIDIA researchers wanted to build an architecture that could do this while integrating seamlessly with machine learning techniques.
"The result, DIB-R, produces high-fidelity rendering by using an encoder-decoder architecture, a type of neural network that transforms input into a feature map or vector that is used to predict specific information such as shape, colour, texture and lighting of an image."
"This is essentially the first time ever that you can take just about any 2D image, adds paper co-author Jun Gao of DIB-R's capabilities, "and predict relevant 3D properties."
The team are looking at more than merely producing realistic 3D models from 2D image inputs, though: NVIDIA claims that DIB-R could have real applications in the field of robotics, improving the depth perception available from a single camera input — though the time required for model training against a given data set, at two weeks using one of NVIDIA's V100 GPUs, is still an issue for generalised use. Once trained — with NVIDIA using datasets including images of birds, among other complex shapes — the system can produce a 3D image of a pictured object in under 100 milliseconds, the company claims.
The PyTorch-based DIB-R is included in NVIDIA's Kaolin library, while the paper can be downloaded from the project's GitHub page now ahead of its presentation at the Conference on Neural Information Processing Systems 2019 (NeurIPS 2019) in Vancouver this week.