Going Behind the Scenes

Artificial intelligence (AI) algorithms are becoming very adept at skills like navigation and obstacle avoidance, which are linchpins in the development of autonomous vehicles. These sophisticated algorithms enable self-driving cars to interpret and respond to their surroundings with high levels of precision as they make real-time decisions that enhance safety. However, the predictions made by an AI system are only as good as the data that is fed into them. In the case of a self-driving car, for example, if the sensors cannot detect a pedestrian hidden in the shadows, it will not be able to calculate a path that safely avoids them.

In the real world these types of scenarios happen all the time, which is a major reason why so few fully autonomous cars are actually on the roads today. The fact of the matter is that in order to respond to changing conditions quickly enough when traveling at high rates of speed, the navigation systems almost need to be able to peer around other vehicles, corners, and other obstructions — in addition to being able to see what lies in the shadows.

Surprisingly enough, that is exactly what a team of researchers at MIT and Meta have just made possible. Using their novel system, called PlatoNeRF, they have shown that it is possible to create a 3D model of a scene that reveals what is hidden from view. And they did not use a large array of sensing equipment that would be impractical to integrate into a real product either. PlatoNeRF collects data from a single vantage point using a type of sensor that is already commonly found in autonomous vehicles.

By bouncing light off of objects and measuring the reflections with a lidar sensor, many self-driving vehicles acquire a high-resolution picture of their surroundings. But these traditional methods can only get a view of objects that have a direct line of sight to the sensor. So in developing PlatoNeRF, the team focused on secondary reflections, rather than the primary reflections returning from the object in front of the lidar unit. Most light waves sent out from the lidar unit bounce and strike other, secondary objects before returning to the sensor.

By determining the time it takes light to return to the sensor after a second bounce, the team can better understand where shadows are present in the scene. And those shadows can reveal the locations of otherwise hidden objects. Deciphering this data is no small task, however, so the researchers built an AI model called a neural radiance field (NeRF) to do the job. These types of models have an excellent ability to reconstruct novel views of a 3D scene.

Combining multibounce lidar with the NeRF model proved to be quite challenging. But after diving into the physics of light waves and determining how best to encode that knowledge into a machine learning model, the team found that they were able to produce highly accurate scene reconstructions using PlatoNeRF. In fact, PlatoNeRF outperformed the existing alternative approaches that were evaluated in this work.

Looking to the future, the team is planning to explore how they might use lidar measurements from light that has bounced more than two times. They are also considering how color images could be incorporated into the pipeline. They suspect that these enhancements could further improve the accuracy of PlatoNeRF.