Reflective, Transparent Objects Prove No Match for Carnegie Mellon's Clever Computer Vision System
Combining a traditional 2D camera with depth-sensing data, researchers have solved a tricky problem in computer vision.
A team of researchers at Carnegie Mellon University have come up with a solution to a major hurdle in robotics: The difficulty in accurately sensing, locating, and grasping transparent or reflective objects like glasses and knives.
Computer vision systems are becoming increasingly adept at locating arbitrary objects, with one exception: If they're transparent or reflective, it scatters or reflects the infrared light used by depth camera systems — making it unable to ascertain the shape of the object.
CMU's Robotics Institute has an alternative approach: The use of traditional two-dimensional color camera systems which feed a machine learning system. The first step: Training the system on opaque, colored objects visible to both camera systems — allowing the machine learning system to associate the 2D imagery with depth data from the camera.
Once trained, the objects were replaced with transparent and reflective equivalents. While the data from the depth camera were incomplete, as was expected, it the machine learning algorithm could fill in the blanks using information from the two-dimensional color camera — with impressive accuracy.
"We do sometimes miss," admits Assistant Professor David Held of his team's creation, "but for the most part it did a pretty good job, much better than any previous system for grasping transparent or reflective objects."
The team's work has been published under open-access terms on arXiv.org following its presentation at the International Conference on Robotics and Automation (ICRA 2020), with additional information and source code available on the project website.