Seek and You Shall Find

It may not be fun to search through a pile of items to find the one specific thing you are looking for, but it is easy enough to do. Most people would simply start pulling the largest items out of the way, one after another, until the thing that they are looking for is visible and can be retrieved. For robots, however, the problem is much more difficult. They tend to be very inefficient in seeking out hidden objects in a pile. When that robot has been deployed in an e-commerce warehouse to help process returned orders, for example, that can cause everything to slow to a snail’s pace. That makes for higher costs and dissatisfied customers, so any improvements over current methods would be very welcome.

A group of engineers at the MIT Media Lab have been working on exactly this problem. They are trying to give robots the ability to use complex reasoning to efficiently locate the proverbial needle in a haystack. By fusing data from both a camera and a radio-frequency identification (RFID) transceiver, they have taught a robotic arm to perform a quick search that beats current state-of-the-art systems. Importantly, their technique does not require that the target object has been tagged with an RFID sticker.

FuseBot, the UR5e robotic arm that employs the team’s technology, is outfitted with a wrist-mounted Intel RealSense Depth camera D415 and an RF module. The robot uses these instruments to scan a pile of unknown objects in search of a target item. The measurements collected are first analyzed by a novel RF-Visual Mapping algorithm that locates any objects in a pile that have been tagged with an RFID sticker. Using that positioning information, as well as data linked to the tag IDs that describe the characteristics of the objects, an RF-Visual occupancy distribution map is generated that tells the robot the most likely three dimensional positioning of those objects. While all items in the pile may not be tagged, those that are help to narrow down the search space through this step.

With at least a rough sketch of the likely layout of hidden objects in the pile now available, the next phase involves generating an RF-Visual Extraction policy. This algorithm seeks to extract the target object in the smallest number of moves, and does so by leveraging the probabilistic occupancy distribution previously generated, as well as the predicted information gain from future actions taken by the robotic arm.

This approach certainly sounds promising, but does it work? To answer that question, the team conducted 180 real-world experimental trials of FuseBot. Their robot went head-to-head with the state-of-the-art system named X-Ray. X-Ray uses visual information only, and has no mechanism to receive or interpret RF information, so it may not have been quite a fair fight, but in any case, FuseBot had quite a good showing. In terms of efficiency, FuseBot won out by better than 40% throughout the course of the experiments. Furthermore, in terms of success at finding the target object, FuseBot came through 95% of the time. In contrast, X-Ray only succeeded in 84% of trials.

In the future, the researchers would like to explore how to incorporate deformability of objects into their models, with the hope that this addition would improve the robot’s accuracy and efficiency even further. With the performance that has already been demonstrated, FuseBot may have a bright future in warehousing, manufacturing, retail, and beyond.