Researchers Build an Image Sensor Which Doubles as a Neural Network for Nanosecond Recognition

A team working at the Vienna University of Technology (TU Wein) have developed an image sensor which doubles as a neural network, dramatically speeding up image analysis by skipping the need to transfer the captured image data to an external processor.

"Typically, the image data is first read out pixel by pixel and then processed on the computer," explains TU Wein's Thomas Mueller. "We, on the other hand, integrate the neural network with its artificial intelligence directly into the hardware of the image sensor. This makes object recognition many orders of magnitude faster."

The image sensor is based on photodetectors build of tungsten diselenide just three atomic layers thick. Each photodetector, representing one pixel of the captured image, is linked to a small number of output elements which provide an external system with the result of an object recognition task — and in just 50 nanoseconds per analysis, the researchers claim. Impressively, it manages this without connection to a traditional power source: The chip is powered by the very photons it is imaging.

"In our chip, we can specifically adjust the sensitivity of each individual detector element - in other words, we can control the way the signal picked up by a particular detector affects the output signal," says Lukas Mennel, primary author of the publication. "All we have to do is simply adjust a local electric field directly at the photodetector."

"Our test chip is still small at the moment, but you can easily scale up the technology depending on the task you want to solve," claims Mueller. "In principle, the chip could also be trained to distinguish apples from bananas, but we see its use more in scientific experiments or other specialised applications.

"From fracture mechanics to particle detection - in many research areas, short events are investigated. Often it is not necessary to keep all the data about this event, but rather to answer a very specific question: Does a crack propagate from left to right? Which of several possible particles has just passed by? This is exactly what our technology is good for."

The team's work has been published under closed-access terms in the journal Nature, with more information available from IEEE Spectrum and MIT Technology Review articles on the work.