Clever CMOS Image Sensor Can Process Data as It's Captured, Providing Delay-Free Image Filters
By passing image filters off to the analog domain at the point of capture, future CMOS sensors could dramatically boost performance.
A team of researchers from Harvard University, the Korea Institute of Science and Technology (KIST), Pusan National University, and Samsung Electronics have developed an image sensor capable of processing the data it captures — and say the technology could easily be integrated into today's commercial CMOS-based image sensors.
"Our work can harnesses the mainstream semiconductor electronics industry to rapidly bring in-sensor computing to a wide variety of real-world applications," claims Donhee Ham, professor and senior author of the work, on what the team has created: the claimed first commercially-viable in-sensor image processor.
Based on a silicon photodiode array, just as with standard CMOS image sensors, the team's prototype uses electrostatic doping to allow each photodiode to be tuned through adjustable voltages. By alternating the voltages going into the array, the prototype is capable of performing image processing tasks in the analog domain — handling things like multiplication and addition, enhancing parts of the image or filtering out noise, without delay or the need to pass the data to a central processor.
"These dynamic photodiodes can concurrently filter images as they are captured," explains co-first author Houk Jang of the prototype, "allowing for the first stage of vision processing to be moved from the microprocessor to the sensor itself. By replacing the standard non-programmable pixels in commercial silicon image sensors with the programmable ones developed here, imaging devices can intelligently trim out unneeded data, thus could be made more efficient in both energy and bandwidth to address the demands of the next generation of sensory applications."
"Looking ahead, we foresee the use of this silicon-based in-sensor processor not only in machine vision applications, but also in bio-inspired applications," adds co-first author Henry Hinton, "wherein early information processing allows for the co-location of sensor and compute units, like in the brain."
The team's work, which saw the production of a 3×3 pixel array image sensor, has been published under closed-access terms in the journal Nature Electronics; a preprint is available under a Creative Commons license as a PDF download from Research Square.