Neuromorphic Camera Sees the Future

As certain technologies continue to develop, the need for efficient dynamic machine vision systems is becoming more apparent. These systems use sophisticated algorithms and machine learning techniques to analyze real-time video feeds and make predictions about the future movements of objects and people.

Dynamic machine vision systems can be used in a variety of industries, including transportation, security, and entertainment. For example, self-driving cars rely on dynamic machine vision to predict the movement of other vehicles and pedestrians on the road to prevent accidents. Security systems can use this technology to monitor public spaces and predict potential threats. In the entertainment industry, it can be used to create immersive virtual reality experiences.

Developing efficient dynamic machine vision systems poses several challenges, with one of the biggest being the heavy resource requirements of current techniques. These systems process multiple video frames in real-time to make accurate predictions about future trajectories, which requires a significant amount of processing power and energy. In particular, this can be a problem for mobile and edge devices that have limited computing and battery resources.

A group of researchers at Aalto University have developed a new, much more efficient dynamic machine vision system that was inspired by the human visual system. The key innovation is a neuromorphic image sensor that not only senses the present, but also has a memory of the past. Both past and present are represented in the same image frame, so relatively simple and efficient machine learning algorithms can analyze the data to make future trajectory predictions.

The novel image sensor is composed of an array of photomemristors, which produce an electric current proportional to the amount of light that falls on them. Where a traditional image sensor measures its exposure to light at a precise moment in time, this neuromorphic sensor has something of a memory. After being excited by light, the electric current produced by the photomemristor only gradually decays. This gives them the ability to remember the past.

To test the system, a video was played that displayed the letters of a word, one at a time. Each of the words ended with the same letter, so the final frame would be the same in all cases, if a traditional image sensor was in use. A neural network was trained to recognize what word had been spelled out, given only the final image frame. After videos displaying words like “APPLE” or “GRAPE” were displayed, it was found that the new process could identify each case with nearly 100% accuracy.

Similar tests were conducted using videos of people walking at various speeds. This time, it was found that the system could accurately predict future frames of the video, while only using a single frame as an input. Traditional methods would require many frames from previous states, and would require much more complex neural networks, and the associated computational resources.

At the present time, the photomemristors are limited in their response to light to within the 320 to 400 nm (UV to blue light) wavelength range, which limits the practical applications of the technology. The research team does, however, have a plan to broaden the range of the sensors’ light sensitivity. If that works out as hoped, photomemristor-based neuromorphic image sensors could be a huge boon to many industries, especially where on-site motion perception and prediction are required.