Crosswalks and traffic lights. Vehicles zipping by. Street signs and low-hanging tree branches. Most of us navigate these everyday situations without a second thought, but for the visually impaired, they can present serious challenges.
While discussing difficulties such as these with a visually impaired friend, Jagadish Mahendran came up with an idea that he thought might be able to help. As a graduate student studying Artificial Intelligence (AI) at that time, Mahendran wondered if he might be able to design an AI-powered, voice-activated device that could provide the visually impaired with useful, relevant information about the world around them.
The device consists of a processing unit, such as a laptop or a Raspberry Pi, and a power source stowed inside of a backpack. A Luxonis OAK-D spatial AI camera is hidden inside a vest, while a second OAK-D and additional battery pack are tucked away in a fanny pack. A GPS receiver is installed on the top of the backpack, and Bluetooth-enabled earphones allow for user interaction with the system.
The OAK-D cameras, which provide depth information along with the imagery, are powered by software that makes use of both OpenCV and the open source DepthAI library. The onboard batteries supply enough power for about eight hours of operation.
To use the device, a wearer will issue simple verbal commands through the included earbuds. For example, saying “describe” will trigger the system to tell the user about all detected objects in their path. To prevent overwhelming the user with constant chatter about their surroundings, feedback is only provided on explicit request, except in the case of emergencies — to avoid collisions or warn about elevation changes, for example.
The spoken responses to a request may sound something like: “Yellow pavement, 10 o’clock. Person, 11 o’clock. Traffic light, 2 o’clock. Car, 3 o’clock.” These simple descriptions are enough to paint a picture of the immediate surroundings in the mind of a wearer.
Mahendran’s realization that the technologies used to give robots vision can also be used to improve the lives of humans was a revelation that is likely to profoundly impact the lives of many visually impaired individuals. With a bit more refinement, we can hope to see a more compact, commercial version of this device in the near future.