This "Robotic White Cane" Uses a Depth-Sensing Camera and IMU to Guide Its User Around a Room

Engineers from Virginia Commonwealth University have published a paper detailing a "robotic white cane," which uses depth-sensing camera and computer vision system to assist blind and partially-sighted users with navigation.

Designed to provide real-time navigational feedback to its user, the robotic navigation aid developed by the team is based around an Intel RealSense D435 RGB-D camera system — one of a family of products which the company has confirmed it is winding down, leaving anyone hoping to implement the system looking for alternatives — and a VectorNav VN100 inertial measurement unit (IMU).

The two sensors are fed into an UP Board single-board computer, which handles figuring out the cane's location and orientation on a two-dimensional floor plan of the room. The key to its success: Combining the IMU and depth data with full-colour visuals and using computer vision to track room elements, improving accuracy while dropping the computational requirements.

"In essence," Cang Ye, engineer and corresponding author on the paper, explains, "the method uses geometric features such as door frames, hallways, junctions, etc. from the 2D floor plan map to reduce pose estimation errors."

Feedback, meanwhile, is provided in two forms. The first is an active rolling tip (ART), which steers the cane in a particular direction to guide the user as they walk; when deactivated, the cane can be used like a traditional white cane and relies on a small vibrator on the grip for feedback. Voice guidance is also sent to the user, through a connected Bluetooth headset.

The approach, which the team dubbed depth-enhanced visual-inertial odometry (DVIO), proved its worth in experimental testing — and work is now underway to reduce the size and weight of the prototype robotic cane system.

The team's work has been published under open-access terms in the IEEE/CAA Journal of Automatica Sinica.