DriverPal V2

Introduction

With the advent of deep learning, a rapid change is taking place in almost every technology field. Now with deep learning, we are able to detect objects with near human accuracy. This project is to aimed at using the latest deep learning and computer vision techniques to realize a real time vehicle distance estimation system for autonomous driving.

Design Approach

The solution used for vehicle distance estimation is a two stage process. The first stage is focused on identifying the vehicles using a dashboard camera and generating a bounding box around the vehicle. For this stage I have considered a classifier neural network (ResNet) and a single shot detector (YOLO). In the second stage, the image is transformed to a bird-eye view using perspective transformation. Then the distances to vehicles are calculated using the euclidean distance in the transformed image.

Bird-EyeView

Calculating the distance is straight forward if we have a top view (bird-eye view). But since we only have the dashboard camera input feed we need perform several computer vision algorithms to get a bird-eye view. These steps are shown below

(a) This is the direct output from dashboard camera.

(b) After running canny edge detection on (a).

(c) Region of interest mask that will remove edges that are not on the road.

(d) Detecting lane lines by running probabilistic hough-line detector. The point where lane lines meet is called the vanishing point.

(e) A trapezoidal region that expands from the vanishing point.

(f) A warped view of the trapezoidal region (e). You can see in this view lane lines are parallel to each other. This confirms the view is a bird-eye view

DistanceEstimation

We can calculate the distance to a vehicle by calculating the distance from the bottom center of the image (f) to the bounding box of the vehicle and multiplying it by meters per pixel parameter.

Video Demonstration

Stereo Vision Approach

We also attempted vehicle distance estimation using stereo cameras. This work is still in progress. Following Python notebooks explain the current work we have done.

DenseOpticalFlow.py

• Finds the dense optical flow across the whole image. Shows the optical flow as a heatmap or vector field. You could take the vector field and only show the vectors that are within a bounding box to show the relative velocity of each car

MonoVisOdom.py

• Does feature tracking between frames. Sort of unfinished, wanted to do a bit more with this.

SparseOpticalFlow.py

• Finds the sparse optical flow, a lot less points than the dense optical flow

StereoDistanceEstimation.py

• Generates a depth map from a stereo input. The intrinsic parameters are specific to this dataset (found them in the evaluationutils.py script) and could be used to put real-world distance values to the depth map

StereoSGBMEstimation.py

• Similar to StereoDistanceEstimation but with a different algorithm. Didn't notice a big difference in performance with this algorithm, but I tried it anyways.

What's Next

The current design is not optimized for multi-core processors. The next step would be to use multi-threading to pipeline the two stages.

Acknowledgement

We like to thank Xilinx for sponsoring the hardware (ZCU104) for this project