Smarter Parker: Smart Parking Lot Monitoring System

Smarter Parker w/ PYNQ on Xilinx Ultra96 v2 Board

The Smarter Parker project is a system which uses a single camera to extract meaningful information from parking lots for later use in decision making, monitoring and ensuring safety of people and property. Millions of people waste several minutes if not hours of their lives in trying to find available parking spots to park their cars every day, albeit in offices, universities, restaurants, shopping malls, sports stadiums, hospitals, etc. Efficient parking spot localization and broadcasting is the best use case for a system such as Smarter Parker.

The implementation for Smarter Parker utilizes the latest Ultra96 v2 development board by Xilinx for edge computing of the necessary computer vision algorithms. The system consists of three parts: (1) Deep Learning Based Inferencing, (2) Post-processing and Analysis, and (3) Web Server Dashboard. The pipeline of algorithms that Smarter Parker begins by taking a stream of images from a monocular camera as input. The images are cropped and pre-processed before being passed into an optimized YOLOv3 running on the Xilinx Vitis AI stack. Finally, the results are post-processed and served on a web-based dashboard, all while running on just the single Ultra96 board.

The pipeline relies on two Python scripts running in parallel, utilizing file-based inter-process communication. The first script handles data input and YOLOv3 bounding box generation and storage into CSV files. The second script parses the CSV scripts, processes the information stored and displays the results using a Flask-Bootstrap web server. The following are all the steps needed to set up the system:

Step 1: Setup and Configure PYNQ on Ultra96 v2

The first step is to flash a PYNQ image onto a Micro SD card and configure features such as WiFi on the Ultra96 v2 board. Follow this fantastic video on setting up your Ultra96 board for running PYNQ. (Video).

Step 2: Setup Xilinx Vitis AI on top of the PYNQ image

The next step is to install the Hardware Acceleration layer for using PYNQ-DNN based Deep Learning models on your Ultra96 board. Follow this great Hackster tutorial by Wadulisi on how to do so. (Tutorial)

Step 3: Download the Code and Run the Smarter Parker Scripts.

The source code consists of two Python scripts 'parking.py' and 'parking_server.py', a Bash script 'launch.sh' for handling deployment of the system, and the Flask-Bootstrap template files needed for the web-based dashboard. Download the Source Code ZIP from the Github Repository file onto the PYNQ installation on your Ultra96 board, extract it to your desired directory and run the 'launch.sh' script as "bash launch.sh".

The 'parking.py' script uses the pynq_dpu to create an FPGA environment and loads the optimized YOLOv3 weights into the device for inferencing. The complete code for the script has been attached below.

from pynq_dpu import DpuOverlay
overlay = DpuOverlay("dpu.bit")
overlay.load_model("dpu_tf_yolov3.elf")

import numpy as np
import random
import cv2
import colorsys
from matplotlib.patches import Rectangle
import matplotlib.pyplot as plt
#%matplotlib inline
from pynq_dpu.edge.dnndk.tf_yolov3_voc_py.tf_yolov3_voc import *

anchor_list = [10,13,16,30,33,23,30,61,62,45,59,119,116,90,156,198,373,326]
anchor_float = [float(x) for x in anchor_list]
anchors = np.array(anchor_float).reshape(-1, 2)

classes_path = "files/voc_classes.txt"
class_names = get_class(classes_path)

num_classes = len(class_names)
hsv_tuples = [(1.0 * x / num_classes, 1., 1.) for x in range(num_classes)]
colors = list(map(lambda x: colorsys.hsv_to_rgb(*x), hsv_tuples))
colors = list(map(lambda x: 
                  (int(x[0] * 255), int(x[1] * 255), int(x[2] * 255)), 
                  colors))
random.seed(0)
random.shuffle(colors)
random.seed(None)

KERNEL_CONV="tf_yolov3"
CONV_INPUT_NODE="conv2d_1_convolution"
CONV_OUTPUT_NODE1="conv2d_59_convolution"
CONV_OUTPUT_NODE2="conv2d_67_convolution"
CONV_OUTPUT_NODE3="conv2d_75_convolution"

def draw_boxes(image, boxes, scores, classes):
    _, ax = plt.subplots(1)
    ax.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
    image_h, image_w, _ = image.shape

    for i, bbox in enumerate(boxes):
        [top, left, bottom, right] = bbox
        width, height = right - left, bottom - top
        center_x, center_y = left + width*0.5, top + height*0.5
        score, class_index = scores[i], classes[i]
        label = '{}: {:.4f}'.format(class_names[class_index], score) 
        color = tuple([color/255 for color in colors[class_index]])
        ax.add_patch(Rectangle((left, top), width, height,
                               edgecolor=color, facecolor='none'))
        ax.annotate(label, (center_x, center_y), color=color, weight='bold', 
                    fontsize=12, ha='center', va='center')
    return ax
    
def evaluate(yolo_outputs, image_shape, class_names, anchors):
    score_thresh = 0.2
    anchor_mask = [[6, 7, 8], [3, 4, 5], [0, 1, 2]]
    boxes = []
    box_scores = []
    input_shape = np.shape(yolo_outputs[0])[1 : 3]
    input_shape = np.array(input_shape)*32

    for i in range(len(yolo_outputs)):
        _boxes, _box_scores = boxes_and_scores(
            yolo_outputs[i], anchors[anchor_mask[i]], len(class_names), 
            input_shape, image_shape)
        boxes.append(_boxes)
        box_scores.append(_box_scores)
    boxes = np.concatenate(boxes, axis = 0)
    box_scores = np.concatenate(box_scores, axis = 0)

    mask = box_scores >= score_thresh
    boxes_ = []
    scores_ = []
    classes_ = []
    for c in range(len(class_names)):
        class_boxes_np = boxes[mask[:, c]]
        class_box_scores_np = box_scores[:, c]
        class_box_scores_np = class_box_scores_np[mask[:, c]]
        nms_index_np = nms_boxes(class_boxes_np, class_box_scores_np) 
        class_boxes_np = class_boxes_np[nms_index_np]
        class_box_scores_np = class_box_scores_np[nms_index_np]
        classes_np = np.ones_like(class_box_scores_np, dtype = np.int32) * c
        boxes_.append(class_boxes_np)
        scores_.append(class_box_scores_np)
        classes_.append(classes_np)
    boxes_ = np.concatenate(boxes_, axis = 0)
    scores_ = np.concatenate(scores_, axis = 0)
    classes_ = np.concatenate(classes_, axis = 0)

    return boxes_, scores_, classes_
    
    
image_path = "img/greyfox-672194.JPEG"
video_path = "/home/xilinx/jupyter_notebooks/pynq-dpu/video/parking_lot_k_01.mp4"

cap = cv2.VideoCapture(video_path)
if (cap.isOpened()== False): 
  print("Error opening video stream or file")

n2cube.dpuOpen()
kernel = n2cube.dpuLoadKernel(KERNEL_CONV)
task = n2cube.dpuCreateTask(kernel, 0)

count = 0

ux, uy, dx, dy = 400, 400, 800, 1000
fourcc = cv2.VideoWriter_fourcc(*'MPEG')
out = cv2.VideoWriter("../output/result.avi", fourcc, 20.0, (dy-uy,dx-ux))

while(cap.isOpened()):
    ret, frame = cap.read()
    image = frame[ux:dx, uy:dy].copy()
    count += 1
    if ret == True and count % 4 == 0:

        image_size = image.shape[:2]
        image_data = np.array(pre_process(image, (416, 416)), dtype=np.float32)

        input_len = n2cube.dpuGetInputTensorSize(task, CONV_INPUT_NODE)
        n2cube.dpuSetInputTensorInHWCFP32(task, CONV_INPUT_NODE, image_data, input_len)
        n2cube.dpuRunTask(task)
        
        conv_sbbox_size = n2cube.dpuGetOutputTensorSize(task, CONV_OUTPUT_NODE1)
        conv_out1 = n2cube.dpuGetOutputTensorInHWCFP32(task, CONV_OUTPUT_NODE1, conv_sbbox_size)
        conv_out1 = np.reshape(conv_out1, (1, 13, 13, 75))
        conv_mbbox_size = n2cube.dpuGetOutputTensorSize(task, CONV_OUTPUT_NODE2)
        conv_out2 = n2cube.dpuGetOutputTensorInHWCFP32(task, CONV_OUTPUT_NODE2, conv_mbbox_size)
        conv_out2 = np.reshape(conv_out2, (1, 26, 26, 75))
        conv_lbbox_size = n2cube.dpuGetOutputTensorSize(task, CONV_OUTPUT_NODE3)
        conv_out3 = n2cube.dpuGetOutputTensorInHWCFP32(task, CONV_OUTPUT_NODE3, conv_lbbox_size)
        conv_out3 = np.reshape(conv_out3, (1, 52, 52, 75))
        yolo_outputs = [conv_out1, conv_out2, conv_out3]   
        
        boxes, scores, classes = evaluate(yolo_outputs, image_size, class_names, anchors)
                            
        np.savetxt("../output/text/" + str(count) + ".txt", boxes, delimiter=',', fmt='%d')    
        print(count)

cap.release()
out.release()
n2cube.dpuDestroyTask(task)
n2cube.dpuDestroyKernel(kernel)

The 'parking_server.py' script parses the CSV files generated by the 'parking.py' script and generates the final post-processed output for being displayed on the web server using Flask and Flask-Bootstrap. The complete code for parking_server.py has also been attached below.

#!/usr/bin/env python
import os
import time

import cv2
import numpy as np
from flask import Flask, render_template, Response, redirect
from flask_bootstrap import Bootstrap

desktopVideo = "../Videos/ParkingLotKCropped.mp4"
ultra96Video = "/home/xilinx/jupyter_notebooks/pynq-dpu/video/parking_lot_k_01.mp4"
resultVideo = "../output/result.avi"

ultra96Skip = 4
desktopSkip = 1

video = ultra96Video
skip = 1

app = Flask(__name__)
Bootstrap(app)

result = None
yolo3_output = np.array([0, 0])
ready = False
yolo3_ready = False
process = False
count = 0
finish = False


@app.route('/')
def index():
    images = os.listdir(app.static_folder)
    return render_template('index.html', images=images)


@app.route('/signup', methods=['POST'])
def signup():
    global finish
    print("POST: Received")
    finish = True
    return redirect('/')


def gen(cap):
    global result, ready
    count = 0
    while True:
        count += 1
        ret, frame = cap.read()

        if not (ret):
            continue

        if not (count % 4 == 0):
            continue

        boxes = np.loadtxt("../output/text/" + str(count) + ".txt", dtype=np.int32, delimiter=",")

        frame = frame[400:800, 400:1000]
        for box in boxes:
            cv2.rectangle(frame, (box[1], box[0]), (box[3], box[2]), (255, 0, 255), 3)

        time.sleep(0.02)
        frame = cv2.resize(frame, (int(frame.shape[1] / 2), int(frame.shape[0] / 2)))
        frame = cv2.resize(frame, (int(frame.shape[1] / 2), int(frame.shape[0] / 2)))
        success, img_enc = cv2.imencode('.jpg', frame)
        result = img_enc.tobytes()

        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + result + b'\r\n')


@app.route('/video_feed')
def video_feed():
    return Response(gen(cv2.VideoCapture(video)),
                    mimetype='multipart/x-mixed-replace; boundary=frame')


if __name__ == '__main__':
    app.run(host='0.0.0.0', debug=True)

The web server displays the tracked vehicles and humans in the parking lot in real-time. The server also catalogues all the vehicles and people observed by Smarter Parker since the the system start time.

Step 6: Download Video to Test Smarter Parker On

For testing purposes, we provide a free and open-licensed video file to test the Smarter Parker system on. Please download the video below:

Use the "DOWNLOAD ALL" button on this Google Drive folder to download the video (Drive) to your desired location.

Modify this line in 'parking.py' to point to the video you downloaded.

ultra96Video = "/home/xilinx/jupyter_notebooks/pynq-dpu/video/parking_lot_k_01.mp4"

Modify this line in 'parking_server.py' to point to the same video file:

video_path = "/home/xilinx/jupyter_notebooks/pynq-dpu/video/parking_lot_k_01.mp4"

Step 6: Access the Output on Flask Web Server

Watch the output on the web server using the IP address provided by the console output from 'launch.sh' bash script. The real-time video stream should be the first element on the web page. The catalogue of all vehicles and persons is displayed towards the lower part of the dashboard. Most likely the server will appear at 0.0.0.0:5000 on your localhost so browse to it on your browser..

The Smarter Parker system is not only a complete pipeline of algorithms for parking lot monitoring in itself, but it also provides a platform for hobbyists and researchers alike to create more advanced parking monitoring applications for future use. That brings us to the end of this tutorial. I hope you enjoyed the ride. I'd like to thank and acknowledge all the great hackers and makers which helped generate the intermediate aspects for this entire pipeline.