NVive - Nvidia Intelligent Virtual Environmentalism

Overview:

This project demonstrates how to build a device using Nvidia Jetson nano and it's AI frameworks to solve some critical problems which is being neglected by us. The device will be using GPU processing and Deep Stream SDKs to solve blind spot problems in vehicles and a novel approach using the device to curb growing noise pollution on roads which is affecting humans and animals behavior patterns intensely. The project also shows how we can have deep insight about our driving on the road. I got inspiration for this project - " Statistics from the National Highway Traffic Safety Administration show that nearly 840,000 blind spot accidents occur each year in the United States resulting in 300 fatalities. This is partly due to individuals not adjusting their mirrors properly, or not properly checking their blind spots before changing lanes." and also the wikipedia says "Environmental noise can cause hearing impairment, tinnitus, hypertension, ischemic heart disease, annoyance, and sleep disturbance. Changes in the immune system and birth defects have been also attributed to noise exposure." I found that the Nvidia device can be an effective, reliable, cheap solution for all these problems.

Technical Overview:

I used Nvidia Jetson Nano as it has 128 Maxwell cores GPU which is very good for doing transfer learning and IVA, intelligent video analytics plus Nvidia provides it's own SDKs to excel your experience. Also it is much reliable and affordable for the above problem proposed. Also auto driving and smart car assistants have become trending and essential part of any vehicle. However it was challenging for me at first because I am just learning about how things work, a poor 16 years old guy but thanks to Nvidia Hello AI world and DLI course which saved my lots of time and effort. I may have not presented the best way to do things but surely have understood how to analyse a problem in a best way.

Background:

I am focusing upon two major problems which can be solved using AI and image processing in real life.

1) Even though sources of noise pollution include industries, uncontrolled use of loudspeakers and construction activity such as drilling, vehicles are considered to be the main source of noise pollution in cities. Vehicle engines and horns are the main contributors to this. Traffic congestion leads to honking, which in turn leads to noise pollution, he said.When noise is above 45 Db (decibels), it can affect sleep patterns in human beings.Noise pollution can cause physiological as well as psychological damage in human beings, KSCST said, adding that high blood pressure, hearing loss, stress-related illness and sleep disruption are among the common effects. Even pollinating agents and many such animals are affected by this problem, due to the noise they cannot find their way to home. Focusing on healthcare and environment care.

2) Blind spots (A blind spot in a vehicle is an area around the vehicle that cannot be directly observed by the driver while at the controls, under existing circumstances.) and no zone (A no zone is one of several areas around a large truck, where the truck driver cannot see. Collisions frequently occur in no zones.) leads to many fatal accidents now a days. The most common are the rear quarter blind spots, areas towards the rear of the vehicle on both sides. Vehicles in the adjacent lanes of the road that fall into these blind spots may not be visible using only the car's mirrors. Focusing on transport safety.

Step 1: Setup Jetson Pack and get Components Needed for the Project

Installation of OS Image and Jetpack can be downloaded at https://developer.nvidia.com/embedded/jetpack

I used the Jetson DLICourse OS image https://developer.download.nvidia.com/training/nano/ainano_v1-1-1_20GB_200203A.zip with preinstalled Jetpack, and ready for Deep Stream and all other required frameworks. Follow the procedure mentioned in Hello AI world documentation and you will finish installing all required AI frameworks for Jetson Nano https://github.com/dusty-nv/jetson-inference

Setting up the libraries and dependencies apart from jetson inference, Since we are using built in SDKs OS so will have to download very less packages.

1. Installing Tensorflow on Jetson nano

$ sudo apt-get install libhdf5-serial-dev hdf5-tools libhdf5-dev zlib1g-dev zip libjpeg8-dev
$ sudo apt-get install python3-pip
$ sudo pip3 install -U pip
$ sudo pip3 install -U numpy grpcio absl-py py-cpuinfo psutil portpicker six mock requests gast h5py astor termcolor protobuf keras-applications keras-preprocessing wrapt google-pasta
# TF-2.0
$ sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v43 tensorflow-gpu==2.0.0+nv20.1
# TF-1.15
$ sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v43 tensorflow-gpu==1.15.0+nv20.1

2. Install Requests python package which we will use to send commands to particular IoT devices using REST API and in return they will proceed with our intended actions.

$sudo pip install requests

For IoT device I have used Particle Argon, they are cool to use and have well documentation and support many APIs and Thread Protocols.

3. Install Nvidia Jetson Nano GPIO library, in jetson nano OS image,go here /opt/nvidia/jetson-gpio/doc/README.txt. you will find all the steps

4. Install tesseract on your device and download python package

sudo apt install tesseract-ocr libtesseract-dev
pip install pytesseract

Programming the AI:

I will first solve the blind spot problem because many things will be similar in vehicle noise pollution reducer problem. Features to focus upon here:

* Real-time video streaming with high priority obstacle tracking and alert * Rear drivers path predictions, lane change detection, feature extraction, using optical flow

I used ssd-mobilenet-v2, which came with jetson inference package, it is extremely light and can support upto 30fps.

""" importing all the required dependencies; jetson packages are essential to use prebuilt models, cv2 and numpy plays the whole game of AI"""
import jetson.inference
import jetson.utils
from graphics import *
#from PIL import Image
import cv2
import numpy as np

net = jetson.inference.detectNet("ssd-mobilenet-v2", threshold=0.5)
camera = jetson.utils.gstCamera(480, 360, "0")  # using CSI
display = jetson.utils.glDisplay()

make an array or list to track only those items which are relevant to our use case,

class_id = {1, 2, 3, 4, 6, 8, 18, 21}  # class id for detection to look for
vehicle = {3, 6, 8}                    # these are the class id for bus,truck and cars which will be shown in different colour spots to track their path
living = {1, 18, 21}                   # person, cow ,dog class id which will be shown in different colour spot

Setting up our own custom window where I show detected objects relative to actual video streaming in form of spots, this is very meaningful because image contain too much details for any human even if they are properly labeled, it will still be tedious for him to understand through images so I decided to show the spots where the objects are present on a relative window, different color will represent different category but my intention was to use a background filter and only display only meaningful objects but image segmentation still need a better understanding and does not fit with my code so I displayed the relative positions on a black background and make their path predictions and number of objects present nearby easier. However background overlay can be used to refine it but since I am on 5v/2.5A I would not try to push the limits of Jetson(lack of resource ). Here is the code.

for detection in detections :
identity = detection.ClassID
if identity in class_id:
x, y = detection.Center # take note of detected object central node
display.RenderOnce(img, width, height)
display.SetTitle("Object Detection | Network {:.0f} FPS".format(net.GetNetworkFPS()))
if identity in vehicle:
head = Circle(Point(x,y), 10)
head.setFill('blue')
head.draw(win)
win.update()
elif identity in living:
head = Circle(Point(x,y), 10)
head.setFill('green')
head.draw(win)
win.update()
else:
head = Circle(Point(x,y), 10)
head.setFill('red')
head.draw(win)
win.update()

Using the dense optical flow to compute the optical flow for all the points in the frame. It is based on Gunner Farneback’s algorithm. It shows the motion and helps in better and faster motion prediction of vehicles or other objects compromising the device's computational power.

def optic(frame1):
global fpr
if fpr == None:
global prvs
global hsv
prvs = cv2.cvtColor(frame1,cv2.COLOR_BGR2GRAY)
hsv = np.zeros_like(frame1)
hsv[...,1] = 255
fpr=1
else:
fil = cv2.cvtColor(frame1,cv2.COLOR_BGR2GRAY)
flow = cv2.calcOpticalFlowFarneback(prvs,fil, None, 0.5, 3, 15, 3, 5, 1.2, 0)
mag, ang = cv2.cartToPolar(flow[...,0], flow[...,1])
hsv[...,0] = ang*180/np.pi/2
hsv[...,2] = cv2.normalize(mag,None,0,255,cv2.NORM_MINMAX)
rgb = cv2.cvtColor(hsv,cv2.COLOR_HSV2BGR)
cv2.imshow('frame2',rgb)
#cv2.imwrite('frame2.png',rgb)   to save frame as picture
k=cv2.waitKey(30) & 0xff
if k==27:
global p
p=1
elif k == ord('s'):
print("updated")
prvs = fil

Let's check the blind-spot detection model: I truly wanted to use 3D cameras and Lidar sensors but my budget was very small, a poor 16 yrs old guy. To test the model I opened some videos on laptop and focused NVIDIA camera to it and it ran it's functions well, it would be exciting to use it on road but I don't owe any vehicle.

AI vehicular noise pollution reduction:

To tackle this problem I found a new way to introduce horn in vehicles. Traditional methods include loud, frustrating, persistent honking for the front vehicles but mine one is truly different because "I have trained it on the lines of AI" - so I introduce here two horns concept in vehicles, one horn called internal horn and the other external horn. what happens here is that when the AI program runs it tracks the objects around using the camera, again we specified the particular objects to be detected, so whenever anything is detected it checks whether it comes from defined object class or not and after that it checks whether to active internal horns or external horns to clear our way. When the user press the hron button the program would decide whether to use interal or external horn

Since i used Particle devices because I don't want to trigger nvidia device remotely making it more prone to get hacked easily, so IoT devices are reliable and we can make encryption for our access token more tight too. Also particle devices support Cloud API based upon REST API, read here https://docs.particle.io/reference/device-cloud/api/ so I can call any exposed function of device remotely. I used this way because we cannot expect every device to be on internet and flooding with data , so I needed a lighter protocol to communicated among all the vehicles so REST API suits the best for me.

here is the python code to communicate with device with CLOUD API, check the status of your device and send commands to activate pins.

#!/usr/bin/python
import requests
access_token = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxx'    # open your particle IDE and copy access token from there
device_id = 'xxxxxxxxxxxxxxxxxxxxxxxxx' # each device has one unique id
command1='D7 LOW'
command2='D7 HIGH'
address = 'https://api.particle.io/v1/devices/{0}'.format(device_id)
address1 = 'https://api.particle.io/v1/devices/{0}'.format(device_id)+'/'+'digitalwrite'
headers = {'Authorization':'Bearer {0}'.format(access_token)}
data = {'access_token': access_token,'args': command2}
get = requests.get(address, headers=headers)
r = requests.post(address1, data=data)
print(get.text)
print(r.text)

Coming to code of our Jetson nano:

I will break this part into chunks so that it becomes easier to understand, first we will learn how to take input from the camera and check whether it's living thing or vehicle, the function starts rendering the camera and checks whether the desired objects are encountered or not if incase it be the vehicle then it will try to detect it's license plate number.

while display.IsOpen():
img, width, height = camera.CaptureRGBA(zeroCopy=1)
jetson.utils.cudaDeviceSynchronize ()
img1, width1, height1 = camera.CaptureRGBA(zeroCopy=1)
jetson.utils.cudaDeviceSynchronize ()
detections = net.Detect(img, width, height)
l = 0
p = 0
prev_value = None
for detection in detections :
#print(detection)
class_id = detection.ClassID
if class_id in identity:
display.RenderOnce(img, width, height)
display.SetTitle("Object Detection | Network {:.0f} FPS".format(net.GetNetworkFPS()))
if class_id in vehicle:
l = l+1
array = jetson.utils.cudaToNumpy(img1, width1, height1, 4)
dt = array.dtype
sh = array.shape
array2 = np.ones(sh, np.float32)
arr = array + array2
aimg = cv2.cvtColor(arr.astype (np.uint8), cv2.COLOR_RGBA2BGR)
#cv2.imshow('img', aimg)
#cv2.waitKey(0)
device_id = license_as_id() # calling function to convert the detected contour to license number
print (device_id)
device_id.replace(" ", "")
ids.append(device_id)

else:
p = p+1
else:
display.RenderOnce(img1, width1, height1)
display.SetTitle("Object Detection | Network {:.0f} FPS".format(net.GetNetworkFPS()))
global s
s = GPIO.input(horn_button) # check whether horn button was pressed or not
if s != prev_value:
if p >= 1:
GPIO.output(ext_horn, GPIO.HIGH)
else:
if l >= 1:
message() # if button pressed and car detected send internal horn activation using REST API
GPIO.output(ext_horn, GPIO.LOW)
for key in ids:
ids.remove(key) #remove every device id to start new detection

Now detecting license from contours found from above snippet using pytesseract

def license_as_id():
image = cv2.resize(aimg,(620,480))
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
gray = cv2.bilateralFilter(gray, 11, 17, 17)
edged = cv2.Canny(gray, 30, 200)
cnts = cv2.findContours(edged.copy(), cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
cnts = imutils.grab_contours(cnts)
cnts = sorted(cnts, key = cv2.contourArea, reverse = True)[:10]
screenCnt = None
text = ""
for c in cnts:
peri = cv2.arcLength(c, True)
approx = cv2.approxPolyDP(c, 0.018 * peri, True)
if len(approx) == 4:
screenCnt = approx
break
if screenCnt is None:
detected = 0
print("No contour detected")
else:
detected = 1
if detected == 1:
cv2.drawContours(image, [screenCnt], -1, (0, 255, 0), 3)
mask = np.zeros(gray.shape,np.uint8)
new_image = cv2.drawContours(mask,[screenCnt],0,255,-1,)
new_image = cv2.bitwise_and(image,image,mask=mask)
(x, y) = np.where(mask == 255)
(topx, topy) = (np.min(x), np.min(y))
(bottomx, bottomy) = (np.max(x), np.max(y))
Cropped = gray[topx:bottomx+1, topy:bottomy+1]
text = pytesseract.image_to_string(Cropped, config='--psm 11')
print("Detected Number is:",text)#return (text)
cv2.imshow('Cropped',Cropped) # I used this to check weather the device id is recognized while running the program
#cv2.waitKey(0)
#cv2.destroyAllWindows()
return (text)

Now comes the code to send the command to particular device using REST API, the internal horn once triggered they can be normalized either by the targeted user or making a shot tone through the program and automatically switching it off. Since I could not make a database for license and subsequent device ID so I made a small python dictionary to hold my corresponding device id to my number plate.

dictionary = {'4da7a7a16973cf8da612a3b47a658b1810b9b290' : 'HR26DA2330', 'e00fce681ecfedb1b1d5b74a' : 'KA03AB3289'} #device id associated with license number, I had only two argon devices

def message():
for key in ids:
for example_device_id,ref_device_id in dictionary.items():
if key == ref_device_id:
data = {'access_token': access_token,'args': command2} #activating the internal horn of targeted vehicle
r = requests.post(address, data=data)
print(r.text)

Testing our code and implementation:

The license feature was not working so well as the contours formed looked very different sometimes, I need to train the model on a heavy GPU then it would work fine, but my bad I don't have enough money to buy those things, I wish NVIDIA and other future tech companies like this implementation and it is the time now to implement this as 5G will be a boon for this.

So we have made a system in which we only need to trigger the external horn only when living things are present otherwise we will communicate with vehicle around and send commands to trigger their internal horn, however I tried to do selective command sending but couldn't do for now but complete it within few days. So if this feature is implemented in all vehicles we will solve one of the greatest unchecked problems with vehicle horns and environment noise pollution.

I will improve more upon it further, I am really thankful to NVIDIA for sponsoring such innovative contest being open to even school kids like me.