It is a much-known fact nowadays that the traffic congestion in urban areas has increased to a level that most of the personnel operating get disturbed in their day to day activities. The population growth and the increase in the number of vehicles on the streets are leading to more congestion, higher travel times and more emissions. Therefore, traffic congestion affects the national economy and the productivity of the workforce by wasting long hours on the roads and polluting the environment as well.
Project targets to reduce traffic congestion at a junction by providing a scalable solution. Traffic congestion at a junction is to be reduced by predicting better timing adjustments for each traffic light phases. The conventional signaling systems use timing circuitry, which is time-independent, also, they turn to red and green without any estimation of the current traffic and let the vehicle flow from a junction without knowing the traffic congestion at downstream junctions, blocking a large area with flooding vehicles. This solution is developed in three main parts; Extracting traffic parameters from the live video stream, predicting better timing adjustments for traffic lights and Passing predicted timing adjustments to the existing system.
First part related to work with vision algorithms. From cameras at a junction, we get live video streams to analyze. By analyzing this live video stream, it output vehicle count at the green light phase. The second part of the system is predicting better timing adjustments. The output of the first part, vehicle count feed into the algorithm and it output what time change that should do to increase the number of vehicles passing across the junction at a cycle. The final part of our solution is to pass the predicted timing adjustment to the controller of the existing traffic light system.
The main purpose of developing this system as separate three parts is to implement each parallelly without getting interfered with other modules, so the implementation of those can be done efficiently. Due to the capability of being scalable, this solution can be easily used even in a large traffic network, with minor changes to treat traffic congestion.
The architecture of the expected system has shown below as explained before.
Since this project is associated with an R&D part, it takes a lot of time and effort to implement the system. Therefore there is a practical problem to mount all the systems near a road and perform implementations. For our convenience, we have used a third-party service (Firebase) to transfer data between the traffic light and our implemented system which is at a remote location. In that case, we can monitor videos from IP traffic cameras, test our algorithms and send outputs to the traffic light controller from our remotely located lab.
Following is the modified system architecture. The intermediate device consists of a raspberry pi and a logic level converter.
The main objective from this vison part is to formulate algorithms to extract information from the environment (in our case it is a roundabout) which can be easily used as a state for the time prediction algorithm as a good measure for the congestion level in each phase. We prefer the information which is extracted from the scene should be generalizable for all types of junctions with different characteristics like phases, the field of view, geological pattern and types. We even considered the information that we extracted from the real world should be highly correlated with the simulation environment without scene-specific calibrations. Our goal is to build an algorithm robust in accuracy and capable of real-time performance.
We finally decided to take the number of vehicles that pass in green signal in each phase by considering the above reasons. This gave us a special advantage to get partial information on the traffic conditions in upstream and downstream of the node.
We followed these steps to extract the information of vehicle count.
- Vehicle localization
- Vehicle tracking through consecutive frames by using detected objects
More specifically we need to localize the individual moving vehicles from the scene. We formulated this problem as a foreground separation instead of a classification point of view. Initially, we tried standard digital video processing algorithms such as adaptive background subtraction using Gaussian mixture model and dense optical flow. Their results in localization were not robust due to the bad weather conditions, occlusion and camera shake. In order to implement the real world, we wanted a more consistent algorithm than this. Finally, we used yolo tiny v3 prn architecture and train it from scratch using our collected dataset.
Yolov3 prn achieved the same accuracy as yolov3 tiny with 37% reduction in memory and 38% less computation compares to yolov3-tiny
It runs in 45fps in nano board.
3.1.1.Training the CNN
We collected 3000 images from the Horton place junction. We initially annotated 500 of them and trained yolov3-tiny prn and annotated the balance images using it. Then we went through the model annotation and correct the errors and retrain the model with all images with several image augmentations like random flip, changes in hue and saturation, varying scales and mix-up method. We abled to get 84% mean average precision. We used the
Darknet framework to train the model.
Our trained models are available in the provided repository. The paths of models can be configured using the following lines available in the
configPath = "data_yolov3_tiny/yolov3-tiny-prn.cfg"
weightPath = 'backup/yolov3-tiny-prn_30000.weights'
metaPath = "data_yolov3_tiny/obj.data"
3.2. Vehicle tracking through consecutive frames
We used SORT (Simple online tracking) for tracking the vehicles over frames. It first estimates the location of the bounding boxes using Kalman filter and it uses the Hungarian algorithm to assign each track to the newly detected boxes based on metric of intersection over the union.
We started with one of the state-of-the-art algorithm SORT (Simple Online and Realtime Tracking). It had the following procedures
- Box detection from YOLO-V3-tiny which we trained before.
- Tracking the box coordinates, flow velocity via Kalman filter
- Association of newly detected boxes in the current frame with tracks via Hungarian algorithm using IoU as the metric.
We created a pipeline to handle the process in real-time. In order to identify the traffic congestion, we need the view of two cameras per phase to get the full view of the phase, Therefore we needed to process images from two channels at a phase simultaneously from Jetson nano.
- 1. Decoding RTSP streams
- 2. Detection
- 3. Tracking
- 4. Visualizing the results
Doing them in a synchronous manner will induce varying delays because the time complexity of tracking algorithms is linear with the number of vehicles in the scene. This makes the reading frame from RTSP stream not to stream at a constant rate. This eventually affected the tracking capability itself.
We devised the above tasks as a different process which is running on different cores in jetson nano asynchronously. Reading from RTSP stream and firebase database is an I/O bound task. we used multithreading for each of them and maintained the image retrieval in a fixed frames per second (fps) which is independent of other tasks.
We used queues data structure to communicate with those different processes.
Final_implementation.py code can be used to execute the above-mentioned processes.
- Process 1: Decoding RTSP streams
We read the current phase from the Firebase.
According to the particular phase, we select the two video streams then we get those RTSP video streams to jetson nano to continue with the analysis Since we are considering only a portion of the full-frame, we add a mask to cover the irrelevant areas in order to avoid irrelevant processing. The mask is implemented as the following functions.
def mask_creater(shape, roi_indicater):def img_operations(frame, mask, split):
All the mask data are stored in
- Process 2 : Detection
Here we detect the vehicles in the phase, using the functions in
- Process 3: Tracking
Now we track the vehicles on the frame in order to calculate the flowrate of vehicles using
Here we get the help of two algorithms
1.Linear assignment (
2.kalman filter (
These will be used to predict the timing adjustments using traffic prediction algorithm Then we calculate the green light time for the particular phase and update the firebase with that
def tracking_channel(inp_queue1, inp_queue2, out_queue1, out_queue2, index):
- Process 4 : Visualize
Then we visualize the result of the process of vehicle detection and tracking. (This process has a low priority since this is an optional task)
def visualize(queue_track1, queue_track2, fps):
Here we try to minimize the degree saturation. effective green time is calculated by considering the actual utilization of the roads, considering,
- current flowrate - Vi
- maximum possible flowrate of vehicles - Vmax
- Area occupied - Ai
- Nt - number of frames
- Gt - green time at the particular phase
this calculated green time value is then passed to the PLC.
5.1. System Controller
We need to process two video streams per phase in real-time and process them to output vehicle count in each traffic light phase. To do this kind of application we need a controller that has much higher processing power normal controllers. To output timings to PLC we need at least 8 output pins and to get input from PLC we need 4 input pins for our controller (In the final consumable implementation Fig: System Architecture 1). To satisfy all these conditions we select Jetson nano which is an embedded AI computing device.
Jetson nano is a fast power-efficient embedded AI computing device. Jetson nano allows doing edge computing by providing 7.5-watt supercomputer power. There are 40 GPIOs that are most suitable for our solution implementation and in terms of processing power.
5.2. Traffic light driver
As the controller for most of the existing traffic light systems, they are using a PLC (Programmable Logic Controller) which is programmed by Ladder logics as logic conditions. Ladder logics make PLCs easy to program without much knowledge about programming concepts or languages. It uses long rungs laid out between two vertical bars representing system power. Along the rungs are contacts and coils, modeled after the contacts and coils found on mechanical relays. The contacts act as inputs and often represent switches or pushbuttons; the coils behave as outputs such as a light or a motor.
For our implementation, we are using the Mitsubishi FX3g – 14M PLC.
There are 8 inputs and 8 outputs in this PLC but can extend up to 128 I/Os using extended I/O units.
- Input signal voltage AC power supply type: 24V DC ±10% DC power supply type: 20.4-28.8 V DC
- Input impedance 3.3k Ohm
- Input signal current 7mA/24V DC
- Output voltage 24V or less
- Minimum load (Relay output) 5V DC, 2mA (reference value)
- Response time Approx. 10ms
5.3. Interfacing Jetson nano with PLC
One of our main objectives is to implement our system to the existing traffic light system by doing minimum changes to the existing system. There were several methods that can be used for communication between Jetsonnano and PLC.
- Using mod bus
With this, we want to write data to internal registers of the PLC. But much difficult to implement from jetson nano.
- Using inputs and outputs
By using inputs and outputs we can simply build communication between PLC and Jetson nano board. Jetson nano has 40 GPIO(General Purpose Input/Output) pins and Mitsubishi FX3G-14M PLC has 8 inputs and 8 outputs. But this I/O number increased through certain additional modules by system extension through extension lines.
Following is a comparison between the logic levels of Jetson nano and PLC.
To fill the gap between voltage differences of PLC and Jetson nano we build a 3.3V to 24V level shifter circuit with maximum current rating 1mA. The level shifter has been built with PC817 optocoupler and transistors which can handle up to 30V maximum rating. To limit current to 1mA, we used a current controller circuit operating with an NPN transistor with base current below 1mA.
In our implementation in System Architecture 2, we have used a Raspberry pi board as an intermediate device. The GPIO outputs from the Jetson nano board are passed to the Raspberry pi board through the Firebase cloud service. Therefore the cloud service and the Raspberry pi board can be identified as a carrier agent between the GPIO pins of Jetson nano and Logic level converter. Since the Raspberry pi board has the same logic level as Jetson nano, we don't need to change the logic level converter circuit in the final design.
All design files for the level shifter circuit including the PCB design and circuit schematic have been attached under the schematic section.
Communication between Jetson nano and PLC is done using I/Os of PLC and GPIOs of Jetson nano. First, we convert predicted timing change into binary, then Jetson nano sends parallel 8 bits data set from 8 GPIOs to PLC inputs indicating which traffic phase wants to update and whether it is an increment or decrement for existing timing. PLC get that input and set new phase time for relevant traffic light phase.
- Interrupt – Only send one when need to update phase time
- Increment/Decrement - Indicate whether time should increase or reduce
- Time - Amount of time that wants to change.
With this bit pattern, we can change all the three traffic phases. Total Cycle time is maintained a constant. Timing adjustments are given such that the green light time is limited only between 7 –90 seconds. This can be an increment or decrement for current timing.
5.4. Raspberry Pi
The raspberry pi board is acting as the bitstream carrier between the Jetson nano and the logic level shifter. The jetson nano uploads the data stream to Firebase. The URLs for Firebase need to be updated accordingly in the
Final_implementation.py in the Jetson nano.
# Fetch the service account key JSON file contents
cred = credentials.Certificate(
'Add firebase URL')
# Initialize the app with a service account, granting admin privileges
'databaseURL': 'database URL'
The same needs to be done in the
Firebase_link.py in the raspberry pi board.
firebase = firebase.FirebaseApplication('Firebase Database link', None)
The GPIO pins are assigned as follows in the code.
- 40 - Stop Script pin
- 11, 13, 15 - Read current phase timings from PLC
- 29, 31, 33 - Set time bits (See Fig: Bitstream pattern)
- 35 - Set direction bit (See Fig: Bitstream pattern)
- 19, 21, 23 - Set interrupt for the phase (See Fig: Bitstream pattern)
Final The final stage of this project is to implement this system in a real environment. To complete this successfully, we got relevant permissions from authorities to test the device in an actual roundabout.
We have installed four cameras in the middle of the roundabout focusing on four lanes and got access to the live stream of these cameras. All traffic lights of this junction controlled by a centralized controller placed near the junction. We placed our device in that controller box and communicated with the PLC to perform timing changes.
The following video shows how our method detects vehicles and tracks them using 2 camera views.
As mentioned before, we use two camera views for each green light phase since this junction allows vehicles from two lanes to enter one lane at a time. The following video shows how the processing is done. Phase 2.1 and 2.2 are for one lane and phase 3.1 and 3.2 are for another.
After testing for two consecutive days we obtained the following graphs. Here the dotted line shows the fixed timing values of each phase while the solid lines show the adjusted timing values. In our case, we have 3 phases. Each color shows the green light time of each phase. The y-axis is in seconds. The bar chart shows the actual utilization of each phase in each cycle.
We can clearly see the inefficiency of the existing fixed timing values since it does not try to match the utilization. Timing values with our adjustments were changed according to the vehicle utilization giving preferable results.
We used Mitsubishi GX works 2 to monitor the bit patterns in PLC in order to see how our device works. A total number of 8 signal light heads are controlled by the PLC. Therefore there are 24 bit patterns to control all 24 lights. The following video is captured from a handheld camera since we didn't want to get our processes affected by a screen recorder and mess up the traffic lights in an actual testing environment. It shows how the PLC gets various bitstreams from our device. The changed bits are marked in blue squares.