License Plate Recognition with Vitis-AI

Introduction

The Xilinx Model Zoo contains many pre-built convolutional neural network models.

This project makes use of several of these models, in order to implement a multi-inference application for license plate recognition.

• vehicle detection : ssd_traffic
• license plate detection : platedetect
• license number recognition : platenum

Let's get started !

Step 1 - Create the SD card

Pre-built Vitis-AI 1.3 SD card images have been provided for the following Avnet platforms:

• u96v2_sbc_base : Ultra96-V2 Development Board
• uz7ev_evcc_base : UltraZed-EV SOM (7EV) + FMC Carrier Card
• uz3eg_iocc_base : UltraZed-EG SOM (3EG) + IO Carrier Card

The download links for the pre-built SD card images can be found here:

• Vitis-AI 1.3 Flow for Avnet Vitis Platforms : https://avnet.me/vitis-ai-1.3-project

Once downloaded, and extracted, the.img file can be programmed to a 16GB micro SD card.

0. Extract the archive to obtain the .img file

1. Program the board specific SD card image to a 16GB (or larger) micro SD card

a. On a Windows machine, use Balena Etcher or Win32DiskImager (free opensource software)

b. On a linux machine, use Balena Etcher or use the dd utility

$ sudo dd bs=4M if=Avnet-{platform}-Vitis-AI-1-3-{date}.img of=/dev/sd{X} status=progress conv=fsync

Where {X} is a smaller case letter that specifies the device of your SD card. You can use “df -h” to determine which device corresponds to your SD card.

Step 2 - Clone the source code repository

The source code used in this project can be obtained from the following repository:

• https://github.com/AlbertaBeef/vitis_ai_cpp_examples

If you have an active internet connection, you can simply clone the repository to the root directory of your embedded platform:

$ cd ~
$ git clone https://github.com/AlbertaBeef/vitis_ai_cpp_examples

Step 3 - Overview of the platedetect example

In order to implement the license plate recognition example, we modify an existing example, platedetect, that can be found in the following directory:

~/Vitis-AI/demo/Vitis-AI-Library/samples/platedetect

If we look at the test_jpeg_platedetect.cpp source code, we can see that it is surprisingly small:

int main(int argc, char *argv[]) {
string model = argv[1];
return vitis::ai::main_for_jpeg_demo(
    argc, argv,
    [model] {
        return vitis::ai::PlateDetect::create(model);
    },
    process_result, 2);
}

A visual representation of this code is shown in the following diagram:

We can see that the main function makes use of a generic main_for_jpeg_demo() function, and passes it an instance of the PlateDetect class that provides create() and run() methods, as well as a process_result() function.

The example can be run using the following commands:

1. After boot, launch the dpu_sw_optimize.sh script, which will optimize the QoS configuration for DDR memory

$ cd ~/dpu_sw_optimize/zynqmp
$ source ./zynqmp_dpu_optimize.sh

2. Disable the dmesg verbose output:

$ dmesg -D

3. Launch the platedetect application with the following arguments:

• specify “sample_platedetect.jpg” for the first argument

$ cd ~/Vitis-AI/demo/Vitis-AI-Library/samples/platedetect
$ ./test_jpeg_platedetect sample_platedetect.jpg

The application will generate the output in the following file:

• sample_platedetect_result.jpg

Two observations can be made at this point:

• the platedetect model assumes that there is one license plate in the image it processes (an additional vehicle detection step is required prior to this model)
• the platedetect model only provides the location of the license plate, not its number (an addition number recognition step is required after this model)

Step 4 - Creating the license plate recognition application

We can make use of this generic main_for_video_demo(), with a custom class that defines our modified use case(s), as shown in the following diagram:

For the license plate recognition example, three models are used to implement the 3-inference pipeline:

• vehicle detection : ssd_traffic
• license plate detection : platedetect
• license number recognition : platenum

The following diagram illustrates the modified code for this example.

The source code can be found in the following location:

~/vitis_ai_cpp_examples/platerecognition/test_jpeg_platerecognition.cpp
~/vitis_ai_cpp_examples/platerecognition/test_video_platerecognition.cpp

1. To build the license plate recognition applications

$ cd ~/vitis_ai_cpp_examples/platerecognition
$ ./build.sh

2. To launch the license plate recognition application on an image

$ cp ~/Vitis-AI/demo/Vitis-AI-Library/samples/platedetect/sample_platedetect.jpg .
$ export PLATERECOGNITION_DEBUG=TRUE
$ ./test_jpeg_platerecognition sample_platedetect.jpg

The PLATERECOGNITION_DEBUG environment variable, when set to TRUE, will display information generated by the 3-inference pipeline:

Frame 1
   SSD : label=1 x,y,w,h=2,1,272,304 confidence=0.906937
      PlateDetect : x,y,w,h=103,257,63,21 confidence=0.99977
         PlateNum : size=288,96 color=Blue number=[jingQ2P6J2]

Note that the PlateNum model output number has two parts:

• region : "京Q" (note that "jing" is the Chinese pinyin for 京 )
• number : "2P6J2"

This license plate's region "京Q" corresponds to one of Beijing's urban areas, according to:

• https://en.wikipedia.org/wiki/Vehicle_registration_plates_of_China

3. To launch the license plate recognition application on a video

$ cd ~/vitis_ai_cpp_examples/platerecognition
$ export DISPLAY=:0.0
$ xrandr--output DP-1 --mode 800x600
$ unset PLATERECOGNITION_DEBUG
$ ./test_video_platerecognition./video/plate_recognition_video.mp4

Known Limitations

The license plate recognition implemented in this project work with Asian license plates. In order to support license plates from other regions, additional work is needed.

How would you tackle this ?

• would you retrain the models with images of local license plates ?
• would you use a different technique ?

Conclusion

I hope this tutorial will help you to get started quickly with Vitis-AI 1.3 on the Avnet platforms.

If there is any other related content that you would like to see, please share your thoughts in the comments below.

Revision History

2021/03/15 - First Version

2021/03/17 - Fix link to https://avnet.me/vitis-ai-1.3-project