Accelerated Edge AI with Vitis-AI

Introduction

This guide provides detailed instructions for targeting the Xilinx Vitis-AI 2.5 flow to the following Avnet Vitis 2022.1 platforms:

• Ultra96-V2 Development Board
• ZUBoard 1CG Development Board

This project proposes the same design methodology used by the Kria family, allowing users to dynamically load their various accelerated applications using a single SD image.

The provided design includes the following dynamically loadable "apps", where {platform} corresponds to one of "u96v2" or "zub1cg":

• avnet-{platform}-benchmark
• avnet-{platform}-ar0144-dual
• avnet-{platform}-ar0830-dual

Design Overview

The avnet-{platform}-benchmark accelerated app features the Vitis-AI 2.5 samples, implemented with the Deep Learning Processing Unit (DPU). This design contains the largest DPU architecture that fits in each specific platform.

The avnet-{platform}-{ar0144|ar0830}-dual apps share the same Vivado-only hardware design that contains the MIPI capture pipeline.

The avnet-{platform}-ar0144-dualcam app configures the design (via device tree) for the following dual AR0144 configuration:

The On Semiconductor AP1302 device is an ISP that synchronously captures images from the two AR0144 image sensors, and provides a single side-by-side image on its MIPI interface.

The avnet-{platform}-ar0830-dual app configures the design (via device tree) for the following dual AR0830 configuration:

The AP1302 ISP synchronously captures images from the two AR0830 image sensors, implements auto-gain and auto-white-balance, and provides a single side-by-side image on its MIPI interface.

Getting Started on Ultra96-V2

A pre-built image is provided for Ultra96-V2 with a set of "accelerated apps" that can be dynamically loaded in the programmable logic.

To get started, download the following image, and program to a 16GB or greater microSDcard:

• http://avnet.me/avnet-u96v2-sbc-2022.1-sdimage(2023/01/04 - md5sum : bcdc70b3d9a2d12fcb5fa6edf9ff0dc9)

Configure the Ultra96-V2 as shown in one of the following two diagrams:

The DualCam Mezzanine is optional, and allows to run the stereo examples.

After booting the Ultra96-V2, the list of "accelerated apps" can be queried with the xmutil utility.

$ xmutil listapps
               Accelerator                        Base       Type  #slots  Active

     avnet-u96v2-benchmark       avnet-u96v2-benchmark   XRT_FLAT       0      0,

   avnet-u96v2-ar0144-dual     avnet-u96v2-ar0144-dual   XRT_FLAT       0      -1
   avnet-u96v2-ar0830-dual     avnet-u96v2-ar0830-dual   XRT_FLAT       0      -1

NOTE : There may be more apps shown (and in different order). We are only showing the apps that are covered in this project tutorial.

This output indicates that the "avnet-u96v2-benchmark" app is loaded by default at boot. This is determined by the following file, which can be modified if desired:

$ cat /etc/dfx-mgrd/default_firmware
avnet-u96v2-benchmark

We can query the details of the DPU inside this overlay with the xdputil utility. For the Ultra96-V2, this will provide details of the B2304 DPU:

$ xdputil query
{
    "DPU IP Spec":{
        "DPU Core Count":1,
        "IP version":"v4.0.0",
        "generation timestamp":"2022-05-14 14-30-00",
        "git commit id":"4772d51",
        "git commit time":2022051323,
        "regmap":"1to1 version"
    },
    "VAI Version":{
        "libvart-runner.so":"Xilinx vart-runner Version: 2.5.0-c26eae36f034d5a2f9b2a7bfe816b8c43311a4f8  2022-08-25-17:31:06 ",
        "libvitis_ai_library-dpu_task.so":"Xilinx vitis_ai_library dpu_task Version: 2.5.0-c26eae36f034d5a2f9b2a7bfe816b8c43311a4f8  2022-06-15 07:33:00 [UTC] ",
        "libxir.so":"Xilinx xir Version: xir-c26eae36f034d5a2f9b2a7bfe816b8c43311a4f8 2022-08-25-15:01:04",
        "target_factory":"target-factory.2.5.0 c26eae36f034d5a2f9b2a7bfe816b8c43311a4f8"
    },
    "kernels":[
        {
            "DPU Arch":"DPUCZDX8G_ISA1_B2304",
            "DPU Frequency (MHz)":200,
            "IP Type":"DPU",
            "Load Parallel":2,
            "Load augmentation":"enable",
            "Load minus mean":"disable",
            "Save Parallel":2,
            "XRT Frequency (MHz)":200,
            "cu_addr":"0xa0000000",
            "cu_handle":"0xaaaad155cb00",
            "cu_idx":0,
            "cu_mask":1,
            "cu_name":"DPUCZDX8G:DPUCZDX8G_1",
            "device_id":0,
            "fingerprint":"0x101000016010405",
            "name":"DPU Core 0"
        }
    ]
}

Notice that we have one kernel of type DPU with the B2304 architecture:

"DPU Arch":"DPUCZDX8G_ISA1_B2304",
            "DPU Frequency (MHz)":200,

Getting Started on ZUBoard

A pre-built image is provided with a set of "accelerated apps" that can be dynamically loaded in the programmable logic.

To get started, download the following image, and program to a 16GB or greater microSDcard:

• http://avnet.me/avnet-zub1cg-sbc-2022.1-sdimage(2023/01/04 - md5sum : 1a6b22065b9610736657a3b7b9d63717)

Configure the ZUBoard as shown in the following diagram:

The DualCam module is optional, and allows to run the stereo examples.

After booting the ZUBoard, the list of "accelerated apps" can be queried with the xmutil utility.

$ xmutil listapps
               Accelerator                        Base       Type  #slots  Active

    avnet-zub1cg-benchmark      avnet-zub1cg-benchmark   XRT_FLAT       0      0,

  avnet-zub1cg-ar0144-dual    avnet-zub1cg-ar0144-dual   XRT_FLAT       0      -1
avnet-zub1cg-ar0144-single  avnet-zub1cg-ar0144-single   XRT_FLAT       0      -1
avnet-zub1cg-ar1335-single  avnet-zub1cg-ar1335-single   XRT_FLAT       0      -1

NOTE : There may be more apps shown (and in different order). We are only showing the apps that are covered in this project tutorial.

This output indicates that the "avnet-zub1cg-benchmark" app is loaded by default at boot. This is determined by the following file, which can be modified if desired:

$ cat /etc/dfx-mgrd/default_firmware
avnet-zub1cg-benchmark

We can query the details of the DPU inside this overlay with the xdputil utility. For the ZUBoard, this will provide details of the B512 DPU:

$ xdputil query
{
    "DPU IP Spec":{
        "DPU Core Count":1,
        "IP version":"v4.0.0",
        "generation timestamp":"2022-05-14 14-30-00",
        "git commit id":"4772d51",
        "git commit time":2022051323,
        "regmap":"1to1 version"
    },
    "VAI Version":{
        "libvart-runner.so":"Xilinx vart-runner Version: 2.5.0-c26eae36f034d5a2f9b2a7bfe816b8c43311a4f8  2022-08-25-17:31:06 ",
        "libvitis_ai_library-dpu_task.so":"Xilinx vitis_ai_library dpu_task Version: 2.5.0-c26eae36f034d5a2f9b2a7bfe816b8c43311a4f8  2022-06-15 07:33:00 [UTC] ",
        "libxir.so":"Xilinx xir Version: xir-c26eae36f034d5a2f9b2a7bfe816b8c43311a4f8 2022-08-25-15:01:04",
        "target_factory":"target-factory.2.5.0 c26eae36f034d5a2f9b2a7bfe816b8c43311a4f8"
    },
    "kernels":[
        {
            "DPU Arch":"DPUCZDX8G_ISA1_B512",
            "DPU Frequency (MHz)":300,
            "IP Type":"DPU",
            "Load Parallel":2,
            "Load augmentation":"enable",
            "Load minus mean":"disable",
            "Save Parallel":2,
            "XRT Frequency (MHz)":300,
            "cu_addr":"0xa0000000",
            "cu_handle":"0xaaab1fefa810",
            "cu_idx":0,
            "cu_mask":1,
            "cu_name":"DPUCZDX8G:DPUCZDX8G_1",
            "device_id":0,
            "fingerprint":"0x101000016010200",
            "name":"DPU Core 0"
        }
    ]
}

Notice that we have one kernel of type DPU with the B512 architecture:

"DPU Arch":"DPUCZDX8G_ISA1_B512",
            "DPU Frequency (MHz)":300,

Running the demos

The next sections will provide generic instructions for running the demos. The following replacements can be made for each {platform}:

• u96v2 : Ultra96-V2 Development Board
• zub1cg : ZUBoard 1CG Development Board

Running the "benchmark" demos

The "benchmark" app is analogous to the Kria KV260 "benchmark" app, in the sense that it contains the largest DPU that fits in the device. For the Ultra96-V2, this is the B2304. For ZUBoard, this is the B512 DPU.

If not done so already, load the "benchmark" app, using the xmutil utility.

$ xmutil unloadapp

$ xmutil loadapp avnet-{platform}-benchmark

$ xmutil listapps
                 Accelerator                          Base       Type  #slots  Active

  avnet-{platform}-benchmark    avnet-{platform}-benchmark   XRT_FLAT       0      0,

avnet-{platform}-ar0144-dual  avnet-{platform}-ar0144-dual   XRT_FLAT       0      -1
avnet-{platform}-ar0830-dual  avnet-{platform}-ar0830-dual   XRT_FLAT       0      -1

There are several Vitis-AI demos available, which are available in the "~/Vitis-AI/examples/Vitis-AI-Library" directory.

Multi-Task example

Another more advanced example showcasing a multi-task model (common backbone, multiple heads) can be run as follows:

$ cd ~/Vitis-AI/examples/Vitis-AI-Library/apps/multitask_v3_quad_windows#
$ ./multitaskv3_quad_windows_x d58cbda2-97976be7__640x360.avi -t 4

3D Object Detection

A more advanced example showcasing 3D object detection with lidar point cloud data can be run as follows:

$ cd ~/xilinx_developer/ppdemo/
$ ./demo ./ppd/vlist.txt ./ppd/ 3

If the "demo" application does not run (ie. Permission denied), you can rebuilt it with the following command:

$ source ./build.sh

Running the "ar0144_dual" demos

The "ar0144_dual" app implements a MIPI capture capture in the Vivado project, and is configured for the "dual ar0144" configuration in the device tree.

If not done so already, load the "ar0144_dual" app, using the xmutil utility.

$ xmutil unloadapp

$ xmutil loadapp avnet-{platform}-ar0144-dual

$ xmutil listapps
               Accelerator                        Base       Type  #slots  Active

  avnet-{platform}-benchmark    avnet-{platform}-benchmark   XRT_FLAT       0      -1

avnet-{platform}-ar0144-dual  avnet-{platform}-ar0144-dual   XRT_FLAT       0      0,
avnet-{platform}-ar0830-dual  avnet-{platform}-ar0830-dual   XRT_FLAT       0      -1

A python version of the dual camera passthrough can be launched as follows:

$ cd ~/avnet_dualcam_python_examples
$ python3 avnet_dualcam_ar0144_dual_passthrough.py

Running the "ar0830_dual" demos

The "ar0830_dual" app implements a MIPI capture capture in the Vivado project, and is configured for the "dual ar0830" configuration in the device tree.

If not done so already, load the "ar0830_dual" app, using the xmutil utility.

$ xmutil unloadapp

$ xmutil loadapp avnet-{platform}-ar0830-dual

$ xmutil listapps
               Accelerator                        Base       Type  #slots  Active

  avnet-{platform}-benchmark    avnet-{platform}-benchmark   XRT_FLAT       0      -1

avnet-{platform}-ar0144-dual  avnet-{platform}-ar0144-dual   XRT_FLAT       0      -1
avnet-{platform}-ar0830-dual  avnet-{platform}-ar0830-dual   XRT_FLAT       0      0,

A python version of the dual camera passthrough can be launched as follows:

$ cd ~/avnet_dualcam_python_examples
$ python3 avnet_dualcam_ar0830_dual_passthrough.py

Known Issues with Vitis-AI 2.5

In version 2.5 of Vitis-AI, the support for caffe was dropped.

For this reason, the following models are no longer supported:

• face detection (densebox_640_360, densebox_320_320)
• face applications (face_landmark, face_quality)
• pedestrian detection (ssd_pedestrian_pruned_0_97)
• pose estimation (SP_Net, Openpose_pruned_0_3)
• license plate recognition (plate_detection, plate_recognition)
• etc...

If you need the above models for your development, I recommend using the previous version of the designs, which can be found here:

https://avnet.me/avnet-zub1cg-sbc-2021.2

Conclusion

I hope this tutorial, with its pre-built SD card images, will help you to get your custom AI applications up and running quickly on the Ultra96-V2 and ZUBoard.

If there are any other accelerated apps you would like to see on Ultra96-V2 and/or ZUBoard, please share your thoughts in the comments below.

If you would like detailed instructions on how to re-built these designs, please let me know in the comments below.

Revision History

2023/01/04

Preliminary Version, supporting following platforms

• Ultra96-V2
• Ultra96-V2 + DualCam Mezzanine
• ZUBoard
• ZUBoard + DualCam Module