Easy Machine Learning on Ubuntu with the AMD Kria KV260

Overview

The release of the Ubuntu Desktop 20.04.3 LTS for AMD development boards, which includes the Kria KV260 Vision AI Starter Kit, opens up a world of prototyping and development opportunities. Ubuntu is a great development environment for creating machine learning and computer vision applications, and it is now officially supported by Canonical on several Xilinx development boards.

This project will look at how Ubuntu 20.04.3 on the Xilinx Kria KV260 Vision AI Starter Kit can be used to easily implement accelerated machine learning applications on Xilinx hardware.

Requirements

• KV260 Starter Kit with power supply
• USB Webcam or KV260 basic accessory pack

Project set up

The set up for this project is quite simple, and is broken down into steps below that roughly follow the getting started instructions here.

Step1: Download the Ubuntu Desktop image for the Kria KV260 Vision AI Starter Kit from the Canonical download page.

The download can be found here, or if you are using a Linux PC to download the desktop image file you can use the following command from a terminal:

wget https://people.canonical.com/~platform/images/xilinx/kria/iot-kria-classic-desktop-2004-x03-20211110-98.img.xz?_ga=2.188245288.188752909.1641824366-54830457.1604064728 -O ~/Downloads/iot-kria-classic-desktop-2004-x03-20211110-98.img.xz

Step 2: Burn the downloaded Ubuntu Desktop image file to a blank SD card.

The image file can be written to an SD card using an imaging utility such as BalenaEtcher or if on Linux the dd command. Please make sure to read the following disclaimer before proceeding if using the dd command on Linux:

The following dd command can be used to write the SD card image using Linux:

xzcat ~/Downloads/iot-kria-classic-desktop-2004-x03-20211110-98.img.xz | sudo dd of=/dev/sd<X> bs=32M status=progress

NOTE: The SD card /dev/sd<X> referenced above will be unique to your system. You will need to replace the sd<X> with the appropriate drive mapping for your system. For example, sd<X> could be equal to sda, sdb, sdc, etc. depending on how your system enumerates the SD card device. Make sure the name specified in the of= argument above is the device enumeration, and not a partition (i.e. of=/dev/sd<X> is correct, but of=dev/sd<X>1 would be incorrect).

When the SD card write process completes you can eject the device with the following command

sudo eject /dev/sd<X>

NOTE: The SD card device enumeration in the previous step applies to this step as well.

Step 3: Set up the hardware

The KV260 should be connected as described in the getting started guide located at https://www.xilinx.com/products/som/kria/kv260-vision-starter-kit/kv260-getting-started-ubuntu/connecting-everything.html

Step 4: Boot Ubuntu on the Kria KV260

The Xilinx getting started page describes how to boot Ubuntu on the KV260, and initialize the Ubuntu image with Xilinx utilities. Please follow the instructions at https://www.xilinx.com/products/som/kria/kv260-vision-starter-kit/kv260-getting-started-ubuntu/booting-your-starter-kit.html

Step 5: Install a machine learning accelerator overlay on the KV260

The programmable logic (FPGA fabric) of the KV260 can be dynamically loaded with user customizable hardware accelerators. A pre-built snap is available to install basic capabilities including the programmable logic overlay. The pre-built overlay is the same one used in the NLP-SmartVision accelerated application from the Kria App Store. The NLP-SmartVision app features a B3136 DPU for accelerating machine learning inference applications as well as a MIPI capture pipeline for interfacing with the on board AP1302 ISP. Even though we are installing the NLP-SmartVision app, we can still use the overlay that it provides for creating our own custom application.

The following command taken from the Xilinx Wiki will install the NLP-SmartVision snap.

sudo xlnx-config --snap --install xlnx-nlp-smartvision

NOTE: The xlnx-config utility was installed in step 4 above.

We can also install the Vitis-AI Library examples using a pre-built snap. The Vitis-AI Library snap installs pre-compiled test applications for running various machine learning tasks associated with the models in the Xilinx Vitis-AI Model Zoo.

The following command will install the Vitis-AI Library snap.

sudo snap install xlnx-vai-lib-samples

After the Vitis-AI Library samples snap is installed, we are now set up and ready to create our own custom machine learning application.

Step 6: Load the NLP-SmartVision overlay

Before we can run a Vitis-AI Library example we need to load the programmable logic (PL) overlay that contains the machine learning inference accelerator (i.e. the DPU). The following commands will unload the current PL overlay and load the NLP-SmartVision overlay.

sudo xlnx-config --xmutil unloadapp
sudo xlnx-config --xmutil loadapp nlp-smartvision

Next, create a soft link to the loaded dpu.xclbin

sudo ln -sf /var/snap/xlnx-config/current/assets/dpu.xclbin /usr/lib/dpu.xclbin

NOTE: the dpu.xclbin soft link is used for the dexplorer utility - it is not necessary for running inference applications.

Then, verify the DPU accelerator information

dexplorer -w

You should see the following information:

Step 7: Run a person detection example

In order to run person detection on live video, we need to determine the camera device enumeration using the v4l2-ctrl library. The following command will list the video capture devices in the system

v4l2-ctl --list-devices

You should see something similar to the following

The Webcam device is /dev/video0, and we can pass that as an argument to the person detection test application to use live video from the camera

xlnx-vai-lib-samples.test-video refinedet refinedet_pruned_0_96 /dev/video0

NOTE: if you are using a remote terminal (SSH or serial port) you will need to have the Ubuntu desktop unlocked in order to see the application output. Additionally, you will need to set the XAUTHORITY and DISPLAY environment variables with the commands

export XAUTHORITY=$HOME/.Xauthority
export DISPLAY=:1.0

The MIPI camera from the KV260 basic accessory pack is enumerated as /dev/video2 in Figure 2 above, and the media pipeline is enumerated as /dev/media1. Additional set up is needed for the MIPI camera before it can be used. The commands below will set up the KV260 to use the MIPI camera - make sure to use the correct /dev/video* and /dev/media* for your system.

Set up the media capture pipeline resolutions & formats

media-ctl -d /dev/media1 -V '"ap1302.4-003c":2 [fmt:UYVY8_1X16/1280x720 field:none]'
media-ctl -d /dev/media1 -V '"b0100000.scaler":0 [fmt:UYVY8_1X16/1280x720 field:none]'
media-ctl -d /dev/media1 -V '"b0100000.scaler":1 [fmt:RBG24/1280x720 field:none]'

The commands above set up the MIPI capture pipeline for 1280x720 BGR data.

Run the sample face detection application with MIPI camera input (requires the KV260 basic accessory pack)

xlnx-vai-lib-samples.test-video refinedet refinedet_pruned_0_96 "v4l2src device=/dev/video2 ! video/x-raw, width=1280, height=720, format=BGR, framerate=30/1 ! queue ! appsink"

Create and run a custom machine learning application

The Ubuntu image on the KV260 functions as a development environment, so it's possible to code and compile applications directly on the target hardware. In order to help our development process we will probably want to install the OpenCV development library. The following command will install the OpenCV development package.

sudo apt-get install libopencv-dev python3-opencv

To compile applications with the Vitis-AI Library we also need to install the glog package. The following command installs glog.

sudo apt-get install libgoogle-glog-dev

The following simple code example can be used to perform person detection.

#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <vitis/ai/refinedet.hpp>

using namespace std;

int main( int argc, char *argv[] )
{
  if (argc < 2)
  {
    cout << "Input source not specified" << endl;
    cout << "USAGE: ./person_detection <video source>" << endl;
    return -1;
  }

  cv::VideoCapture vid(argv[1]);

  auto person_detect = vitis::ai::RefineDet::create("refinedet_pruned_0_96");

  while (1)
  {
    cv::Mat frame;
    vid >> frame;

    auto detections = person_detect->run(frame);
    float width = frame.cols;
    float height = frame.rows;

    for (auto &roi : detections.bboxes)
    {
      int xmin = std::min(std::max(roi.x * width, 0.0f), width);
      int ymin = std::min(std::max(roi.y * height, 0.0f), height);
      int xmax = std::min(std::max(xmin + (roi.width * width), 0.0f), width);
      int ymax = std::min(std::max(ymin + (roi.height * height), 0.0f), height);

      cv::rectangle( frame, 
                     cv::Point(xmin, ymin), 
                     cv::Point(xmax, ymax), 
                     cv::Scalar(0,255,0), 
                     1, 
                     1, 
                     0 );
    }

    cv::imshow("Person Detection", frame);
    if (cv::waitKey(1) == 'q') break;
  }
  return 0;
}

You can also download the example code on your Kria with the commands

mkdir ~/person_detect
cd ~/person_detect
wget https://www.hackster.io/code_files/574226/download -O main.cpp

Compile the C++ application above with the following command

g++ -std=c++17 -O3 ~/person_detect/main.cpp -o ~/person_detect/person_detection \
   -I/usr/include/opencv4 \
   -lpthread \
   -lopencv_core \
   -lopencv_video \
   -lopencv_videoio \
   -lopencv_imgproc \
   -lopencv_imgcodecs \
   -lopencv_highgui \
   -lvitis_ai_library-refinedet

Create a symbolic link to the person detection model that was downloaded when the Vitis-AI snap was run. This is necessary because the Vitis-AI-Library API looks for the ML model in the current directory or the /usr/share/vitis_ai_library/models directory if it is not found in the current directory.

ln -sf ~/snap/xlnx-vai-lib-samples/current/models/refinedet_pruned_0_96 ~/person_detect/refinedet_pruned_0_96

NOTE: If you didn't run the Vitis-AI snap person detection example then you can download the model from here. Once the model is downloaded extract it to the ~/person_detect directory.

The following command will run the custom person detection application with a USB webcam (assumes the USB webcam is enumerated as /dev/video0)

cd ~/person_detect
./person_detection /dev/video0

The following command will run the custom person detection application with the MIPI camera (assumes the MIPI camera is enumerated as /dev/video2, and the media pipeline has been set up for 720p resolution using the media-ctl commands shown earlier)

cd ~/person_detect
./person_detection "v4l2src device=/dev/video2 ! video/x-raw, width=1280, height=720, format=BGR, framerate=30/1 ! queue ! appsink"

Create and install a new custom programmable logic overlay

Packaging a custom PL overlay is quite simple using the platform asset container (PAC) format described in the Xilinx wiki. The PAC is a directory structure that uses products from a custom overlay build. To create a custom overlay please see my Create a custom Kria KV260 accelerated ML application project.

The instructions in this section assume that you have followed the Create a custom Kria KV260 accelerated ML application project, and still have the project directory in tact.

NOTE: The KV260 Ubuntu image is based on 2020.2, so make sure to use the 2020.2 build products from the Create a custom Kria KV260 accelerated ML application project.

The PAC directory structure we are going to create looks like:

Execute the following commands on your build machine (the machine that has the build products from the Create a custom Kria KV260 accelerated ML application project)

mkdir -p ~/kv260_ml_accel/kv260-pac/hwconfig/ml-accel/kv260

Copy the build products to the PAC directory using the command below. The build products include a bitstream binary file (kv260-ml-accel.bit.bin), device tree blob overlay file (kv260-ml-accel.dtbo), and an accelerator binary overlay file (kv260-ml-accel.xclbin).

cp ~/kv260_ml_accel/xilinx-k26-starterkit-2020.2.2/build/tmp/sysroots-components/zynqmp_generic/kv260-ml-accel/lib/firmware/xilinx/kv260-ml-accel/* ~/kv260_ml_accel/kv260-pac/hwconfig/ml-accel/kv260/.

Create a shell.json file using the command below. The shell.json file informs the DFX manager of the overlay type. The DFX manager handles loading the overlay.

echo '{
  "shell_type" : "XRT_FLAT",
  "num_slots": "1"
}' >  ~/kv260_ml_accel/kv260-pac/hwconfig/ml-accel/kv260/shell.json

Create the manifest file with the commands below. The manifest file contains meta information describing the hardware configuration.

echo "name: ml-accel
description: KV260 ML inference accelerator (B4096 DPU + Vitis Vision xf::cv::resize)
revision: 1.0
assets:
    kv260: kv260" > ~/kv260_ml_accel/kv260-pac/hwconfig/ml-accel/manifest.yaml

The PAC directory structure should now look like Figure 5 above.

Now that the PAC is created we need to copy it to the KV260. The following command will copy the PAC directory from the build machine to the KV260 over Ethernet.

scp -rp ~/kv260_ml_accel/kv260-pac ubuntu@<kv260-ip-address>:~/Downloads/.

NOTE: You will need to replace the <kv260-ip-address> with the IP address of the KV260. You can find the KV260 IP address by executing ip address show eth0 on the KV260 and then look at the inet address.

From this point forward all commands will be executed on the KV260.

Once the PAC is copied to the KV260 you will need to move the PAC directory to either the /usr/local/share/xlnx-config or the /boot/firmware/xlnx-config directories. The following commands can be used to move the PAC to the correct location on the KV260

sudo mkdir -p /usr/local/share/xlnx-config
sudo mv ~/Downloads/kv260-pac /usr/local/share/xlnx-config/.

We can now query the system to determine which PACs are available. The following command performs the query

xlnx-config --query

You should see a response from the query that looks similar to Figure 6 below. Take note that there is no asterisk next to the ml-accel PAC which we just copied to the /usr/local/share/xlnx-config directory. The lack of asterisk indicates that the PAC is not installed.

We can then install the PAC with the command

sudo xlnx-config --install ml-accel

Run the query again to see that the ml-accel PAC is now installed

xlnx-config --query

Executing the command above should show something similar to Figure 7 below. There is now an asterisk next the ml-accel PAC indicating it is installed and ready for use.

You can now load the new PL overlay with the following commands

sudo xlnx-config --xmutil unloadapp
sudo xlnx-config --xmutil loadapp ml-accel

We can verify that the new overlay is loaded by inspecting the DPU configuration using the dexplorer utility

Figure 8 shows that a B4096 DPU is now loaded in the programmable logic.

Download the person detection model for the B4096 DPU and extract it to a B4096 models directory

mkdir -p ~/xilinx_models/B4096
wget https://www.xilinx.com/bin/public/openDownload?filename=refinedet_pruned_0_96-zcu102_zcu104-r1.3.1.tar.gz -O ~/Downloads/refinedet_pruned_0_96-zcu102_zcu104-r1.3.1.tar.gz
tar -xvzf ~/Downloads/refinedet_pruned_0_96-zcu102_zcu104-r1.3.1.tar.gz -C ~/xilinx_models/B4096

Update the symbolic link in the person detection application directory to point to the B4096 model

cd ~/person_detect
ln -sf ~/xilinx_models/B4096/refinedet_pruned_0_96

We can now run the pedestrian detection application as before with the command

cd ~/person_detect
./person_detection /dev/video0

Summary

This project looked at performing machine learning inference on the Xilinx Kria KV260 Vision AI Starter Kit. Using the Ubuntu image for the KV260 we can easily prototype machine learning inference solutions and create programmable logic overlays with custom hardware accelerators. This method is a great way to quickly get started on a Xilinx platform while working in a familiar Ubuntu environment.

References

• Canonical Ubuntu on Xilinx Wiki
• Xilinx getting started Wiki
• Booting Certified Ubuntu on Xilinx Wiki
• Building Vitis-AI sample applications on Ubuntu for Xilinx Wiki
• Ubuntu tips & tricks on Xilinx Wiki
• Snaps for Ubuntu on Xilinx Wiki
• KV260 Ubuntu Getting Started Guide

Revision History

• 2/9/2022 - Initial release
• 2/9/2022 - Additional content added around installing PACs, added additional references.
• 2/11/2022 - Added note about using the 2020.2 build products when creating a custom PAC