Creating a Vitis-AI GStreamer Plugin for the Ultra96-V2

This project describes steps that can be used to create a GStreamer plugin that uses the Xilinx Vitis-AI Library. This tutorial provides detailed steps to create face detection GStreamer plugin. The plugin is then tested on the Ultra96-V2 platform, but can be used with any Xilinx Vitis-AI based platform. Project development is done on an Ubuntu 18.04 Linux host machine.

Part 1: Project setup

• Create a project directory of your choosing (i.e. ~/gst_plugin_tutorial) and create an environment variable that points to that location.

mkdir ~/gst_plugin_tutorial
export GST_PROJ_DIR=~/gst_plugin_tutorial

• Clone this tutorial's repository from the Avnet GitLab server

cd $GST_PROJ_DIR
git clone https://xterra2.avnet.com/xilinx/ml/vai-gst-plugin-tutorial

Part 2: Setting up the Linux host cross-compilation environment

This section describes how to add Vitis-AI-Library support to a common PetaLinux SDK available for download from Xilinx.

NOTE 1: This section follows the steps provided in the Setting Up the Host section of the Vitis-AI-Library setup instructions.

NOTE 2: All steps in this section are performed on the host machine

• Download and extract the common PetaLinux SDK

wget https://www.xilinx.com/bin/public/openDownload?filename=sdk-2020.1.0.0.sh -O sdk-2020.1.0.0.sh 
sh -x sdk-2020.1.0.0.sh -d ~/petalinux_sdk_2020.1 -y

The PetaLinux SDK is installed in the ~/petalinux_sdk_2020.1 directory

• Create an environment variable that points to the cross-compilation target root file system.

export SYSROOT=~/petalinux_sdk_2020.1/sysroots/aarch64-xilinx-linux

• Source the cross-compilation environment setup script

unset LD_LIBRARY_PATH
source $SYSROOT/../../environment-setup-aarch64-xilinx-linux

• Download the Vitis-AI package

wget https://www.xilinx.com/bin/public/openDownload?filename=vitis_ai_2020.1-r1.2.0.tar.gz -O vitis_ai_2020.1-r1.2.0.tar.gz

• Install the Vitis-AI package in the cross-compilation environment

tar -xvzf vitis_ai_2020.1-r1.2.0.tar.gz -C $SYSROOT

Part 3: Creating and cross-compiling the GStreamer plugin

This section describes how to create a GStreamer video filter plugin from template, and customize it to use the Vitis-AI-Library. The culmination of this section will result in a custom Vitis-AI face detection plugin

• Clone the GStreamer plugins-bad repository

git clone https://github.com/GStreamer/gst-plugins-bad.git

• Create the face detection plugin from the video filter template

mkdir -p vaifacedetect 
cd vaifacedetect
../gst-plugins-bad/tools/gst-element-maker vaifacedetect videofilter
rm *.so *.o
mv gstvaifacedetect.c gstvaifacedetect.cpp

NOTE: You may see a warning indicating gst-indent: command not found. As long as the plugin .c and .h file are created then it should be okay.

• The face detection plugin will process video frames in-place instead of copying data from an input buffer to an output buffer. Remove references in the plugin template code to the tranform_frame() function, but make sure to leave the references to transform_frame_ip(). The following command will delete the lines of code that set the transform_frame function in the class_init() function.

sed -i '/video_filter_class->transform_frame = /d' gstvaifacedetect.cpp

NOTE: The transform_frame() function is used when data from the input buffer is processed and then copied to a different output buffer. In this particular application the buffer copy is not necessary and the in-place (transform_frame_ip) function is used instead to modify the video frame data.

• Add the OpenCV and Vitis-AI-Library header files to gstvaifacedetect.cpp

/* OpenCV header files */
#include <opencv2/core.hpp>
#include <opencv2/opencv.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>

/* Vitis-AI-Library specific header files */
#include <vitis/ai/facedetect.hpp>

• Update the pad templates in gstvaifacedetect.cpp to reflect the supported pixel formats

FROM:

/* FIXME: add/remove formats you can handle */
#define VIDEO_SRC_CAPS \
    GST_VIDEO_CAPS_MAKE("{ I420, Y444, Y42B, UYVY, RGBA }")

/* FIXME: add/remove formats you can handle */
#define VIDEO_SINK_CAPS \
    GST_VIDEO_CAPS_MAKE("{ I420, Y444, Y42B, UYVY, RGBA }")

TO:

/* Input format */
#define VIDEO_SRC_CAPS \
    GST_VIDEO_CAPS_MAKE("{ BGR }")

/* Output format */
#define VIDEO_SINK_CAPS \
    GST_VIDEO_CAPS_MAKE("{ BGR }")

• Update the *_class_init() function in gstvaifacedetect.cpp

FROM:

/* Setting up pads and setting metadata should be moved to
   base_class_init if you intend to subclass this class. */
gst_element_class_add_pad_template (GST_ELEMENT_CLASS(klass),
    gst_pad_template_new ("src", GST_PAD_SRC, GST_PAD_ALWAYS,
      gst_caps_from_string (VIDEO_SRC_CAPS)));
gst_element_class_add_pad_template (GST_ELEMENT_CLASS(klass),
    gst_pad_template_new ("sink", GST_PAD_SINK, GST_PAD_ALWAYS,
      gst_caps_from_string (VIDEO_SINK_CAPS)));

gst_element_class_set_static_metadata (GST_ELEMENT_CLASS(klass),
    "FIXME Long name", "Generic", "FIXME Description",
    "FIXME <fixme@example.com>");

TO:

/* Setting up pads and setting metadata should be moved to
   base_class_init if you intend to subclass this class. */
gst_element_class_add_pad_template (GST_ELEMENT_CLASS(klass),
    gst_pad_template_new ("src", GST_PAD_SRC, GST_PAD_ALWAYS,
      gst_caps_from_string (VIDEO_SRC_CAPS ", width = (int) [1, 640], height = (int) [1, 360]")));
gst_element_class_add_pad_template (GST_ELEMENT_CLASS(klass),
    gst_pad_template_new ("sink", GST_PAD_SINK, GST_PAD_ALWAYS,
      gst_caps_from_string (VIDEO_SINK_CAPS ", width = (int) [1, 640], height = (int) [1, 360]")));

gst_element_class_set_static_metadata (GST_ELEMENT_CLASS(klass),
    "Face detection using the Vitis-AI-Library", 
    "Video Filter", 
    "Face Detection",
    "FIXME <fixme@example.com>");

• Update the transform_frame_ip() function. The code snippet shown below that is used to draw bounding boxes is based on face detection code from the Vitis-AI-GitHub repository. In gstvaifacedetect.cpp change

FROM:

static GstFlowReturn
gst_vaifacedetect_transform_frame_ip (GstVideoFilter * filter, GstVideoFrame * frame)
{
  GstVaifacedetect *vaifacedetect = GST_VAIFACEDETECT (filter);

  GST_DEBUG_OBJECT (vaifacedetect, "transform_frame_ip");

  return GST_FLOW_OK;
}

TO:

static GstFlowReturn
gst_vaifacedetect_transform_frame_ip (GstVideoFilter * filter, GstVideoFrame * frame)
{
  GstVaifacedetect *vaifacedetect = GST_VAIFACEDETECT (filter);

  /* Create face detection object */
  thread_local auto face = vitis::ai::FaceDetect::create("densebox_640_360");

  /* Setup an OpenCV Mat with the frame data */
  cv::Mat img(360, 640, CV_8UC3, GST_VIDEO_FRAME_PLANE_DATA(frame, 0));

  /* Perform face detection */
  auto results = face->run(img);

  /* Draw bounding boxes */
  for (auto &box : results.rects)
  {
    int xmin = box.x * img.cols;
    int ymin = box.y * img.rows;
    int xmax = xmin + (box.width * img.cols);
    int ymax = ymin + (box.height * img.rows);

    xmin = std::min(std::max(xmin, 0), img.cols);
    xmax = std::min(std::max(xmax, 0), img.cols);
    ymin = std::min(std::max(ymin, 0), img.rows);
    ymax = std::min(std::max(ymax, 0), img.rows);

    cv::rectangle(img, cv::Point(xmin, ymin), cv::Point(xmax, ymax), cv::Scalar(0, 255, 0), 2, 1, 0);
  }

  GST_DEBUG_OBJECT (vaifacedetect, "transform_frame_ip");

  return GST_FLOW_OK;
}

• Update the plugin package information in gstvaifacedetect.cpp

FROM:

#ifndef VERSION
#define VERSION "0.0.FIXME"
#endif
#ifndef PACKAGE
#define PACKAGE "FIXME_package"
#endif
#ifndef PACKAGE_NAME
#define PACKAGE_NAME "FIXME_package_name"
#endif
#ifndef GST_PACKAGE_ORIGIN
#define GST_PACKAGE_ORIGIN "http://FIXME.org/"
#endif

GST_PLUGIN_DEFINE (GST_VERSION_MAJOR,
    GST_VERSION_MINOR,
    vaifacedetect,
    "FIXME plugin description",
    plugin_init, VERSION, "LGPL", PACKAGE_NAME, GST_PACKAGE_ORIGIN)

TO:

#ifndef VERSION
#define VERSION "0.0.0"
#endif
#ifndef PACKAGE
#define PACKAGE "vaifacedetect"
#endif
#ifndef PACKAGE_NAME
#define PACKAGE_NAME "GStreamer Xilinx Vitis-AI-Library"
#endif
#ifndef GST_PACKAGE_ORIGIN
#define GST_PACKAGE_ORIGIN "http://xilinx.com"
#endif

GST_PLUGIN_DEFINE (GST_VERSION_MAJOR,
    GST_VERSION_MINOR,
    vaifacedetect,
    "Face detection using the Xilinx Vitis-AI-Library",
    plugin_init, VERSION, "LGPL", PACKAGE_NAME, GST_PACKAGE_ORIGIN)

• Compile the plugin using the make file provided in the Avnet GitLab repo cloned in Part 1

cd $GST_PROJ_DIR/vaifacedetect
make -f $GST_PROJ_DIR/vai-gst-plugin-tutorial/solution/vaifacedetect/Makefile

• When the compilation completes you should find a libgstvaifacedetect.so file

Part 4: Running on the Ultra96-V2 hardware

This section describes how to run the custom Vitis-AI GStreamer plugin on the Ultra96-V2 platform from the command line.

NOTE: it is assumed that the Ultra96-V2 has been setup with the Vitis-AI v1.2 SD card image created by Mario Bergeron - link. For more details please see Mario's project page - link

• Copy the compiled ML plugin from the host machine to the target hardware using an Ethernet connection. Execute the following command on the host machine to perform the copy via scp

cd $GST_PROJ_DIR
scp vaifacedetect/libgstvaifacedetect.so root@$TARGET_IP:/usr/lib/gstreamer-1.0/.

NOTE:$TARGET_IP in the command above should be replaced with the IP address of the target hardware

• Execute the following command on the Ultra96-V2 to download the dpu_clk python script from Xilinx GitHub. The dpu_clk script changes the DPU clock frequency in order to fix a potential power issue when running the Ultra96-V2 DPU at 300 MHz

wget https://raw.githubusercontent.com/Xilinx/Vitis_Embedded_Platform_Source/2019.2/Xilinx_Official_Platforms/zcu104_dpu/petalinux/project-spec/meta-user/recipes-ai/dpuclk/files/dpu_clk -O ~/dpu_clk.py

• Testing with the GStreamer face detection plugin has been known to work when setting the DPU clock to 50% with the following command

python3 ~/dpu_clk.py 50

• Execute the following commands on the Ultra96-V2 to set the display resolution to 480p

export DISPLAY=:0.0
xrandr --output DP-1 --mode 640x480

• Launch the GStreamer pipeline from the Ultra96-V2 terminal to perform face detection on a USB webcam (enumerated as /dev/video0)

gst-launch-1.0 -v \
  v4l2src device=/dev/video0 ! \
  video/x-raw, width=640, height=360, format=YUY2, framerate=30/1 ! \
  queue ! \
  videoconvert ! \
  video/x-raw, format=BGR ! \
  queue ! \
  vaifacedetect ! \
  queue ! \
  videoconvert ! \
  fpsdisplaysink sync=false text-overlay=false fullscreen-overlay=true

• If everything is successful you will see the the GStreamer pipeline roll as well as the frame rate output to the terminal. The Ultra96-V2 should easily achieve 30 FPS.