Object Detection with KR260 from a Webcam

We tried object detection with a regular USB-connected webcam using the KR260.

By using the FPGA’s DPU for YOLOX inference, we can achieve fast, real-time detection.

Subproject for the AMD Pervasive AI Developer Contest

This project is part of a subproject for the AMD Pervasive AI Developer Contest.

Be sure to check out the other projects as well.

**The main project is currently under submission. ***

0. Main project << under submission

1. PYNQ + GPIO(LED Blinking)

2. PYNQ + PWM(DC-Motor Control)

3. Object Detection(Yolo) with DPU-PYNQ

4. Implementation DPU, GPIO, and PWM

5. Remote Control 360° Camera

6. GStreamer + OpenCV with 360°Camera

7. 360 Live Streaming + Object Detect(DPU)

8. ROS2 3D Marker from 360 Live Streaming

9. Control 360° Object Detection Robot Car

10. Improve Object Detection Speed with YOLOX

11. Benchmark Architectures of the DPU

12. Power Consumption of 360° Object Detection Robot Car

13. Application to Vitis AI ONNX Runtime Engine (VOE)

14. Appendix: Object Detection Using YOLOX with a Webcam << this project

Please note that before running the above subprojects, the following setup, which is the reference for this AMDcontest, is required.

https://github.com/amd/Kria-RoboticsAI

Introduction

Many people may not have the 360° camera required for Main project.

Therefore, as a reference, we will introduce a subproject using a generic webcam, as well as controlling DPU, GPIO and Output ROS2.

We tried object detection with a regular USB-connected webcam using the KR260.

By using the FPGA’s DPU for YOLOX inference, we can achieve fast, real-time detection.

We will introduce the program along with the test results.

Object Detection with DPU and YOLOX from a USB Webcam

I tried object detection with a regular USB-connected webcam using the KR260.

By using the FPGA’s DPU for YOLOX inference, we can achieve fast, real-time detection.

Here is the test video:

As for the normal speed of the webcam, 640x480 at 25 fps can be confirmed.

Even with YOLOX, a real-time speed of 17 fps was achieved.

(The GPIO is also operated simultaneously with object detection.)

Because we use the DPU instead of the CPU, we can perform high-speed object detection.

We will introduce the program along with the test results.

Installing GStreamer, libuvc, and v4l2loopback-dkms

Install libraries to enable live streaming from a USB camera.

GStreamer, libuvc, and v4l2loopback-dkms are required.

The installation method is introduced in the following subproject:

6. GStreamer + OpenCV with 360°Camera

We also provide an environment to use GStreamer with the PYNQ environment.

(This is because we drive a check LED from GPIO using PYNQ.)

This is introduced in the following subproject:

7. 360 Live Streaming + Object Detect(DPU)

Webcam in Use: logicool C270n

The webcam used is the "logicool C270n."

It is a very affordable webcam, but (according to specifications) it has a speed of 640x480 at 30 fps. It is sufficient for testing purposes.

When you connect the webcam to the KR260, you can verify the device.

ubuntu@kria:~$ sudo dmesg | grep usb
[    8.850427] input: UVC Camera (046d:0825) as /devices/platform/axi/ff9d0000.usb/fe200000.usb/xhci-hcd.1.auto/usb1/1-1/1-1.2/1-1.2:1.0/input/input1

Using v4l2-ctl, you can verify that it is recognized as video0.

root@kria:/home/ubuntu# v4l2-ctl --list-device
UVC Camera (046d:0825) (usb-xhci-hcd.1.auto-1.2):
        /dev/video0
        /dev/video1
        /dev/media0

You can also check which formats the camera supports.

Here is the actual information from the webcam:

root@kria:/home/ubuntu# v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'YUYV' (YUYV 4:2:2)
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                        Interval: Discrete 0.040s (25.000 fps)
                        Interval: Discrete 0.050s (20.000 fps)
                        Interval: Discrete 0.067s (15.000 fps)
                        Interval: Discrete 0.100s (10.000 fps)
                        Interval: Discrete 0.200s (5.000 fps)

Test Program for GStreamer with a Webcam

The program is saved in the following GitHub repository:

https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/tree/main/src/usb-camera

The live streaming is done via USB with the following pipeline:

It is set to 640x480 at 30 fps.

# Definition of the GStreamer pipeline (software)
pipeline = "v4l2src device=/dev/video0 ! video/x-raw, width=640, height=480, framerate=30/1 ! videoconvert ! appsink"

From here, it is the same as the demo of the test video.

First, test the program that only does live streaming (usb-camera-test.py).

Live streaming on the KR260 was successful.

sudo su
cd /home/ubuntu/AMD-Pervasive-AI-Developer-Contest/src/usb-camera/
source /etc/profile.d/pynq_venv.sh
python3 usb-camera-test.py

When counting on the program, it was actually about 25 fps.

Live Streaming Object Detection with YOLOX and DPU

Perform object detection using live streaming from the webcam.

We use the KR260’s DPU and YOLOX (app_gst-yolox-real-normal-camera-gpio.py).

This time, when a yellow ball (sports ball) is detected, the LED (GPIO) is turned on.

python3 app_gst-yolox-real-normal-camera-gpio.py

The GPIO is controlled from the PL (FPGA) IP.

The DPU and FPGA files (dpu.bit, dpu.hwh, dpu.xclbin) are also saved in the same repository.

https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/tree/main/src/usb-camera

For details on the integration method, including DPU and GPIO, please refer to the following subproject:

4. Implementation DPU, GPIO, and PWM

When the program starts, you can see the webcam image with added object detection.

In the actual live streaming with YOLOX, about 17 fps is achieved.

Since it would be only a few fps with a CPU, you can see that the DPU is properly accelerating it.

A comparison between CPU and DPU is introduced in the following subproject:

13. Application to Vitis AI ONNX Runtime Engine (VOE)

When a yellow ball (sports ball) is detected, you can see that the LED (GPIO) is turned on.

This is operated from GPIO via the PMOD connector.

Testing Marker and Image Output with ROS2

We will test outputting Marker and Image data to ROS2 using data from a webcam.

Please refer to the following subproject for installing the libraries required for ROS2.

8. ROS2 3D Marker from 360 Live Streaming

To visualize ROS2, start rviz2.

sudo su
source /opt/ros/humble/setup.bash 
rviz2

Once rviz2 is set up, start the program (gst-yolox-ros2-normal-camera.py).

sudo su
source /etc/profile.d/pynq_venv.sh
source /opt/ros/humble/setup.bash
cd /home/ubuntu/AMD-Pervasive-AI-Developer-Contest/src/usb-camera/
python3 gst-yolox-ros2-normal-camera.py

The test video is as follows:

Using the information detected by YOLOX from the webcam, the output to ROS2 is executed at about 17 fps.

Markers and images are published to ROS2 without any issues.

Summary

Many people may not have the 360° camera required for Main project.

Therefore, as a reference, we will introduce a subproject using a generic webcam, as well as controlling DPU, GPIO and Output ROS2.

In the next project, we will introduce main project for the AMD Pervasive AI Developer Contest.

0. Main project << under submission