Published February 21, 2022 © Apache-2.0

AI face follow focus assistant by kv260 in ubuntu20.04

Use an RGBD camera. Use AI to find faces in RGB footage. Using the coordinates of the face, find the distance in the depth map. Use motors a

IntermediateWork in progress20 hours747

AI face follow focus assistant by kv260 in ubuntu20.04

Things used in this project

Hardware components

AMD Kria™ KV260 Vision AI Starter Kit

Software apps and online services

Snappy Ubuntu Core

Intel RealSense SDK

AMD PYNQ Framework

Story

I have a micro 3/4 camera. It can change lenses, but the autofocus performance is not very strong. Sometimes I just use manual lenses. When I want to take a selfie. I always have trouble getting my face in perfect focus.

Last year, DJI released a cinema camera that uses lidar-assisted autofocus. In fact, as long as some motor equipment is added, it can also assist manual lenses to achieve autofocus. It works by using RGB-D cameras that are very close together. Use the image of the RGB camera to obtain the coordinates of the face. In the depth camera, look at the depth information near the coordinates of the face. This gave me a lot of inspiration.

When I got a chance to get a realsense sensor, I thought I could implement an autofocus follow focus assistant myself.

To realize this project. We need the following components:

a camera (it can output hdmi video stream),

an HDMI2USB converter (it can wrap HDMI video stream into a usb camera),

a platform with artificial intelligence computing power (here I use KV-260),

a manual lens, a servo motor (I used a motor with a FOC controller. This makes the control easy and the rotation is smooth),

some 3d printing kits to hold the motor and lens.

3D printed

Here's the thing: the KV-260 platform is the heart of the whole system. It needs to get the picture of the movie machine. Here an hdmi2usb converter is used to package the hdmi signal into a usb camera signal. KV-260 gets the picture of the movie machine just like getting a usb camera. Algorithms for performing face detection using DPU units. Get the coordinate value of the face in the current screen. Then KV-260 also needs to read the depth information of realsense. Find the depth value corresponding to the face from it. Convert the depth value to the angle that the motor needs to rotate, and use the serial port to send the motor rotation command. The motor drives a series of gears to turn the focus ring of the manual lens into the proper position.

To make this easy. We use KV-260 to install ubuntu image. Refer the link below:

https://www.hackster.io/whitney-knitter/ubuntu-20-04-3-desktop-on-kria-kv260-with-edge-impulse-f32139

To install PYNQ in Ubuntu, refer to the link below:

https://github.com/Xilinx/Kria-PYNQ

To install librealsense in Ubuntu, refer to the link below:

https://www.hackster.io/jack-bonnell2/realsense-support-for-kria-kv260-petalinux-2021-1-f5dbe3

LibUVC-backend installation

If kernel patching is not possible, or any steps during official installation.md fail, you should try libuvc-backend version of librealsense This method is not officially validated but can run on wider range of platforms including older / newer kernel version

If kernel patching is not possible, or any steps during official installation.md fail, you should try libuvc-backend version of librealsense This method is not officially validated but can run on wider range of platforms including older / newer kernel version

This script requires internet connection. Please make sure network proxy settings are correct

This script requires internet connection. Please make sure network proxy settings are correct

Make sure no RealSense device is connected
Open the terminal, run:

wget https://github.com/IntelRealSense/librealsense/raw/master/scripts/libuvc_installation.sh
sed -i 's/-DCMAKE_BUILD_TYPE=release/-DCMAKE_BUILD_TYPE=release -DBUILD_PYTHON_BINDINGS:bool=true/g' libuvc_installation.sh
chmod +x ./libuvc_installation.sh
./libuvc_installation.sh

Wait until Librealsense script completed message is displayed (may take some time)
Connect RealSense device
Run rs-enumerate-devices from the terminal to verify the installation

In order to use python api. When configuring cmake, you also need to add -DBUILD_PYTHON_BINDINGS:bool=true.

After compiling and installing, you need to copy the so file in the build directory to the same directory as the python file. while running the python example. Otherwise you will encounter errors like

ubuntu@kria:~/Documents$ python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyrealsense2 as rs
>>> pipeline = rs.pipeline()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'pyrealsense2' has no attribute 'pipeline'
>>>
ubuntu@kria:~/Documents/github/try_run_realsense$ ls
pybackend2.cpython-38-aarch64-linux-gnu.so
pybackend2.cpython-38-aarch64-linux-gnu.so.2
pybackend2.cpython-38-aarch64-linux-gnu.so.2.50.0
pyrealsense2.cpython-38-aarch64-linux-gnu.so
pyrealsense2.cpython-38-aarch64-linux-gnu.so.2.50
pyrealsense2.cpython-38-aarch64-linux-gnu.so.2.50.0
pyrealsense2Config.cmake
pyrealsense2ConfigVersion.cmake
python-tutorial-1-depth.py
ubuntu@kria:~/Documents/github/try_run_realsense$ python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyrealsense2 as rs
>>> pipeline = rs.pipeline()
>>>

Install a machine learning accelerator overlay on the KV260

1 Install the xlnx-config snap that is required for system management:

sudo snap install xlnx-config --classic

2 Run the xlnx-config sysinit command to install a custom Xilinx version of Gstreamer – accept all defaults:

xlnx-config.sysinit

3 install the NLP-SmartVision snap:

sudo snap install xlnx-vai-lib-samples

4 install the NLP-SmartVision snap.

sudo xlnx-config --snap --install xlnx-nlp-smartvision

5 Load the NLP-SmartVision overlay:

sudo xlnx-config --xmutil unloadapp
sudo xlnx-config --xmutil loadapp nlp-smartvision

6 link to the loaded dpu.xclbin

sudo ln -sf /var/snap/xlnx-config/current/assets/dpu.xclbin /usr/lib/dpu.xclbin

7 verify the DPU accelerator information

$ dexplorer -w
[DPU IP Spec]
IP  Timestamp            : 2020-11-02 15:15:00
DPU Core Count           : 1

[DPU Core Configuration List]
DPU Core                 : #0
DPU Enabled              : Yes
DPU Arch                 : B3136
DPU Target Version       : v1.4.1
DPU Freqency             : 300 MHz
Ram Usage                : Low
DepthwiseConv            : Enabled
DepthwiseConv+Relu6      : Enabled
Conv+Leakyrelu           : Enabled
Conv+Relu6               : Enabled
Channel Augmentation     : Enabled
Average Pool             : Enabled

8 List the video capture devices in the system（As you can see here I have three devices, mipi camera, usb camera and realsense sensor）.

$ v4l2-ctl --list-devices
isp_vcap_csi output 0 (platform:isp_vcap_csi:0):
    /dev/video8
Xilinx Video Composite Device (platform:xilinx-video):
    /dev/media3
Intel(R) RealSense(TM) 515: Int (usb-xhci-hcd.1.auto-1.3):
    /dev/video0
    /dev/video1
    /dev/video2
    /dev/video3
    /dev/video4
    /dev/video5
    /dev/video9
    /dev/video10
    /dev/media0
    /dev/media1
HD Pro Webcam C920 (usb-xhci-hcd.1.auto-1.4):
    /dev/video6
    /dev/video7
    /dev/media2

9 test application to use live video.For some reason, I can't open the video device of the RealSense sensor for the time being. So for now we use the usb camera.You can use other examples from this link:xlnx-vai-lib-samples.Here we use retinaface, which can detect the positions of key points such as faces and eyes. Because our mission requires this information.

xlnx-vai-lib-samples.test-video retinaface retinaface /dev/video6

The model will be downloaded to this path after running. We need to remember here.

home/ubuntu/snap/xlnx-vai-lib-samples/17/models/

NOTE: if you are using a remote terminal (SSH or serial port) you will need to have the Ubuntu desktop unlocked in order to see the application output. Additionally, you will need to set the XAUTHORITY and DISPLAY environment variables with the commands

export XAUTHORITY=$HOME/.Xauthority
export DISPLAY=:1.0

If you can see the image for face detection. Prove that the accelerator overlay installation is complete.

Same example for retinaface, but this time we compile from source.

Our task requires writing C code. Modifying in the official routine is the fastest.

1 Install the Necessary Packages

Before building the sample applications, install the required dependency packages:

$ sudo apt -y update 
$ sudo apt -y install libopencv-dev 
$ sudo apt -y install libgoogle-glog-dev

2 clone the Vitis AI repository

All of the source code is available on the Xilinx GitHub. After you clone the Vitis AI repository, switch to the v1.3.2 tag. Make sure you’re in your home directory before cloning.

$ git clone https://github.com/Xilinx/Vitis-AI.git 
$ cd Vitis-AI 
$ git checkout tags/v1.3.2

3 Build the Application Code

Depending on which sample application you want to build, switch to the sample application directory and build it using the provided script build.sh.

In releases of the repository prior to version 1.4, the paths for opencv4 in the build.sh script are incorrect. This will be fixed in Vitis AI 1.4.
For sample applications other than facedetect, be sure to update the build.sh script to include the necessary opencv4 include path before building. This can be done quickly with the following command:

$ cd demo/Vitis-AI-Library/samples/retinaface/
$ sed -i 's/-std=c++17/-std=c++17 -I\/usr\/include\/opencv4/g' build.sh
$ ./build.sh

4 link to the xmodel

sudo ln home/ubuntu/snap/xlnx-vai-lib-samples/17/models/retinaface/retinaface.xmodel ./retinaface.xmodel 
sudo ln home/ubuntu/snap/xlnx-vai-lib-samples/17/models/retinaface/retinaface.prototxt ./retinaface.prototxt

5 Run the Sample Application

The sample applications can take input from either a jpeg image or a USB Camera. HDMI input is not supported by the sample apps.

/test_video_retinaface  retinaface.xmodel /dev/video6

If all is well, you should see the same result as xlnx-vai-lib-samples.test-video.

Because the path of Vitis-ai has been installed and formulated in the ubuntu environment (not the path of git clone). So in fact you can copy the retinaface directory to any path to compile.

Unfortunately bouth ubuntu20.04 image and petalinux2021.1 image do not have usb serial driver. I didn't find any step for build ubuntu image.So I have to rebuild a petalinux image.

This project is on hold

LibUVC-backend installation

#!/bin/bash -xe

#Locally suppress stderr to avoid raising not relevant messages
exec 3>&2
exec 2> /dev/null
con_dev=$(ls /dev/video* | wc -l)
exec 2>&3

if [ $con_dev -ne 0 ];
then
	echo -e "\e[32m"
	read -p "Remove all RealSense cameras attached. Hit any key when ready"
	echo -e "\e[0m"
fi

lsb_release -a
echo "Kernel version $(uname -r)"
sudo apt-get update
cd ~/
sudo rm -rf ./librealsense_build
mkdir librealsense_build && cd librealsense_build

if [ $(sudo swapon --show | wc -l) -eq 0 ];
then
	echo "No swapon - setting up 1Gb swap file"
	sudo fallocate -l 2G /swapfile
	sudo chmod 600 /swapfile
	sudo mkswap /swapfile
	sudo swapon /swapfile
	sudo swapon --show
fi

echo Installing Librealsense-required dev packages
sudo apt-get install git cmake libssl-dev freeglut3-dev libusb-1.0-0-dev pkg-config libgtk-3-dev unzip -y
rm -f ./master.zip

wget https://github.com/IntelRealSense/librealsense/archive/master.zip
unzip ./master.zip -d .
cd ./librealsense-master

echo Install udev-rules
sudo cp config/99-realsense-libusb.rules /etc/udev/rules.d/ 
sudo udevadm control --reload-rules && sudo udevadm trigger 
mkdir build && cd build
cmake ../ -DFORCE_LIBUVC=true -DCMAKE_BUILD_TYPE=release  -DBUILD_PYTHON_BINDINGS:bool=true
make -j2
sudo make install
echo -e "\e[92m\n\e[1mLibrealsense script completed.\n\e[0m"