AI face follow focus assistant using kv260 in petalinux

I have a micro 3/4 camera. It can change lenses, but the autofocus performance is not very strong. Sometimes I just use manual lenses. When I want to take a selfie. I always have trouble getting my face in perfect focus.

Last year, DJI released a cinema camera that uses lidar-assisted autofocus. In fact, as long as some motor equipment is added, it can also assist manual lenses to achieve autofocus. It works by using RGB-D cameras that are very close together. Use the image of the RGB camera to obtain the coordinates of the face. In the depth camera, look at the depth information near the coordinates of the face. This gave me a lot of inspiration.

When I got a chance to get a realsense sensor, I thought I could implement an autofocus follow focus assistant myself.

To realize this project. We need the following components:

a camera (it can output hdmi video stream),

an HDMI2USB converter (it can wrap HDMI video stream into a usb camera),

a platform with artificial intelligence computing power (here I use KV-260),

a manual lens, a servo motor (I used a motor with a FOC controller. This makes the control easy and the rotation is smooth),

some 3d printing kits to hold the motor and lens.

Here's the thing: the KV-260 platform is the heart of the whole system. It needs to get the picture of the movie machine. Here an hdmi2usb converter is used to package the hdmi signal into a usb camera signal. KV-260 gets the picture of the movie machine just like getting a usb camera. Algorithms for performing face detection using DPU units. Get the coordinate value of the face in the current screen. Then KV-260 also needs to read the depth information of realsense. Find the depth value corresponding to the face from it. Convert the depth value to the angle that the motor needs to rotate, and use the serial port to send the motor rotation command. The motor drives a series of gears to turn the focus ring of the manual lens into the proper position.

To make this easy. We use KV-260 to install ubuntu image. Refer the link below:

https://www.hackster.io/whitney-knitter/ubuntu-20-04-3-desktop-on-kria-kv260-with-edge-impulse-f32139

To install PYNQ in Ubuntu, refer to the link below:

https://github.com/Xilinx/Kria-PYNQ

To install librealsense in Ubuntu, refer to the link below:

https://www.hackster.io/jack-bonnell2/realsense-support-for-kria-kv260-petalinux-2021-1-f5dbe3

Unfortunately bouth ubuntu20.04 image and petalinux2021.1 image do not have usb serial driver. I didn't find any step for build ubuntu image.So I have to rebuild a petalinux image.

1 Install PetaLinux and update the eSDK

To install PetaLinux, extract the petalinux installer, accept the license and source the tool’s settings script.

petalinux-v2021.1-final-installer.run
source settings.sh

The eSDK can be used to update the petalinux tool for creating new images or SDKs. The eSDK updates are published at http://petalinux.xilinx.com/sswreleases/rel-v2021/sdkupdate/2021.1_update1/ Upgrade the tool with new eSDK for ‘2021.1 update1’ release and source the tool’s settings script.

petalinux-upgrade -u 'http://petalinux.xilinx.com/sswreleases/rel-v2021/sdkupdate/2021.1_update1/' -p 'aarch64'
source settings.sh

2.1 Create the Project

Run the following command to create a new Petalinux project from the provided K26 SOM bsp file:

petalinux-create -t project -s xilinx-k26-starterkit-v2021.1-final.bsp  -n kv260_RealSense
cd kv260_RealSense

2.2 Configure Project for KV260 Baseboard

The Kria K26 BSP configures the PetaLinux project for the Kria K26 SoM, but does not include configuration for the KV260 baseboard in the initial configuration. Which lends itself to the whole concept of a SoM where the FPGA is isolated to a smaller PCB that is interchangeable between baseboards that contain the peripheral hardware. To configure the PetaLinux project to include the packages in the root filesystem for the peripherals of the KV260 board in the starter kit, set the BOARD_VARIANT variable to kv in the petalinuxbsp.conf file.

echo 'BOARD_VARIANT = "kv"' >>  project-spec/meta-user/conf/petalinuxbsp.conf

Then in the future as more baseboards become available for the Kria K26 SoM, it's simply a matter of changing this variable to re-target the PetaLinux project.

Open the petalinuxbsp.conf file to verify BOARD_VARIANT has been set.

2.3 Add Xilinx Accelerated Applications

Next, add the package groups for the accelerated applications to the root filesystem configuration file (user-rootfsconfig) so that they pop up as options to add to the next build in the root filesystem configuration editor:

echo 'CONFIG_packagegroup-kv260-smartcam' >> project-spec/meta-user/conf/user-rootfsconfig
echo 'CONFIG_packagegroup-kv260-aibox-reid' >> project-spec/meta-user/conf/user-rootfsconfig
echo 'CONFIG_packagegroup-kv260-defect-detect' >> project-spec/meta-user/conf/user-rootfsconfig
echo 'CONFIG_packagegroup-kv260-nlp-smartvision' >> project-spec/meta-user/conf/user-rootfsconfig

Then launch the root filesystem configuration editor, where the accelerated applications will appear under the user packages menu:

Once the console has booted up navigate to:

Petalinux Package Groups-->packagegroup-petalinux-self-hosted

2.4 Add Vitis Meta Layer to Project

$ petalinux-config --> yocto settings --> user layer

Add Path to "meta-vitis-ai"

Add Vitis Meta Layer to user packages

2.5 Add Vitis Meta Layer to user packages

$ echo "CONFIG_packagegroup-petalinux-vitisai" >> project-spec/meta-user/conf/user-rootfsconfig

$ echo "CONFIG_packagegroup-petalinux-vitisai" >> project-spec/meta-user/conf/user-rootfsconfig

2.6 Enable package in rootfs

$ petalinux-config -c rootfs

$ petalinux-config -c rootfs

user packages --> packagegroup-petalinux-vitisai

2.7 Add usb serial driver

petalinux-config -c kernel

Device Drivers --->USB support --->USB Serial Converter support --->USB CP210x family of UART Bridge Controllers

Device Drivers --->USB support --->USB Serial Converter support --->USB Winchiphead CH341 Single Port Serial Driver

2.8 Fast build

To reduce compilation time, You can download the file sstate_aarch64 and download first.

then link to the right path like below.

mkdir build
cd build/
sudo ln -s ../../xilinx-k26-starterkit-2021.1/build/downloads/ downloads
sudo ln -s ../../xilinx-k26-starterkit-2021.1/build/sstate-cache/ sstate-cache

2.9 Build PetaLinux Project

Run the following command to run the PetaLinux build process:

$ petalinux-build

2.10 package to image

petalinux-package --wic --bootfiles "ramdisk.cpio.gz.u-boot boot.scr Image system.dtb"

you can find image file in this path :"images/linux/petalinux-sdimage.wic"

use balenaEtcher flash a tf card.

some fast scrip :

petalinux-create -t project -s ../download/xilinx-k26-starterkit-v2021.1-final.bsp -n kv260_RealSense_self4
cd kv260_RealSense_self4/
mkdir build
cd build/
sudo ln -s ../../xilinx-k26-starterkit-2021.1/build/downloads/ downloads
sudo ln -s ../../xilinx-k26-starterkit-2021.1/build/sstate-cache/ sstate-cache
ll
cd ../
echo 'BOARD_VARIANT = "kv"' >>  project-spec/meta-user/conf/petalinuxbsp.conf
echo 'CONFIG_packagegroup-kv260-smartcam' >> project-spec/meta-user/conf/user-rootfsconfig
echo 'CONFIG_packagegroup-kv260-aibox-reid' >> project-spec/meta-user/conf/user-rootfsconfig
echo 'CONFIG_packagegroup-kv260-defect-detect' >> project-spec/meta-user/conf/user-rootfsconfig
echo 'CONFIG_packagegroup-kv260-nlp-smartvision' >> project-spec/meta-user/conf/user-rootfsconfig
echo "CONFIG_packagegroup-petalinux-vitisai" >> project-spec/meta-user/conf/user-rootfsconfig
echo "CONFIG-intel-realsense" >> project-spec/meta-user/conf/user-rootfsconfig
petalinux-config -c rootfs
petalinux-config -c kernel 
petalinux-build
petalinux-package --wic --bootfiles "ramdisk.cpio.gz.u-boot boot.scr Image system.dtb"
petalinux-package --wic --bootfiles "ramdisk.cpio.gz.u-boot boot.scr Image system.dtb"

3 Build development environment

3.1 GCC/G++ installation

We need to install packagegroup-petalinux-self-hosted to be able to compile on target. To install run the below command to install it and anything need else.

sudo dnf install packagegroup-petalinux-self-hosted libx11-dev libxcursor-dev packagegroup-petalinux-x11-dev  libusb-1.0-dev usbutils-dev libusb-1.0-dev -y

3.2 build realsense sdk

This process requires a lot of storage space.Make sure you have more than 16G spaces.

wget https://github.com/IntelRealSense/librealsense/archive/master.zip
unzip ./master.zip
cd librealsense-master/

echo Install udev-rules
sudo cp config/99-realsense-libusb.rules /etc/udev/rules.d/ 
sudo udevadm control --reload-rules && sudo udevadm trigger 
mkdir build && cd build

cmake ../ -DFORCE_LIBUVC=true -DCMAKE_BUILD_TYPE=release -DBUILD_PYTHON_BINDINGS:bool=true -DBUILD_GRAPHICAL_EXAMPLES=false
make -j4
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib64/"
sudo make install

use this to test :

rs-depth

3.3 Clone Vitis-ai

git clone https://github.com/Xilinx/Vitis-AI.git
cd Vitis-AI/

3.4 find module link and download

grep kv260- models/ -rh
wget https://www.xilinx.com/bin/public/openDownload?filename=retinaface-zcu102_zcu104_kv260-r2.0.0.tar.gz -O retinaface-zcu102_zcu104_kv260-r2.0.0.tar.gz
tar -xzvf retinaface-zcu102_zcu104_kv260-r2.0.0.tar.gz
sudo mkdir /usr/share/vitis_ai_library/models/
sudo cp -r retinaface /usr/share/vitis_ai_library/models/

3.5 build retinaface demo

cd demo/Vitis-AI-Library/samples/retinaface
chmod +x ./build.sh
./build

3.6 test retinaface demo

first make sure use which usb camera.

v4l2-ctl --list-devices
Intel(R) RealSense(TM) 515: Int (usb-xhci-hcd.1.auto-1.3):
        /dev/video8
        /dev/video9
        /dev/media2

HD Pro Webcam C920 (usb-xhci-hcd.1.auto-1.4):
        /dev/video0
        /dev/video1
        /dev/media0

install and load dpu by:

sudo xmutil getpkgs
sudo dnf install packagegroup-kv260-nlp-smartvision.noarch
sudo xmutil unloadapp
sudo xmutil loadapp kv260-nlp-smartvision

then run:

sudo chmod o=rw /dev/ttyUSB0
./test_video_retinaface retinaface.xmodel /dev/video0

if you find some thing like can not open display. try to use ssh by:

ssh -X petalinux@xxx.xxx.xxx.xxx
sudo dnf install packagegroup-petalinux-x11

4 build face follow prj

4.1 clone prj

git clone retinaface_follow_focus
cd retinaface_follow_focus/

4.2 install depends and build

sudo dnf install libglog-dev opencv-dev vitis-ai-library-dev -y
./build

4.3 run demo

 sudo chmod o=rw /dev/ttyUSB0
./test_video_retinaface retinaface.xmodel /dev/video0

5 demo show