Hardware components | ||||||
 |
| × | 1 | |||
| × | 1 | ||||
| × | 1 | ||||
| × | 1 | ||||
| × | 1 | ||||
| × | 1 | ||||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
 |
| |||||
 |
| |||||
| ||||||
Hand tools and fabrication machines | ||||||
 |
| |||||
| ||||||
A journey through machine learning and Xilinx Kria KV260 Vision AI Starter Kit
Disclaimer: The following project was done for the Adaptive Computing Challenge 2021. Most of the code and 3D models are under The MIT License | Open Source Initiative or Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) unless specified to the contrary. A copy of this document and the corresponding project files are located at the KV260-ATRover GitHub repository. As part of the application process, I received a free Kria KV260 AI Starter Kit + Basic Accessory Pack.
⚠ This project uses high DC amperage which can be ☠ - please use extreme caution.
⚠ All links and references are set to match the tools version used in this project, which most of the time are not the latest. Readers are welcome to go further, as new releases came with significant improvements.
Vitis-AI v1.4 and v1.4.1 share the same database, and I found that the information available is more complete on v1.4.1Introduction
The idea of having a self-driving vehicle platform has been going around my head for a long time now. First, it started with my personal experience with some robot vacuums which became a relation of love and not-that-much hate. With the passing of the years, I have enjoyed the benefits of keeping my floors clean (I suffer a lot from dust allergies), but with the passing of the years, I keep seeing the same problems - which became worst when your family grows and your kids behave like kids.
And then we moved to a rural house outside Ottawa, and cutting the grass although relaxing, is time-consuming. I started working around a full autonomous acreage lawn mower which I wanted to have an FPGA (just because), when I came across the Adaptive Computing Challenge 2021 on HacksterIO - and that was my application for free hardware. But after starting from almost zero knowledge on both Xilinx Zynq Ultrascale+ and AI/ML (Artificial Intelligence/Machine Learning) I run out of time, plus I live in Canada, so a lawnmower was out of the question during winter.
So I decided to step back, and start with a basic ATRover (AuTonomous Rover) and the KRIA KV260-ATRover mini version was born - the prototype of an autonomous self-driving mini-rover based on the Kria KV260 Vision AI Starter Kit (Zynq Ultrascale+)
My objectives when joining the challenge were:
- Learn about the Xilinx Zynq Ultrascale+ devices
- Learn about the Xilinx Zynq Ultrascale+ devices
- Go into AI (Artificial Intelligence) and ML (Machine Learning)
KV260-ATROVER (mini)
The KV260-ATROVER (mini) has the following features:
- Full 3D Printed chassis with OpenSCAD
- KRIA KV260 board (master) with AI Object Identification with YOLOv4 and OpenCV Stereo Vision for distance estimation running Ubuntu with PYNQ
- TTGO-T1 board (slave) for Dual DC motor control
Motivation
Stereo Vision, DC motor control, route planning, and navigation require a lot of calibration. In the long term, my final goal is to explore and implement a fully autonomous robot with self-learning capabilities, where the required effort for initial calibration is minimized.
As an example, you can watch OpenCV Python videos from Clayton Darwin where he does a good explanation of the required process to do triangulation using two cameras and OpenCV. An that is just triangulation, PID motor control, route planning, and context awareness by themself are quite complex, doable but limited, and space-constrained - and this is why and where AI/ML (CNN) keeps growing in many fields.
HardwareThe ATRover chassis is a custom-made 3D printed model. The design was done in OpenSCAD. The chassis is modular, so it is possible to reuse for example the KV260 enclosure for future projects. The weight of the assembled ATRover is around 6Kg.
Kria KV260 Vision AI Starter Kit
The hardware selected for this project is the Xilinx KRIA KV260 board, which was awarded as free hardware in this challenge. Another good candidate is the Avnet - Ultra96-V2 Board which has similar characteristics. One advantage of the KRIA Zynq (XCK26) vs the ZU3EG in the Ultra96-V2 is the VCU (Video Codec Unit) which makes it an ideal platform for R&D video applications (live stream)
The KRIA SOM installed in the KV260 Vision AI Starter Kit is a special (non-commercial) SOM, with similar characteristics to the Kria K26C SOM. The KV260 has only a single 240-Pin connector and is not certified.
Stereo Camera
For this project, the HBVCAM-1780-2 binocular camera was selected as it already outputs a dual stereo image synchronized in a single USB port (stream), and it has a decent price.
The camera was tested with https://webcamtests.com/ to validate the manufacturer parameters:
- Max. resolution: 1.84 MP, 30 FPS at 2560×720 (dual image, 2x 1280x720)
- Angle of view: 72 degrees (3.6mm lenses)
- Distance between lenses: 60mm
- Object distance: 30cm ~ ∞ (fixed focus - manual adjustable)
- Dual Image Resolution:
640*240 (30fps), 1280*480 (30fps), 2560*720 (25fps)(validated) - Single camera:
320*240 (30fps), 640*480(30fps), 1280*720 (30fps) - Connection: micro USB2.0
- 2xOV9732 Sensors with Optical format: 1/4" and Array size: 1280*720
As the camera lenses produce some distortion on the captured images, it is necessary to perform a camera calibration as described in Stereo Vision Camera Calibration in Python with OpenCV. A modified set of Python scripts used for this project can be found in the GitHub repository. The parameters obtained are particular for the module used in this project, and it must be done on a per-module basis.
The scripts are based on niconielsen32-ComputerVision/stereoVisionCalibration modified to use one single stereo camera stream plus some personal modifications.
DC Motors
For the KV260-ATRover the CHIHAI GM37-550 DC motor with a gear ratio of 50:1 and 12Vdc with a maximum input power of 40W was selected.
The current DC motors are too powerful for the KV260-ATRover mini. They were selected initially for the lawnmower which has a bigger heavier chassis (higher torque requirement) and larger wheels (reducing the speed of the ATRover and therefore increasing the minimum required PWM duty cycle)
DC Motor Driver(s)
For the motor drivers, the ZK-5AD board was selected. It has two TA6586 monolithic IC for driving bi-directional DC motors.
- Working Voltage: DC 3.0V-14V
- Input Signal Voltage: DC 2.2V-6.0V
- Drive Current: 5A
- Standby Current: 10uA
- Working Temperature:-20 to 85 Celsius
For the DC motor control, the plan is to use the Zynq+ Dual-Core ARM Cortex-R5 processors with FreeRTOS. The initial test was done with a TTGO T-Display (aka TTGO-T1) board. Although I was planning to use the PMOD to generate the required PWM signals, after burning two drivers and one TTGO-T1 board I decided to leave it for another day.
You can buy a TTGO-T1 board at Aliexpress (the Official LILYGO store). It is based on the ESP32 Espresiff (Wikipedia), a 32bit MCU Tensilica Xtensa LX6 with integrated WiFi, Bluetooth, and a lot of peripherals. This is the board I usually use for R&D.
USB WiFi dongle
To do some telemetry and get some real-time feedback when testing the system, wireless communication with good throughput and low latency was required. As the KV260 lacks a dedicated wireless communication interface, a USB WiFi dongle was selected for this. For more detailed information on how to install the drivers and set up the device used in this project please refer to Adding USB-WiFi for the Kria KV260 (Ubuntu 20.04.3).
Power considerations
As the ATRover is a mobile platform, an appropriate power supply was required. SLA batteries are widely used in robotic applications. A SLA12V/12Ah battery was selected given the following rough estimates:
- KV-260 power requirements: 36W (4Ah)
- DC Motors: 2x21W (42W, 3.5Ah)
Roughly at full power, the ATRover would require 80W or 6.7Ah. Note that the 12Ah capacity is rated for a 20Hr operation. A constant 3~4A discharge would reduce the capacity to ~9Ah, and give you between 2~3hr. The motors are running at half the capacity or a lower speed.
During the tests, I got ~45min of working power.
Other components
To complete assemble an ATRover mini, some other components are required:
- 3D printed chassis
- M3 screws
- Power distribution block
- Power cables (AWG16) (KRIA power cable, DC motors, TTGO-T1 ZK-5AD)
- micro-USB cable for Stereo Camer
- USB-C cable for TTGO-T1
Please refer to the BOM at the repository for more information.
The journey through the KRIA KV-260 and Xilinx Development Tools was extensive and extenuating. Although I have learned a lot, in the end, with the timeline nearer every day, I have to make a choice and start doing some real deployment.
Given that there was no easy way to install the USB/WiFi drivers or any other driver (like the TTGO-T1 USB-Serial) in the Petalinux flow, I decided to work with Ubuntu. PYNQ for the KV260 was made available two months ago. Given that scripting is clearer with Python, I decided to switch to PYNQ. Initially, it worked right away without any issues. Running the examples and tutorials is more comfortable and easier to grasp.
Required tools
- Kria-PYNQ
- Vitis-AI v1.4.1: the DPU-PYNQ uses version 1.4, but using v1.4.1
- Visual Code + PlatformIO: the KV260 ATRover uses a TTGO-T1 board to control the DC motors via USB/Serial port. Please follow the instructions on each website to install the tools.
Kria SOMs x Ubuntu x PYNQ: nice introductory webinar
The PYNQ framework on the KRIA KV-260 is pretty simple to use. It runs on Ubuntu 20.04.04 LTS where it is easier to install new drivers or tools for development (e.g. USB WiFi dongle, USB/UART drivers)
- Easier to install new drivers or tools for development (e.g. USB WiFi dongle, USB/UART drivers)
- Visual Code SSH + PlatformIO Remote Development
- It runs on Ubuntu 20.04.04 LTS, so it was easier to install new drivers or tools for development (e.g. USB WiFi dongle, USB/UART drivers)
- Faster to test and deploy new ideas with Python + Jupyter-lab
- DPU ready overlay (B4096)
- Easy to install and error prune (which unfortunately proved me wrong as at the end the dependencies on Ubuntu/PYNQ were broken)
- Remote access with remote graphics applications (
ssh -X)
PYNQ install
At the moment, PYNQ - KRIA installation is "broken" Unable to install PYNQ #9 It is possible to get PYNQ with Xilinx Dev. working by skipping the initial:sudo apt update; sudo apt upgradeand do it only at the end!Currently Canonical and PYNQ are working on a new release that will address this issue.
To install, just follow the instructions on the official GitHub repository https://github.com/Xilinx/Kria-PYNQ
Note: As soon as you power up the KRIA KV260, after login - the OS will start an update status query, which will lock some process causing the next steps to lock for a couple of minutes.
Some info related to this "issue" can be read here https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2037317633/Getting+Started+with+Certified+Ubuntu+20.04+LTS+for+Xilinx+Devices
Vitis-AI (v1.4.1)
Vitis-AI requires around 100GB of hard drive space
As the current Vitis-AI version is 2.0, but DPU-PYNQ requires v1.4.1, please follow the next steps to install the proper tool version. Vitis-AI is installed and run in your local machine (Ubuntu)
Installation steps
# Prerequisites
sudo apt install -y ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# Docker engine
sudo apt install -y docker-ce docker-ce-cli containerd.io
# Setup group access
sudo groupadd docker
sudo usermod -aG docker $USER
su $USER
# Test installation
sudo docker run hello-world- Install Vitis-AI
mkdir -p ~/repos
cd repos
git clone --recurse-submodules https://github.com/Xilinx/Vitis-AI
cd Vitis-AI
git checkout 1.4- Get Docker image:
docker pull xilinx/vitis-ai-cpu:1.4.916- Launch Vitis-AI
./docker_run.sh xilinx/vitis-ai-cpu:1.4.916Additional Drivers & Ubuntu configuration
Ubuntu packages
sudo apt install -y dkms net-tools build-essentialUSB WiFi
mkdir repos; cd repos
git clone https://github.com/morrownr/88x2bu-20210702
cd 88x2bu-20210702
sudo ARCH=arm64 ./install-driver.shDo you want to edit the driver options file now? [y/N] n Do you want to reboot now? (recommended) [y/N] n
# TTGO-T1 UART/USB drivers
cd ~/repos
git clone https://github.com/WCHSoftGroup/ch343ser_linux.git
cd ch343ser_linux/driver
make ARCH=arm64
sudo make install
sudo rebootWiFi configuration
The easiest way to configure the WiFi is through GNOME. After login, click on the down icon near the power button ⏻▼ and select your network. This connection will be preserved after a reboot or shutdown without the need to log in to establish a WiFi connection.
Alternatively, you can set it on the command line. Currently, I do not know how to make a connection permanently (aka WiFi connection without login), so the GUI method is recommended.
Final steps
sudo apt update -y; sudo apt upgrade -y; sudo apt autoremove -y; sudo reboot💡In the KV260-ATRover GitHub repository I have provided a set of.bash configuration scripts that include some extra setup for Ubuntu command line terminals. It will give you a nicer prompt in the command line.
cd ~
mkdir dev
cd dev
git clone https://github.com/dramoz/kv260-atroverNOTE: you can use Visual Code remotely with SSH.
Click ≶ icon on the bottom left corner and create a new connection.
Create a symbolic link to the repository so it is accessible from Jupiter-Lab
cd $PYNQ_JUPYTER_NOTEBOOKS
ln -s $HOME/dev/kv260-atrover/ kv260-atroverCopy the .bashrc configuration (optional)
cp -fv ~/dev/kv260-atrover/scripts/.bash* ~/
source ~/.bashrcFinally, install the required python dependencies:
pip install -r ~/dev/kv260-atrover/scripts/requirements.txtTTGO-T1 firmware
The TTGO-T1 code is a simple four-channel PWM generator connected to the ZK-5AD motor driver. It communicates to the KV260 board with a USB/Serial port. The KV260 application sends the following commands as required:
*m###, m in [f, b, r, l, s], with f: forward, b: backward, r: turn right, l: turn left, s: stop and ### the final step value, between min/max_step range (number string must be 3 numeric characters), example: *f064
!xxxx: emergency stop (any four xxxx characters)
The following parameters can be configured (required to flash new firmware into the board):
min/max speed: in steps to control the PWM duty cyclePWM frequencyandPWM resolutionacceleration: by means ofspeed_stepand update rate (accel_step_ms)
Initial parameters are set on a trial an error basis as
The firmware of the TTGO-T1 board was developed using Visual Code PlatformIO. After installing Visual Code + PlatformIO, clone the project repository on a local machine and add the ttgo-atrover-motor-control project to PlatformIO. Connect the board to an available USB serial port, compile (✓) and flash (➔) the board by clicking on the icons located at the bottom toolbar in VS code.
💡It is possible to program the TTGO-T1 board directly from the KRIA KV-260 with PlatformIO Remote Development tools.
A free PlatformIO account is required for remote development.
On your local machine
- Open Visual Studio Code
# CTRL+SHIFT+P
# Start typing Platf...
# -> Click PlatformIO: New Terminal
pio account register
pio account loginOn the KV260
- Install
python3 -c "$(curl -fsSL https://raw.githubusercontent.com/platformio/platformio/master/scripts/get-platformio.py)"- Update
.bashrc(already present if.bashrcwas updated from the repository)
export PATH=$PATH:$HOME/.platformio/penv/bin- Install the platformio-udev-rules
curl -fsSL https://raw.githubusercontent.com/platformio/platformio-core/master/scripts/99-platformio-udev.rules | sudo tee /etc/udev/rules.d/99-platformio-udev.rules
sudo service udev restart
sudo usermod -a -G dialout $USER
sudo usermod -a -G plugdev $USER- Login and start the remote agent
pio account login
pio remote agent startThe first call to pio remote agent start will require some time while the required packages and dependencies are installed. Local machine remote programming
- On a PlatformIO terminal in VS code
Check that the terminal current path is at ~/dev/kv260-atrover/ttgo-atrover-motor_control/- Check KV260 remote agent
#pio account login (if required)
pio remote agent listkria ---- ID: 827c99104301d4120ea475faee3371703eddc89f Started: 2022-03-27 00:28:45
- Compile and download the new firmware to the TTGO-T1
pio remote run -t upload --upload-port /dev/ttyCH343USB0PlatformIO Remote Development is a useful remote programing tool, as if required for example to change the PWM frequency for the DC motors control, we can reprogram the TTGO-T1 without the need of unplugging it from the KV260 board.
Basic system testsAll the scripts are done in Python using Visual Studio code. Although JupyterLab was nice for the initial R&D, it is faster to run some OpenCV like imshow() outside the Jupyter environment, with ssh -X.
TTGO-T1 and DC motors
ssh ubuntu@kria_ip
screen /dev/ttyCH343USB0 115200Press the TTGO-T1 reset button, a message similar to this would appear
rst:0x1 (POWERON_RESET), boot:0x13 (SPI_FAST_FLASH_BOOT)configsip: 0, SPIWP:0xee clk_drv:0x00, q_drv:0x00, d_drv:0x00, cs0_drv:0x00, hd_drv:0x00, wp_drv:0x00mode:DIO, clock div:2load:0x3fff0018, len:4load:0x3fff001c, len:1216ho 0 tail 12 room 4load:0x40078000, len:10944load:0x40080400, len:6360entry 0x400806b4?TTGO
Run some commands: *f064, *b128, *s0000, *l128, !0000...When done, disconnect screen to free the serial port.
ctrl+a, kReally kill this window [y/n]y[screen is terminating] ubuntu@kria:~/$
Stereo camera test
To run a simple test on the stereo camera
ssh -X ubuntu@kria_ip
cd ~/dev/kv260-atrover/scripts/
python camera_test.py
The triangulation test is based on niconielsen32-ComputerVision/StereoVisionDepthEstimation/ modified to use one single stereo camera stream.
The triangulation test is loading the calibration matrices and applying the OpenCV remap on both frames. As can be seen in the pictures, the black regions surrounding the frames are a side effect of remapping the frames. It is also noticeable the difference between both camera module lenses.
ssh -X ubuntu@kria_ip
~/dev/kv260-atrover/scripts/triangulation_example/
python camera_test.py
python stereoVision.py
☝ At the moment, the triangulation demo is exiting with a Segmentation fault (core dumped) error which I did not have time to investigate further at the moment of writing this project.Movement Test
A simple script to validate the communication between the KV260 board and the TTGO while testing the motors was done.
Selecting the model(s)Xilinx provides with the Vitis-AI framework a set of pre-trained NN models that can be used as a starting point. It is referenced as Vitis-AI AI-Model-Zoo. The models are deployed on the FPGA on the DPU IP. As the DPU comes in different flavors, please note that if the model is not available for the current DPU model, extra steps are required with the Vitis-AI framework to deploy the solution properly.
- The KRIA KV260 examples run on Petalinux and use a DPU B3136, with Xilinx tools 2021.1
- The PYNQ has an overlay DPU-PYNQ with a DPU B4096, and uses Vitis-AI 1.4.1
When selecting the model for this project, one useful metric available at the AI Model Zoo worth mentioning is the Performance of Kria KV260 SOM. It summarizes the KV260 latency and throughput of the available model.
CNN's are available in different topologies, which basically define the application target. Furthermore, each model can be trained using different data sets (ImageNet,COCO,CityScapes,SYNTHIA among others)
From the different topologies available in the AI Model Zoo, the following were selected to do some R&D:
- YOLO: (You Only Look Once) object detection and classification
- SSD: (Single Shot MultiBox Detector) object detection.
- FADnet (GitHub repository): disparity estimation
Models are downloaded to the local machine where Vitis-AI was installed and later transfer with spc to the ATRover.
Models are downloaded to the local machine where Vitis-AI was installed and later transfer with spc to the ATRover.To validate that the models downloaded from the web were the proper models for the current system, the resnet50 model was also downloaded.To download the models, for each model do:
cd ~/repos/Vitis-AI/models/AI-Model-Zoo/
# -----------------------------------------------------------
# Caffe resnet50
# file: resnet50-zcu102_zcu104_kv260-r1.4.0.tar.gz
python downloader.py
> input: `cf resnet50`
2: ['type', 'xmodel']['board', 'zcu102 & zcu104 & kv260']
> input num: `2`
done
# -----------------------------------------------------------
# Darknet YOLOV4
# file: yolov4_leaky_spp_m-zcu102_zcu104_kv260-r1.4.0.tar.gz
python downloader.py
> input:dk yolov4
1 : dk_yolov4_coco_416_416_60.1G_1.4
> input num:1
2: ['type', 'xmodel']['board', 'zcu102 & zcu104 & kv260']
> input num:2
done
# -----------------------------------------------------------
# PyTorch FADnet
# file: FADNet_2_pt-zcu102_zcu104_kv260-r1.4.0.tar.gz
python downloader.py
> input:pt fadnet
4: ['type', 'xmodel']['board', 'zcu102 & zcu104 & kv260']
> input num:4
done
# -----------------------------------------------------------
# TensorFlow SSD
# file:
python downloader.py
> input:tf ssd
4 : tf_ssdresnet50v1_fpn_coco_640_640_178.4G_1.4
> input num:4
2: ['type', 'xmodel']['board', 'zcu102 & zcu104 & kv260']
> input num:2
done
# ------------------------------------------------------------------------------
# ------------------------------------------------------------------------------
# Models are downloaded as *tar.gz files
du -h *.gz
106M FADNet_2_pt-zcu102_zcu104_kv260-r1.4.0.tar.gz
18M resnet50-zcu102_zcu104_kv260-r1.4.0.tar.gz
45M yolov4_leaky_spp_m-zcu102_zcu104_kv260-r1.4.0.tar.gz
# transfer them to the KV260 SDcard with
scp *.gz ubuntu@192.168.0.198:/home/ubuntu/dev/kv260-atrover/scripts/models
# Connect to the board and untar the files
ssh -X ubuntu@kria_ip
cd ~/dev/kv260-atrover/scripts/models/
for f in ../*.gz; do tar -xvzf "$f"; doneFADNet_2_pt/FADNet_2_pt/FADNet_2_pt.xmodelFADNet_2_pt/md5sum.txtresnet50/resnet50/resnet50.prototxtresnet50/resnet50.xmodelresnet50/md5sum.txtyolov4_leaky_spp_m/yolov4_leaky_spp_m/yolov4_leaky_spp_m.prototxtyolov4_leaky_spp_m/yolov4_leaky_spp_m.xmodelyolov4_leaky_spp_m/md5sum.txt
The *.xmodel files would be the models to load into the DPU.
Each models requires a pre-process to the input frames and a post-process of the output different in each case. The information can be gather from the Vitis-AI examples, and the Board/GPU models that can be downloaded using the same previous steps (python downloads.py) and selecting the board/GPU option.1: ['type', 'float & quantized']['board', 'GPU']The downloaded files are .zip type.du -h *.zip
208M cf_resnet50_imagenet_224_224_7.7G_1.4.zip
529M dk_yolov4_coco_416_416_60.1G_1.4.zip
1.2G pt_fadnet_sceneflow_576_960_359G_1.4.zipThe pre-process information is located in the readme.md file in each case.The post-process depends on each CNN output vector format.Test scripts to run each model are located in the repository under the scripts directory.
Improving the model(s)Vitis-AI is the tool required to tailor the neural network model to project particular needs. As realized by running some tests, it would be necessary to re-train the current CNN models.
- A suitable dataset: although there are some indoor datasets, most are at higher levels.
- Gathering a dataset is time-consuming, and resources hungry.
Moving out of PYNQ
Unfortunately, during the final steps of writing this report, it was not possible to run the YOLOv4 in PYNQ and perform some final tests. A temporal solution was provided by Xilinx support. However, it was not compatible with my Python scripts or disabled the possibility of WiFi and UART/Serial interfaces. And, I ran out of time. Therefore (and sadly), it was not possible to present more videos or capture better examples of the working project.
The main application gathers the identified objects from YOLOv4 or SSD on both video channels, and by using the triangulation OpenCV algorithm it provides a rough estimate of the distance from the ATRover to the object. If a dog is identified by the ATRover, it will try to preserve the distance and follow or move back from it.
Final RemarksThis experience was a short/long journey with the KRIA KV-260. Initially, I was contemplating getting an Ultra96-V2 as it seemed better on paper for this type of project, but the free hardware option was the KV-260.
In retrospect, my gut feeling was 50/50. On one side, I think the Ultra96-V2 is "better" suited for the ATRover with its smaller form factor, ready WiFi, and more I/O ports available out of the box. On the other side, the Kria KV-260 has more USB Ports, extra camera ports, doubles the DDR-RAM and is capable of video streaming with its integrated VCU (Video Codec Unit) - features that make it ideal for R&D, which is the main idea behind the KV-260 Starter kit.
As a plus, there is no need to configure any jumpers to select the boot target, and the capability of dynamic loading overlays is key. Finally, in my case, getting Ubuntu out-of-the-shelf-ready and working without any issues is awesome.
A set of unfortunate events in the last couple of weeks played against the remaining time for the Adaptive Computing Challenge 2021, when things were finally getting interesting.
But it was worth the journey, I came from almost zero AI/ML knowledge and Vitis/Zynq+ to be able to deploy some CNN on a small vehicle platform and make it move around while identifying some objects.
The final activities to have a proper closure of the first ATRover phase that is not done yet given the timing left before submitting the project are:
- BOM: improve the bill of materials with a guide on how to assemble the whole project
- Chassis 3D print: the 3D models are available as OpenSCAD files, but they were fine-tuned to my printer (Ender3 modified to print 2.85mm)
- TTGO-T1 UART drivers installation on the KRIA KV-260 Ubuntu desktop OS: only a short explanation was given in this document.
- DC motor characterization + improve DC motor control: the current scripts do not have any proper motor control (e.g. PID) and there is no proper motor parameters characterization. Also, the current motor setup is generating too much vibration while running at medium speed.
- Provide in the future more videos of the implemented application.
There will be completed later on a more calm sea, and are not part of this deliverable. Finally, the repo is tagged to preserve the status of the project to comply with the challenge deadline.
Future work- Proper Overlay Pipeline
As mentioned earlier, the available architecture for the Kria-PYNQ does not allow to perform video pre-processing while using a DPU. The next step would be to develop a PYNQ overlay more suitable for the KV260-ATRover setup. The overlay should also include a proper video stream generation using the video codec capabilities of the KRIA SOM.
- Training Dataset
From what I found, most current datasets pictures are up to high, and the world is shown from a completely different perspective as compared to an almost floor level ATRover view. Now that the ATRover has some limited mobility, it will be possible to gather new training images to improve the CNN training.
- DC motors downgrade
The current DC motors are too powerful for the KV260-ATRover mini. They were selected initially for the lawnmower which has a bigger heavier chassis (higher torque requirement) and larger wheels (reducing the speed of the ATRover and therefore increasing the minimum required PWM duty cycle)
- More sensors
Additional sensors are planned (and required) to improve the ATRover control
9-DOF: 3-axis accelerometer and 3-axis gyroscopeUltrasound distance measurementAdditional cameras for mappingCliff sensor
- Chassis upgrades
Lower the camera position or update the lens to wide-angle to capture nearer objects and improve the battery compartment for easy battery replacement
- R&D other CNN topologies
Multitask CNN looks like an ideal solution for path control and planning including DC motor control. It would be interesting to spend some time looking at Single CNN dual-camera SSD/YOLO with distance estimation as output, and self-learning/self-calibration CNN for stereovision cameras.
- Explore ROS2
- Web Interface
Although JupyterLab is nice for some work, a minimalistic web interface would be a nice feature to have.
- From PYNQ to C++
After the final drawbacks, I have decided to move to C++ from Python. Although everything looks prettier on Python, an increase in performance should be expected, plus almost everything in the Vitis-AI is done in C++.
Future ProjectsWhile learning about the Kria KV-260 a couple of projects came to my mind:
- Where is my dog? I am in a rural area and occasionally I let my dog out. As he is a small breed I am always worried about him, and a solution like Alexa "show me cam##" does not satisfice my needs. I already got new IP cameras with RTSP and will begin to do some tests in early summer.
- Monitor my-Kids free time. I work from home, which is great, but for a couple of hours, while I am working, my kids are supposed to do the homework - which usually does not work as expected. There should be an easier way to monitor and time screen time.
Finally, I will try to write some starting tutorials on deploying AI/ML on the Xilinx - MPSoCs, while following the contest forums it was clear that I was not the only one kind of lost for moments - and I should have asked more questions rather than trying to solve all by myself (but I think that's how I learn more)
AcknowledgmentsI want to tank the HacksterIO community for sharing, answering, and asking questions. HacksterIO and Xilinx staff for the effort and time into promoting these challenges and activities - and answering questions. I found them a great way to improve my know-how.
Last but not the least, my wife and my kids for supporting and not letting me quit, for understanding my occasional absences (or more often coming late to dinner), and for helping with the chores (the ones I was supposed to do)
As always, comments are most welcome.
References & LinksKV260-ATRover GitHub repository
Tutorials
Stereo Vision Camera Calibration
Camera Calibration using OpenCV
Blogs
Learn OpenCV (Stereo Vision Tutorials)
Xilinx
XIR (Xilinx Intermediate Representation): graph-based intermediate representation of AI algorithms
DNNDK (Deep Neural Network Development Kit User Guide)
⚠ A lot of time was wasted while looking at the wrong Vitis AI User Guide (UG1414) version v1.4 which was incorrect, the v1.4.1 does contains the required VART API information required to deploy with Vitis AI-1.4
Wiki
YOLO output data post-processing
Comments