Vitis Acceleration Flow on KV260 - Vitis Platform

In the final post of my project series on how to create an accelerated FPGA design on the Kria KV260 Vision AI kit, I go over how to create the accelerated software in Vitis. This encompasses steps 3 and 4 in the graphic below from AMD-Xilinx:

In my previous post, I cover the process of adding the accelerated packages/libraries (XRT) to the root filesystem for the Kria using PetaLinux and the generate of the SDK for sysroot. Sysroot is a source directory that serves as the root for headers and libraries specific to the target accelerated application for the compiler in Vitis to pull from when building the accelerated application.

This post picks up where the last left on with installing sysroot, then creating the platform and application project in Vitis, preparing the proper boot components for the Kria KV260, and finally transferring the accelerated application's files to the Kria to test it out.

Since I'm purely focusing on the mechanics and process of how to create and deploy an accelerated application on the Kria KV260, I will simply be using the accelerated vector addition example template unmodified and saving the details of custom accelerated application for it's own post in an upcoming project. There is quite a bit of detail in the algorithm development that would make this post entirely too long.

Install Sysroot

The first thing that needs to be done is sysroot needs to be installed as it contains the header and library files the Vitis compiler will need to build the accelerated application. Create the following directory structure:

custom_package_directory
    -> pfm
        -> boot
        -> sd_dir

To follow my own personal presences of file organization, I chose to create this in the top level directory of the corresponding Vivado project (see that project tutorial post here), kria_kv260, and I chose to name the custom package directory kria_kv260_custom_pkg:

~$ cd ./kria_kv260
~/kria_kv260$ mkdir -p kria_kv260_custom_pkg 
~/kria_kv260$ cd ./kria_kv260_custom_pkg
~/kria_kv260/kria_kv260_custom_pkg$ mkdir -p pfm
~/kria_kv260/kria_kv260_custom_pkg$ cd ./pfm
~/kria_kv260/kria_kv260_custom_pkg/pfm$ mkdir -p boot 
~/kria_kv260/kria_kv260_custom_pkg/pfm$ mkdir -p sd_dir

Then install sysroot into the custom package directory by running the sdk.sh script generated from the petalinux-build --sdk command:

~$ source /tools/Xilinx/PetaLinux/2021.2/settings.sh
~$ cd ./kria_kv260/xilinx-k26-som-2021.2/images/linux
~/kria_kv260/xilinx-k26-som-2021.2/images/linux$ ./sdk.sh -d ../../../kria_kv260_custom_pkg

Answer yes when prompted then wait until install is complete:

Installation of the SDK may take a few minutes.

Prep Boot Components

One more thing before creating the Vitis workspace, the necessary boot components such as the FSBL and U-Boot need to be copied into the platform boot directory from the Kria PetaLinux project with the XRT packages enabled in it.

Change directories into the build output directory of the PetaLinux project ./images/linux and copy the following files into the platform boot directory ~/kria_kv260/kria_kv260_custom_pkg/pfm/boot:

• zynqmp_fsbl.elf
• pmufw.elf
• bl31.elf
• u-boot-dtb.elf
• u-boot.elf
• system.dtb

Note: Since I also have the PetaLinux project that's been created from the 2021.2 BSP located in the top level directory of the corresponding Vivado project, that's why my directory location is ~/kria_kv260/xilinx-k26-som-2021.2/images/linux

~/kria_kv260/xilinx-k26-som-2021.2/images/linux$ cp ./zynqmp_fsbl.elf ../../../kria_kv260_custom_pkg/pfm/boot
~/kria_kv260/xilinx-k26-som-2021.2/images/linux$ cp ./pmufw.elf ../../../kria_kv260_custom_pkg/pfm/boot
~/kria_kv260/xilinx-k26-som-2021.2/images/linux$ cp ./bl31.elf ../../../kria_kv260_custom_pkg/pfm/boot
~/kria_kv260/xilinx-k26-som-2021.2/images/linux$ cp ./u-boot-dtb.elf ../../../kria_kv260_custom_pkg/pfm/boot
~/kria_kv260/xilinx-k26-som-2021.2/images/linux$ cp ./u-boot.elf ../../../kria_kv260_custom_pkg/pfm/boot
~/kria_kv260/xilinx-k26-som-2021.2/images/linux$ cp ./system.dtb ../../../kria_kv260_custom_pkg/pfm/boot

Overall the custom platform package directory should look like this:

-> kria_kv260_custom_pkg
    -> pfm
        -> boot
            * zynqmp_fsbl.elf
            * pmufw.elf
            * bl31.elf
            * u-boot.elf
            * u-boot-dtb.elf
            * system.dtb
        -> sd_dir
    -> sysroots (contains installed SDK)
    * environment-setup-cortexa72-cortexa53-xilinx-linux
    * site-config-cortexa72-cortexa53-xilinx-linux
    * version-cortexa72-cortexa53-xilinx-linux

Create Vitis Platform

Launch the Vitis IDE and set the /kria_kv260/kria_kv260_custom_pkg directory where sysroot was installed in previous steps as the workspace. As I mentioned earlier, this is what provides Vitis with the headers and libraries for the accelerated application in order to build it.

~$ source /tools/Xilinx/Vitis/2021.2/settings64.sh
~$ vitis

Create a new platform project targeting the XSA exported from Vivado for the Kria accelerated design. Select Linux for the Operating system and uncheck/deselect the option for Generate boot components:

Once the platform project has generated, the platform.spr file will open automatically. Select linux on psu_cortexa53 to configure file sources and generate a BIF file:

• Select the dropdown arrow on the Browse button for BIF File and select Generate BIF.
• Set Boot Components Directory to the platform boot directory created in the previous step: /kria_kv260/kria_kv260_custom_pkg/pfm/boot
• Set FAT32 Partition Directory to the platform SD card directory created in the previous step: /kria_kv260/kria_kv260_custom_pkg/pfm/sd_dir

After setting the necessary source directories for the Linux domain of the platform, build the platform project (ctrl+B).

Create Accelerated Application

Create a new application project by selecting File > New > Application and targeting the platform created by the platform project in the previous step and name the application vadd.

In the domain setup window, specify the sysroot path, root filesystem location, and kernel image location.

• Sysroot path: ~/kria_kv260/kria_kv260_custom_pkg/sysroots/cortexa72-cortexa53-xilinx-linux
• Root filesystem location: ~/kria_kv260/xilinx-k26-som-2021.2/images/linux/rootfs.ext4
• Kernel image location: ~/kria_kv260/xilinx-k26-som-2021.2/images/linux/Image

Finally, for the application template select Simple Vector Addition under Acceleration templates with PL and AIE accelerators and click Finish to generate the accelerated application project.

Once the application project has generated, open the vadd.prj file from the Explorer window. In the upper right corner, change the option in the dropdown for Active build configuration from Emulator-SW to Hardware. This is because we won't be doing any emulation of the accelerated application for the Kria KV260, we'll be deploying it straight to hardware.

With the build configuration set to target the hardware, build the accelerated application. Select vadd_system in the Explorer window and right-click to select build project or click the build icon in the toolbar.

The application project will take some time to build, but it is calling Vitis HLS as well under the hood, so be sure that you have applied the y2k22 patch or it will error out.

Convert Bitstream File Type

The bitstream loaded into the Kria's programmable logic at runtime by the fpgamanager needs to be in the.bin format. The bootgen tool can convert the bitstream from .bit to .bit.bin

Change directories into the accelerated application project directory that was created in Vitis, /kria_kv260/kria_kv260_custom_pkg. Create a directory called Package and copy the system bitstream into it:

~$ cd ./kria_kv260/kria_kv260_custom_pkg/
~/kria_kv260/kria_kv260_custom_pkg$ cd ./vadd_system/
~/kria_kv260/kria_kv260_custom_pkg/vadd_system$ mkdir -p Package
~/kria_kv260/kria_kv260_custom_pkg/vadd_system$ cd ./Package/
~/kria_kv260/kria_kv260_custom_pkg/vadd_system/Package$ cp ../Hardware/package.build/package/system.bit ./

Source the Vitis tools in the environment of the terminal to get the bootgen tool. Create a boot image then use bootgen to .bit.bin

~/kria_kv260/kria_kv260_custom_pkg/vadd_system/Package$ source /tools/Xilinx/Vitis/2021.2/settings64.sh
~/kria_kv260/kria_kv260_custom_pkg/vadd_system/Package$ echo 'all:{system.bit}'>bootgen.bif
~/kria_kv260/kria_kv260_custom_pkg/vadd_system/Package$ bootgen -w -arch zynqmp -process_bitstream bin -image bootgen.bif

Finally, rename system.bit.bin to vadd.bit.bin

~/kria_kv260/kria_kv260_custom_pkg/vadd_system/Package$ mv system.bit.bin vadd.bit.bin

Compile Device Tree Blob

Before compiling the device tree blob, the device tree source files need to be generated (or regenerated) using Xilinx software command line tool (XSCT).

Create a new directory and change directories (CD) into it:

~$ mkdir -p ./kria_kv260/kv260_accel_dts
~$ cd ./kria_kv260/kv260_accel_dts

Or if you previously generated this like I did, change directories (CD) into the existing folder and delete the previous version:

~/kria_kv260/kria_kv260_custom_pkg/vadd_system/Package$ cd ../../../kv260_accel_dts/
~/kria_kv260/kria_kv260_custom_pkg/vadd_system/Package$ rm -R ./pl.dtsi

Then source Vitis if using a different terminal than in the last step and launch XSCT. Use HSI to create the accelerated device tree source files:

~/kria_kv260/kv260_accel_dts$ source /tools/Xilinx/Vitis/2021.2/settings64.sh
~/kria_kv260/kv260_accel_dts$ xsct
xsct% hsi open_hw_design ../kv260_custom_platform.xsa
xsct% hsi set_repo_path ../xilinx-dtg/
xsct% hsi create_sw_design device-tree -os device_tree -proc psu_cortexa53_0
xsct% hsi set_property CONFIG.dt_overlay true [hsi::get_os]
xsct% hsi set_property CONFIG.dt_zocl true [hsi::get_os]
xsct% hsi generate_target -dir pl.dtsi
xsct% hsi close_hw_design [hsi current_hw_design]
xsct% exit

Open the pl.dtsi file in the generated device tree source file directory and change the firmware name on line 16 to match the accelerated application's name (vadd):

Finally, CD into the generated device tree source file directory and compile the vadd device tree blob:

~/kria_kv260/kv260_accel_dts$ cd ./pl.dtsi/
~/kria_kv260/kv260_accel_dts/pl.dtsi$ dtc -@ -O dtb -o vadd.dtbo pl.dtsi

Theorectically, I think you're supposed to be able to simply change the firmware name in pl.dtsi and recompile using dtc. But I kept getting an error that the the vadd node did not exist, which I could only resolve by recreating the device tree source files from the XSA using the HSI tools before changing the firmware name and compiling using dtc. I'm not sure if this is just something weird in 2021.2 or what, so keep this in mind if you're using a different version than 2021.2.

Transfer Files to Kria KV260

Create new.wic SD card image in the aforementioned PetaLinux project that includes the XRT packages to boot the Kria KV260 with:

~$ source /tools/Xilinx/PetaLinux/2021.2/settings.sh
~$ cd ./kria_kv260/xilinx-k26-som-2021.2
~/kria_kv260/xilinx-k26-som-2021.2$ petalinux-package --wic --bootfiles "ramdisk.cpio.gz.u-boot boot.scr Image system.dtb"

Flash an SD card with balenaEtcher or from command line:

~$ lsblk
~$ sudo dd bs=4M if=./kria_kv260/xilinx-k26-som-2021.2/images/linux/petalinux-sdimage.wic of=/dev/sdc status=progress && sync
~$ sudo umount /media/<user>/boot
~$ sudo umount /media/<user>/root

Boot the Kria KV260 and connect it to your network (I connect mine straight to my router).

Create a directory on your host machine and copy all files into it that will be transferred to the Kria:

• vadd.dtbo - Device tree with overlay for vadd's hooks to the PL.
• vadd.bit.bin - FPGA bitstream with vadd's hooks to the PL.
• binary_container_1.xclbin - The accelerated binary container for XRT configuration that includes system.bit and kernel metadata.
• vadd - The compiled accelerated app.
• shell.json - The metadata file for the Xilinx library that implements an on-target daemon, dfx-mgr.

~/kria_kv260/kria_kv260_custom_pkg$ mkdir -p vadd_file_transfer
~/kria_kv260/kria_kv260_custom_pkg$ cd ./vadd_file_transfer/
~/kria_kv260/kria_kv260_custom_pkg/vadd_file_transfer$ cp ../../kv260_accel_dts/pl.dtsi/vadd.dtbo ./
~/kria_kv260/kria_kv260_custom_pkg/vadd_file_transfer$ cp ../vadd_system/Package/vadd.bit.bin ./
~/kria_kv260/kria_kv260_custom_pkg/vadd_file_transfer$ cp ../vadd_system/Hardware/binary_container_1.xclbin ./
~/kria_kv260/kria_kv260_custom_pkg/vadd_file_transfer$ cp ../vadd/Hardware/vadd ./

Create shell.json:

~/kria_kv260/kria_kv260_custom_pkg/vadd_file_transfer$ nano shell.json

Add the following lines to shell.json:

{
  "shell_type" : "XRT_FLAT",
  "num_slots": "1"
}

Transfer files using the local IP address assigned to the Kria on the network:

~/kria_kv260/kria_kv260_custom_pkg/vadd_file_transfer$ scp vadd vadd.bit.bin vadd.dtbo binary_container_1.xclbin shell.json petalinux@192.168.1.51:/home/petalinux

Load Hardware & Run Accelerated Application

Once all of the accelerated application files have been uploaded to the Kria, create the directory for it in /lib/firmware/<company name>/<app name> that dfx-mgr (which is what xmutil is using under the hood) will look for the files in and move the files to it:

xilinx-k26-som-2021_2:~$ sudo mkdir -p /lib/firmware/xilinx/vadd
xilinx-k26-som-2021_2:~$ sudo mv vadd.bit.bin vadd.dtbo shell.json /lib/firmware/xilinx/vadd/

Note: you can't transfer the files straight to this location since it's in the root directory and requires superuser privileges.

Next use the xmutil command to list the accelerated applications available and verify the vadd application is there but not the current application loaded. Unload the default application and load vadd:

xilinx-k26-som-2021_2:~$ sudo xmutil listapps
xilinx-k26-som-2021_2:~$ sudo xmutil unloadapp
xilinx-k26-som-2021_2:~$ sudo xmutil loadapp vadd

List the accelerated applications again to verify that vadd is loaded:

xilinx-k26-som-2021_2:~$ sudo xmutil listapps

Before running the vadd application for the first time, you may need to make the file an executable:

xilinx-k26-som-2021_2:~$ chmod +x vadd

Finally run the vadd application:

xilinx-k26-som-2021_2:~$ ./vadd binary_container_1.xclbin

And that's the mechanics of creating an accelerated application and deploying it on the the Kria KV260. As I mentioned before, since this post is long enough already, I will be creating future posts focusing on the algorithm development to replace the vector addition (vadd) code in this post.