Vitis Acceleration Flow on KV260 - Software in PetaLinux

As I mentioned in my previous post, the next step in creating an accelerated FPGA design with AMD-Xilinx is adding the software for the accelerated kernel itself into the embedded Linux image running on the target FPGA by enabling the XRT libraries using PetaLinux.

Since the target FPGA in this series is the KV260, the hardware nodes in the device tree must be deployed as an overlay. This requires some extra steps including the manual compilation of the device tree blob which is also outlined here.

This post assumes the hardware design as outlined in my previous post as a starting point, and using Vivado/Vitis & PetaLinux versions 2021.2.

Create PetaLinux Project

The first step is to create a PetaLinux project targeting the Kria KV260 using the 2021.2 BSP as provided by AMD-Xilinx.

Start by sourcing the PetaLinux tools in the environment:

~$ source /tools/Xilinx/PetaLinux/2021.2/settings.sh

PetaLinux 2021.2 has been updated on the back-end as of the time of this writing so if you saw the upgrade command flow in the AMD-Xilinx documentation for 2021.1, you can ignore that now for both 2021.1 and 2021.2 (the petalinux-upgrade command).

Create a PetaLinux project using the Kria KV260 PetaLinux BSP in the desired directory (I personally like to create my PetaLinux projects in the top level folder of the corresponding Vivado project - the kria_kv260 Vivado project in this case).

~$ cd ./kria_kv260/
~/kria_kv260$ petalinux-create --type project -s /<path to bsp>/xilinx-k26-som-v2021.2-final.bsp

Then change directories into the newly created PetaLinux project for the Kria KV260:

~/kria_kv260$ cd ./xilinx-k26-som-2021.2/

Import XSA Exported from Vivado Project

To make the PetaLinux project aware of the hardware available configured in Vivado, import the XSA file that was exported from the Vivado project created as outlined in my previous post here:

~/kria_kv260/xilinx-k26-som-2021.2$ petalinux-config --get-hw-description=/<path to xsa>

In my previous post, I left the default export location as the top level folder of the Vivado project. So since I also created this PetaLinux project in the top level folder of the Vivado project, I always import my XSA files like this:

~/kria_kv260/xilinx-k26-som-2021.2$ petalinux-config --get-hw-description=../

No changes are needed in the hardware configuration of the PetaLinux project in this instance, so just exit the ASCII configuration GUI.

Add XRT to Root Filesystem

To add the accelerated kernel to the Linux image and the corresponding application layer hooks, enable the XRT (Xilinx Runtime) in the root filesystem.

Launch the root filesystem ASCII configuration editor:

~/kria_kv260/xilinx-k26-som-2021.2$ petalinux-config -c rootfs

Then enable xrt under Filesystem Packages > libs > xrt:

Exit & save the root filesystem configuration.

Build PetaLinux Project & Generate SDK

With the XRT components added, build the PetaLinux project to compile the new embedded Linux image:

~/kria_kv260/xilinx-k26-som-2021.2$ petalinux-build

After the project has been successfully built, generate the SDK for the project. This SDK is what is used as sysroot for the accelerated application development later on.

~/kria_kv260/xilinx-k26-som-2021.2$ petalinux-build --sdk

Don't be alarmed if this step takes a little while, especially compared to the initial PetaLinux project build itself. On my machine (Intel quad-core i9, 64GB RAM), the PetaLinux project build took about 6 minutes while the SDK generation took 15 minutes.

Install Device Tree Generator

On the Kria KV260, Linux is actually responsible for programming the PL with the bitstream as one of the later steps in the boot process. This means that the hardware IP in the PL has to be defined in the device tree by a device tree overlay that can be loaded dynamically at runtime (ie - the whole function of device tree overlays in Linux).

The device tree for any Vitis accelerated design requires two main components: the ZOCL node to load the XRT driver, and all of the nodes for the IP located in the PL (the block design done in Vivado).

AMD-Xilinx has their own device tree generator (DTG) that will convert their archive file format (.XSA) into a device tree source file (.dtsi) - the Linux device tree generator for Xilinx SDK. Xilinx's DTG is also configurable for whatever information needs to be generated.

If not already installed, install the device tree compiler tool in Linux:

~$ sudo apt install device-tree-compiler

Create a directory to clone the Linux device tree generator for the Xilinx SDK into (again, I placed this in the top level directory of the KV260 Vivado project just to keep everything together):

~$ mkdir -p ./kria_kv260/xilinx-dtg
~$ cd ./kria_kv260/xilinx-dtg/

Clone Linux device tree generator for the Xilinx SDK, change directories into the repository, and checkout the 2021.2 branch:

~/kria_kv260/xilinx-dtg$ git clone https://github.com/Xilinx/device-tree-xlnx
~/kria_kv260/xilinx-dtg$ cd device-tree-xlnx
~/kria_kv260/xilinx-dtg/device-tree-xlnx$ git checkout xlnx_rel_v2021.2

At this point, everything needed to generate device trees for Vitis accelerated designs is installed: Vitis, PetaLinux, the Linux device tree compiler tool, and the Xilinx device tree generator.

Generate Device Tree Source From XSA

With all of the necessary tools installed the next step is to create the device tree source file for the Kria KV260 using the XSA file exported from the Vivado project that was configured to be an extensible Vitis platform (for this case, see previous post here).

To keep things organized, I created a specific directory for it in the top level Vivado project folder (separate from where I cloned the Xilinx DTG to):

~$ mkdir -p ./kria_kv260/kv260_accel_dts
~$ cd ./kria_kv260/kv260_accel_dts

Source the Vitis settings and launch the Xilinx software command line tool (XSCT):

~/kria_kv260/kv260_accel_dts$ source /tools/Xilinx/Vitis/2021.2/settings64.sh
~/kria_kv260/kv260_accel_dts$ xsct

As mentioned above Xilinx's DTG is configurable to output a device tree source file with whatever is needed, and the two requirements here (since it's an accelerated design) are to add the ZOCL node to load the XRT driver and the overlay functionality for loading the device nodes for the IP in the PL at runtime. Use the following HSI commands in XSCT to accomplish this:

xsct% hsi open_hw_design <design_name.xsa>
xsct% hsi set_repo_path <path to device-tree-xlnx repository>
xsct% hsi create_sw_design device-tree -os device_tree -proc psu_cortexa53_0
xsct% hsi set_property CONFIG.dt_overlay true [hsi::get_os]
xsct% hsi set_property CONFIG.dt_zocl true [hsi::get_os]
xsct% hsi generate_target -dir <desired_dts_filename>
xsct% hsi close_hw_design [hsi current_hw_design]

So in my case:

xsct% hsi open_hw_design ../kv260_custom_platform.xsa
xsct% hsi set_repo_path ../xilinx-dtg/
xsct% hsi create_sw_design device-tree -os device_tree -proc psu_cortexa53_0
xsct% hsi set_property CONFIG.dt_overlay true [hsi::get_os]
xsct% hsi set_property CONFIG.dt_zocl true [hsi::get_os]
xsct% hsi generate_target -dir pl.dtsi
xsct% hsi close_hw_design [hsi current_hw_design]

Exit XSCT after closing design:

xsct% exit

Compile Device Tree Source into Device Tree Blob

With the device tree source file generated, the final step is to compile it into the device tree blob for the Linux image on the Kria to use. This is done using the general Linux device tree compiler tool with the PetaLinux tools sourced in the shell.

Change directories into the generated folder from the previous step and source the PetaLinux tools:

~/kria_kv260/kv260_accel_dts$ cd ./pl.dtsi/
~/kria_kv260/kv260_accel_dts/pl.dtsi$ source /tools/Xilinx/PetaLinux/2021.2/settings.sh

Then compile the device tree source file into the device tree blob:

~/kria_kv260/kv260_accel_dts/pl.dtsi$ dtc -@ -O dtb -o pl.dtbo pl.dtsi

With the hardware and software components now generated for the accelerated design on the Kria KV260, the final step in the workflow is to package everything into a Vitis Platform which will be covered in the next project post of this series.