System Design with Vitis Unified

Introduction

Over 100 plus projects that we have created here on Hackster, we have when targeting AMD Zynq MPSoC used a flow which is Vivado centric.

That is the design of the programmable logic was created using Vivado and the application software was then developed.

However, when working with embedded solutions such as the Zynq and Versal adaptive SoCs we can develop the solution in a different manner using the Vitis Software Heterogeneous System flow which creates a solution consisting of embedded software, High Level Synthesis Modules, Model Based design modules from Vitis Model Composer, AIE Blocks and of course RTL modules.

What this really means is that when developing in Vitis using the heterogeneous system development we are able to pull together a diverse range of resources for our solution and package it into system solution.

In this project we are going to look at how we are able to develop such a solution for a real-world application. This will pull together several different design modules.

However, before we progress, we need to outline a few key concepts and terms.

The output of the heterogeneous system design compilation provides a SD Card image. What is on the SD Card varies a little depending on the target device for the Zynq UltraScale+ MPSoC and Kria SoMs the output format is a XCLBin, while for Versal designs it is as fixed platform (XSA), I will describe exactly what this is below.

If we are working with a custom Versal platform we can export the Vitis design into a Vitis Metadata Archive (VMA) and re import it into Vivado for further analysis.

Both of these flows use Vivado under the hood and as we get more comfortable using the Vitis heterogeneous system design approach we can also work manually with Vivado during different stages of the implementation to enable customization if desired.

To be able to effectively work with the Vitis Heterogeneous System Design flow there are some key concepts we need to understand.

Extensible Platform – This is a target platform for Vitis System Design Flow. It is called extensible as it can be extended by the addition of PL kernels or AI Engine Graph applications. AI Engine applications are dataflow graph specifications written in C++, these dataflow graphs define compute kernels and data connections.

Fixed Platform – This includes a completed hardware design and supporting software files e.g. OS, Libraries and Boot files.

Kernel Image – This is the embedded Linux kernel image which we will be running on the Kria SoM, Zynq UltraScale+ MPSoC or Versal adaptive SoC.

PS Application – The software application which runs on the arm cores and controls and interacts with the PL Kernels and AI Engines.

RootFS – This is the Root File System which is mounted by the embedded Linux Kernel.

SysRoot – This is a reduced of the filesystem (RootFS), it only contains the libraries and headers which can be compiled and linked against.

Vitis Compiler – The V++ compiler which is able to compile PL Kernels and links the PL kernels and AI Graphs to build the device.

XRT – Xilinx Run Time Library – This is an API and associated drivers which allow data and stats transfer between the OS and the hardware design which contains the PL kernels and AI Graph.

Xilinx Support Archive – This is the hardware container exported from Vivado

The Heterogeneous System design flow is based around PetaLinux, as such we need to be able to provide the Kernel Image, Roots FS and SysRoot also to Vitis.

We also need to develop the solution using a Linux machine or virtual machine, in my case I am using a Ubuntu 22.04 LTS.

To help us get started on this development flow AMD helpfully provide us with a common images, sysroots, rootfs and extensible platforms for a range of Versal and UltraScale+ MPSoC Targets. You can download these from the AMD Downloads page here.

In this project we are going to learn how to create a custom platform and then integrate a range of PL and HLS kernels. This will enable the implementation of an application which is able to filter external signals captured by the System Monitor. This project will include the use of RTL kernels and two HLS kernels to perform filtering and algorithm offloading.

Our final project will looks as below.

Project Flow

As this project is going to use Vivado and Vitis, including several switches, I have outline a summary of the flow

1. Platform Creation in Vivado

• Create a new RTL project, marked as Extensible Vitis Platform.
• Configure ZUBoard hardware, clocks (100, 200, 300 MHz), resets, and AXI interfaces.
• Enable PL-PS interrupts and define platform metadata.
• Export hardware platform as XSA (pre-synthesis).

2. Platform Completion in Vitis

• Import XSA to create Vitis Platform Component.
• Configure OS (Linux), processor (A53), and provide prebuilt files (Kernel, RootFS, SysRoot).
• Build the software platform, making it ready for application development.

3. Example Application

• Import and build a Simple Vector Addition example targeting the new platform.
• Generate SD Card image and optionally validate on real hardware.

4. Custom Hardware Extension in Vivado

• Add SysMon IP to read analog inputs.
• Create custom Sysmon to AXI-Stream RTL adapter.
• Export the updated hardware platform.

5. HLS Kernel Development in Vitis

• Create Low Pass Filter HLS kernel using C++ (sliding window average filter).
• Add to platform, define AXI-Stream interfaces.

6. AXI Stream Connectivity Configuration

• Define stream interconnections between:
• Sysmon Adapter ➔ Low Pass Filter ➔ Vector Addition Kernel
• Specify clock domain connections in connectivity config.

7. Application Kernel and Host Updates

• Modify vector addition kernel to accept and process Sysmon stream data.
• Extend host application to handle new buffers and capture Sysmon output.

8. Build and Test

• Build the full system design including hardware and software.
• Generate SD Card image with custom logic and processing pipeline.

To get started the first thing we need is a platform

Platform

We can create a platform for any target of the system design flow.

To define the platform, we need to start out in Vivado and ensure we have correctly configured the available resources.

Within this platform we will be defining the clocking architecture, available interfaces and interrupts.

All of this development is taking place on a Ubuntu Linux development machine.

To get started from a terminal window we need to run the command below.

The next step is to create an extensible project in Vivado, the starting point is to set a new project.

Select a RTL project and ensure the Project is an extensible Vitis platform is checked.

Select the ZU Board.

Complete the project creation.

Once the project is open the next step is to open a block diagram - name it system.

With the block diagram open the next step is to add in a MPSoC Processing System.

Run the block automation.

Add in a clock wizard and enable three outputs, at 100 MHz, 200 MHz and 300 MHz. Ensure the reset type is set to active low.

Connect the clock out of the processing system clock to the clock in of the clock wizard. Do the same with the reset.

Add in a processor reset block, connect the ext reset to the processor reset, connect the clock wizard out to the slowest clock input and connect the locked signal.

Add in two more processor reset blocks and connect them to the clock wizard.

Reconfigure the Processing System and remove the AXI Interfaces.

Enable the PL-PS Interrupts.

Select the platform tab and enable the clocks we just inserted. I selected the lowest clock as the default.

Select the AXI Ports and enable them all apart from the S_AXI_LPD

Check the interrupt page and select the IRQ0 and IRQ1 as enabled.

On the platform add in the vendor name.

Validate the block design, the final block design should be as below.

Create the HDL wrapper and let Vivado Manage it.

To enable the Processing System to be used for a HW emulation flow, select the processing system and change the selected sim model to TLM.

Generate the output products, select Global as the option as this will generate the intermediate elements but not run the out of context simulation.

Use the Platform tab to export the platform, select Hardware as the type to be exported.

Select Pre Synthesis.

Enter the details for the platform properties.

Select the location for the output.

Click finish to export the platform.

This creates the hardware element of the platform, to create the software element of the platform and complete the platform creation we need to use Vitis Unified.

Once the workspace has been set the next step is to create the platform.

Within Vitis select create platform component option and provide a name for the platform.

Select hardware design and point it to the XSA just exported. Ensure the DT ZOCL is checked.

On the next tab select Linux as the OS, the processor should default to the A53.

Click on finish

Once the platform is open before we can compile it we need to make a few minor tweaks.

Under the domain, change the display name to xrt, click auto generate for the BIF and point the pre built directory to the files we downloaded from AMD earlier.

Click on Build and you will see the platform be built and available for use by applications.

Test Application

Before we start with the development of the main application we need to verify the flow.

From the Examples tab select the Simple Vector addition template project.

Enter the system project name and location.

Select the platform just created.

The project will be created and we are then able to build the application, as it does this it will create a SD Card Image we can write to the SD Card.

We can explore the test application by opening Vivado and seeing the resultant hardware design which was created.

If you desire you can burn the SD Card Image and try running the example on hardware.

Custom Application

Of course the real power of the heterogeneous system flow comes with the ability to integrate a custom platform, with extra additions along with adding in further HLS kernels.

For this application we are going to work with the Sysmon which gives us the ability to read in external analogue signals. We can then work with these signals in Vitis for example filtering or FFTs, using HLS Kernels.

To start with this we will need to update our hardware platform. To this platform we are going to add in a Sysmon module and a custom RTL module to convert the output of the Sysmon to an AXI stream.

The first step is to add in the Sysmon module from the IP Catalogue to the previous Vivado project.

Disable the on chip sensors.

Disable the alarms.

On the ZUBoard the XADC channels are routed to the Click connector.

Select the Vaux6 differential pair.

The next step is to create a HDL module which converts the output of the Sysmon to AXI stream such that we can connect the stream in Vitis.

The RTL is as below.

library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

entity sysmon_to_axis is port(
    aclk : in std_logic;
    rstn : in std_logic;
    
    eoc : in std_logic;
    data : in std_logic_vector(15 downto 0);
    
    m_rdy : in std_logic; 
    m_data : out std_logic_vector(15 downto 0);
    m_valid : out std_logic
    
);
end entity;

architecture rtl of sysmon_to_axis is 
  
      ATTRIBUTE X_INTERFACE_INFO : STRING;
      ATTRIBUTE X_INTERFACE_INFO of m_data: SIGNAL is "xilinx.com:interface:axis:1.0 M_AXIS TDATA";
      ATTRIBUTE X_INTERFACE_INFO of m_valid: SIGNAL is "xilinx.com:interface:axis:1.0 M_AXIS TVALID";
      ATTRIBUTE X_INTERFACE_INFO of m_rdy: SIGNAL is "xilinx.com:interface:axis:1.0 M_AXIS TREADY";
      ATTRIBUTE X_INTERFACE_PARAMETER : STRING;
      ATTRIBUTE X_INTERFACE_PARAMETER of aclk: SIGNAL is "CLK_DOMAIN clk_out_1,PHASE 0,FREQ_HZ 100000000,LAYERED_METADATA false,HAS_TLAST false,HAS_TKEEP false,HAS_TSTRB false,HAS_TREADY false,TUSER_WIDTH 0,TID_WIDTH 0,TDEST_WIDTH 0,TDATA_NUM_BYTES 2";
    
begin 
   
   process(aclk)
   begin
        if rising_edge(aclk) then 
            m_valid <= eoc;
            m_data <= data;
 
        end if;
   end process;


end architecture;

Once the VHDL file has been created we can add it into the IP Integrator design by right clicking on the file and selecting Add Module to Block Design.

Once added we can connect its IO into the design as we would for any other IP Integrator IP Core.

The final updated platform looks as below.

On the platform setup tab, on the AXI Stream port select the M_AXIS interface and provide it a SP_TAG.

Export the platform and re import it into Vitis System project, this will make available the Sysmon AXI Stream to the Vitis System application. What we need to do now is create some kernels within the Vitis application.

Creating a HLS Core

As the Sysmon input can read a low speed analogue signal, for purposes of demonstration I am going to a HLS module which will implement a low pass filter.

This would be useful in a application for removing noise or spurious signals from the input.

To get started we need to create a new HLS component.

Give the component a name.

For the configuration file, I selected empty File.

I left the add sources empty.

Select the platform we are working on.

I defined the clock to be the same as the base clock at 100 MHz.

With the creation process completed I clicked on finish.

Once the HLS component is created you will see it appear under the Vitis Explorer.

We can then create a new C source file which will define the filter.

I named the file as below.

The source code to be included in the file is below, this is a simple three sample moving average filter. This implements a simple low pass filter.

To ensure it can be connected easily with the Sysmon AXI Stream the AXI stream is defined to be 16 bits wide.

The inputs are both AXI Stream while the control interface is set to none. This will mean it is able to free run as data is received over the stream.

#include <hls_stream.h>
#include <ap_axi_sdata.h>

#define WINDOW_SIZE 3

// AXI stream with 32-bit data, no user/ID/keep/strb
typedef ap_axiu<16,0,0,0> axi_t;

void low_pass_filter(hls::stream<axi_t>& in_stream,
                     hls::stream<axi_t>& out_stream) {

#pragma HLS INTERFACE axis port=in_stream
#pragma HLS INTERFACE axis port=out_stream
#pragma HLS INTERFACE ap_ctrl_none port=return

    // Static variables to retain filter state
    static int window[WINDOW_SIZE] = {0, 0, 0};
    static int index = 0;
    static int sum = 0;

    if (!in_stream.empty()) {
#pragma HLS PIPELINE II=1

        // Read input
        axi_t in_data = in_stream.read();

        // Moving average logic
        int old_val = window[index];
        int new_val = in_data.data;

        sum = sum - old_val + new_val;
        window[index] = new_val;

        index = (index + 1) % WINDOW_SIZE;

        // Prepare and write output
        axi_t out_data;
        out_data.data = sum / WINDOW_SIZE;
        out_data.keep = in_data.keep;  // Optional: preserve 'keep'
        out_data.strb = in_data.strb;  // Optional: preserve 'strb'
        out_data.last = in_data.last;  // Important for stream framing
        out_stream.write(out_data);
    }
}

With the source created the next step is to update the synthesis settings for the FIR filter. We are then ready to start with the system build.

System Build

We will add in the new kernel into the existing design which was used to test the platform.

The first stage is to add the HLS kernel to the binary container.

Following this you will observe the low_pass_filter being included within the binary container.

With the Kernel included we need to define the AXI Stream connectivity as we do not want to keep going back to memory in the processor for each stage. This would significantly reduce performance. Instead, we are able to define the connectivity very simply.

We define the connectivity of the AXI Streams using a config file. This file needs to reside under the main project HW_links folder.

Within this file we can define the connectivity of the stream interfaces. Connecting from the Sysmon to the Low Pass Filter input and output streams. This is defining the connections we desire to be used for the AXI stream interfaces within the design.

[connectivity]
stream_connect = SYSMON_AXIS:low_pass_filter_1.in_stream
stream_connect = low_pass_filter_1.out_stream:krnl_vadd_1.sysmon_in
connect = low_pass_filter_1/ap_clk:system_clk_wiz_0_0_clk_out1

In the example kernel add we will also need to update the source code to connect the incoming AXI Stream to the memory.

The first thing we need to do is define the AXI Stream type.

typedef ap_axis<15, 0, 0, 0> pkt;

following that define a process which stores the values of the incoming AXI Stream.

static void sysmon_process( int *wave, hls::stream<pkt> &s_in) {
    for (int i = 0; i < 1024; i++) 
    {
        #pragma HLS PIPELINE II = 1
        pkt value = s_in.read();
        wave[i] = value.data;
    }
}

Within the main kernel function add in the following. This will define the AXI Stream interface and call the function just created.

#pragma HLS INTERFACE axis port = sysmon_in

sysmon_process(wave_out,sysmon_in);void krnl_vadd(uint32_t* in1, uint32_t* in2, uint32_t* out, int size, int *wave_out, hls::stream<pkt> &sysmon_in) {
#pragma HLS INTERFACE m_axi port = in1 bundle = gmem0
#pragma HLS INTERFACE m_axi port = in2 bundle = gmem1
#pragma HLS INTERFACE m_axi port = out bundle = gmem0
#pragma HLS INTERFACE m_axi port = wave_out bundle = gmem2
#pragma HLS INTERFACE axis port = sysmon_in

    static hls::stream<uint32_t> in1_stream("input_stream_1");
    static hls::stream<uint32_t> in2_stream("input_stream_2");
    static hls::stream<uint32_t> out_stream("output_stream");


#pragma HLS dataflow
    // dataflow pragma instruct compiler to run following three APIs in parallel
    load_input(in1, in1_stream, size);
    load_input(in2, in2_stream, size);
    compute_add(in1_stream, in2_stream, out_stream, size);
    store_result(out, out_stream, size);
    sysmon_process(wave_out,sysmon_in);
}

We also then need to change the host application. This is to add the buffers and correctly queue and interact with them in the OpenCL environment.

int main(int argc, char* argv[]) {
    // TARGET_DEVICE macro needs to be passed from gcc command line
    if (argc != 2) {
        std::cout << "Usage: " << argv[0] << " <xclbin>" << std::endl;
        return EXIT_FAILURE;
    }

    std::string xclbinFilename = argv[1];

    // Compute the size of array in bytes
    size_t size_in_bytes = DATA_SIZE * sizeof(int);

    // Creates a vector of DATA_SIZE elements with an initial value of 10 and 32
    // using customized allocator for getting buffer alignment to 4k boundary

    std::vector<cl::Device> devices;
    cl_int err;
    cl::Context context;
    cl::CommandQueue q;
    cl::Kernel krnl_vector_add;
    cl::Program program;
    std::vector<cl::Platform> platforms;
    bool found_device = false;

    // traversing all Platforms To find Xilinx Platform and targeted
    // Device in Xilinx Platform
    cl::Platform::get(&platforms);
    for (size_t i = 0; (i < platforms.size()) & (found_device == false); i++) {
        cl::Platform platform = platforms[i];
        std::string platformName = platform.getInfo<CL_PLATFORM_NAME>();
        if (platformName == "Xilinx") {
            devices.clear();
            platform.getDevices(CL_DEVICE_TYPE_ACCELERATOR, &devices);
            if (devices.size()) {
                found_device = true;
                break;
            }
        }
    }
    if (found_device == false) {
        std::cout << "Error: Unable to find Target Device " << std::endl;
        return EXIT_FAILURE;
    }

    std::cout << "INFO: Reading " << xclbinFilename << std::endl;
    FILE* fp;
    if ((fp = fopen(xclbinFilename.c_str(), "r")) == nullptr) {
        printf("ERROR: %s xclbin not available please build\n", xclbinFilename.c_str());
        exit(EXIT_FAILURE);
    }
    // Load xclbin
    std::cout << "Loading: '" << xclbinFilename << "'\n";
    std::ifstream bin_file(xclbinFilename, std::ifstream::binary);
    bin_file.seekg(0, bin_file.end);
    unsigned nb = bin_file.tellg();
    bin_file.seekg(0, bin_file.beg);
    char* buf = new char[nb];
    bin_file.read(buf, nb);

    // Creating Program from Binary File
    cl::Program::Binaries bins;
    bins.push_back({buf, nb});
    bool valid_device = false;
    for (unsigned int i = 0; i < devices.size(); i++) {
        auto device = devices[i];
        // Creating Context and Command Queue for selected Device
        OCL_CHECK(err, context = cl::Context(device, nullptr, nullptr, nullptr, &err));
        OCL_CHECK(err, q = cl::CommandQueue(context, device, CL_QUEUE_PROFILING_ENABLE, &err));
        std::cout << "Trying to program device[" << i << "]: " << device.getInfo<CL_DEVICE_NAME>() << std::endl;
        cl::Program program(context, {device}, bins, nullptr, &err);
        if (err != CL_SUCCESS) {
            std::cout << "Failed to program device[" << i << "] with xclbin file!\n";
        } else {
            std::cout << "Device[" << i << "]: program successful!\n";
            OCL_CHECK(err, krnl_vector_add = cl::Kernel(program, "krnl_vadd", &err));
            valid_device = true;
            break; // we break because we found a valid device
        }
    }
    if (!valid_device) {
        std::cout << "Failed to program any device found, exit!\n";
        exit(EXIT_FAILURE);
    }
    cl::Buffer buffer_waveout(context, CL_MEM_WRITE_ONLY, 1024*sizeof(int));
    // These commands will allocate memory on the Device. The cl::Buffer objects can
    // be used to reference the memory locations on the device.
    OCL_CHECK(err, cl::Buffer buffer_a(context, CL_MEM_READ_ONLY, size_in_bytes, NULL, &err));
    OCL_CHECK(err, cl::Buffer buffer_b(context, CL_MEM_READ_ONLY, size_in_bytes, NULL, &err));
    OCL_CHECK(err, cl::Buffer buffer_result(context, CL_MEM_WRITE_ONLY, size_in_bytes, NULL, &err));

    // set the kernel Arguments
    int narg = 0;
    OCL_CHECK(err, err = krnl_vector_add.setArg(narg++, buffer_a));
    OCL_CHECK(err, err = krnl_vector_add.setArg(narg++, buffer_b));
    OCL_CHECK(err, err = krnl_vector_add.setArg(narg++, buffer_result));
    OCL_CHECK(err, err = krnl_vector_add.setArg(narg++, DATA_SIZE));
    krnl_vector_add.setArg(narg++,buffer_waveout);
    int *ptr_waveout = (int *) q.enqueueMapBuffer (buffer_waveout , CL_TRUE , CL_MAP_READ , 0, 1024*sizeof(int));
    // We then need to map our OpenCL buffers to get the pointers
    int* ptr_a;
    int* ptr_b;
    int* ptr_result;
    OCL_CHECK(err,
              ptr_a = (int*)q.enqueueMapBuffer(buffer_a, CL_TRUE, CL_MAP_WRITE, 0, size_in_bytes, NULL, NULL, &err));
    OCL_CHECK(err,
              ptr_b = (int*)q.enqueueMapBuffer(buffer_b, CL_TRUE, CL_MAP_WRITE, 0, size_in_bytes, NULL, NULL, &err));
    OCL_CHECK(err, ptr_result = (int*)q.enqueueMapBuffer(buffer_result, CL_TRUE, CL_MAP_READ, 0, size_in_bytes, NULL,
                                                         NULL, &err));

    // Initialize the vectors used in the test
    for (int i = 0; i < DATA_SIZE; i++) {
        ptr_a[i] = rand() % DATA_SIZE;
        ptr_b[i] = rand() % DATA_SIZE;
    }

    // Data will be migrated to kernel space
    OCL_CHECK(err, err = q.enqueueMigrateMemObjects({buffer_a, buffer_b}, 0 /* 0 means from host*/));

    // Launch the Kernel
    OCL_CHECK(err, err = q.enqueueTask(krnl_vector_add));

    // The result of the previous kernel execution will need to be retrieved in
    // order to view the results. This call will transfer the data from FPGA to
    // source_results vector
    OCL_CHECK(err, q.enqueueMigrateMemObjects({buffer_result}, CL_MIGRATE_MEM_OBJECT_HOST));
    q.enqueueMigrateMemObjects({buffer_waveout},CL_MIGRATE_MEM_OBJECT_HOST);


    OCL_CHECK(err, q.finish());
    FILE *fp_dout;
    fp_dout=fopen("wave_out.txt","w");

    for (int i = 0; i < 1024; i++) {
    fprintf(fp_dout,"%d\n",ptr_waveout[i]<<17);
    }

    fclose(fp_dout);
    // Verify the result
    int match = 0;
    for (int i = 0; i < DATA_SIZE; i++) {
        int host_result = ptr_a[i] + ptr_b[i];
        if (ptr_result[i] != host_result) {
            printf(error_message.c_str(), i, host_result, ptr_result[i]);
            match = 1;
            break;
        }
    }

    OCL_CHECK(err, err = q.enqueueUnmapMemObject(buffer_a, ptr_a));
    OCL_CHECK(err, err = q.enqueueUnmapMemObject(buffer_b, ptr_b));
    OCL_CHECK(err, err = q.enqueueUnmapMemObject(buffer_result, ptr_result));
    OCL_CHECK(err, err = q.finish());

    std::cout << "TEST " << (match ? "FAILED" : "PASSED") << std::endl;
    return (match ? EXIT_FAILURE : EXIT_SUCCESS);
}

We are then in a position where we are able to build the project. This will take a few minutes but following completion again we will have a completed SD Image which contains everything we need to test the application.

If we also desire we can open the created Vivado project, examining this project will show the original platform created with the additional design elements included. This includes the two HLS Kernels, one of which was defined by the low pass filter and the second the original and expanded kernel addition.

Wrap Up

This project has shown how we can create a custom Vitis platform for a MPSoC based board. We can then update this platform to include external interfacing logic as in this case with the addition of the Sysmon. For an image processing system this could be a image sensor MIPI input with the image processing pipeline completed by leveraging the Vitis Vision Libraries.

This project has also clearly demonstrated how we are able to connect HLS or existing RTL kernels which use AXI Stream interfaces into the desired processing pipeline.

While the project has used a Avnet ZUBoard as an example development board. This project can be recreated using any board which has uses a Zynq, Zynq MPSoC or Versal device.

Using Vitis Unified in the heterogeneous system development flow enables developers to define a common platform with interfaces connected. The functionality can then be implemented with a much reduced development time schedule by leveraging HLS and Vitis Acceleration Libraries.