TE0726 Zynqberry Software Design in Vitis 2022.1

In my previous project post, I went over how to create a basic hardware design to utilize the Zynqberry's USB ports, Ethernet, and 40-pin GPIO header to make it compatible with Raspberry Pi pHATs. This base hardware design is compatible with both the dual-core Zynq versions and single-core Zynq versions of the Zynqberry (given the correct one is declared as the Vivado project target part number).

As a continuation, this project walks through how to create the software applications to run on those ARM processors of the Zynq-7000 SoC in the Zynqberry. I thought it'd be interesting to demonstrate both how to create a single bare-metal application to run on the main ARM-core of the dual-core Zynqberry (which is the same process as creating a bare-metal application to run on the only ARM-core of the single-core Zynqberry) as well as create and run a second bare-metal application on the other ARM-core of the dual-core Zynqberry at the same time.

Note: I'm using Vitis 2022.1 in this project. The following steps should be applicable back to 2021.1 and up to 2022.2, but I have not explicitly tested it as such.

Create Vitis Workspace

Launch the Vitis IDE and select a target directory to use as a workspace.

I personally like to create my Vitis workspaces in the top level directory of the corresponding Vivado project that I exported the XSA hardware definition from.

Create Platform Project

Once the blank workspace generates, a platform project needs to be created based on the exported XSA from Vivado so the Vitis workspace is aware of the hardware available in the bitstream for the Zynqberry.

Select Create Platform Project, give it the desired name, and point it to the exported XSA from Vivado:

Be sure the processor targeted is ARM-core 0, ps7_cortexa9_0, as it is the main one (will explain later on).

Once the platform project generates, it will immediately show as out-of-date since there are no build output files. Run a quick build on it initially (ctrl+B):

Application Project for ARM-Core 0

With the platform project created to provide the board support package (BSP) and boot components for the ARM processor, the next step is to create an application project for that ARM processor (finally, getting to where the actual source code is).

Select New > Application Project...

Walk through the windows in the New Application Project wizard, first selecting the platform generated by the platform project in the previous steps.

On the next pages, give the desired name to the application and leave the defaults set to create a new system project and target ARM-core 0.

Finally, select the desired application template to use. I'm just going with the Hello World template for now.

Click Finish, and wait for the application to be generated in the Explorer window. The main function is located in ./src/helloworld.c to start adding your own custom code.

If you're using the single-core version of the Zynqberry or only want to run application code on one of the cores in the dual-core, you can carry on as normal for writing the application past what's available in the template, debugging it, etc.

The following steps outline how to configure the Zynq chip to run application code on both of the ARM-cores of the dual-core version of the Zynqberry at the same time.

Add Domain for ARM-Core 1

Each processor is targeted by its own domain in Vitis, so a second domain needs to be created for the second ARM-core.

Open the platform.xpr file from the Explorer window, right-click on the name of the platform project with the green symbol next to it in the window that opens (not in the Explorer menu - the only option that will appear is Add Domain):

Give the new domain a desired name, then select standalone or Linux for the OS (I chose standalone as I'm creating a bare-metal application). Finally, select the second ARM core (ps7_cortexa9_1) as the processor.

Now that the platform project now has the hooks to target the second ARM-core, it needs to be re-built (ctrl+B):

Application Project for ARM-Core 1

With the domain created for ARM-Core 1, a new application project can be created in it.

Create a new application project by selecting New > Application Project..., again targeting our same platform project:

But on the second page, after naming the application project, be sure to select the same system project as before but the second ARM processor (ps7_cortexa9_1) instead. The third page will auto-fill accordingly.

For demonstration purposes, I chose the Hello World template again.

Once the second application project generates, you'll notice it shows up as a single system with two applications:

Update Linker Scripts

Even though we created separate domains for each of the ARM-cores and targeted the application projects to each of them accordingly, Vitis still defaults to generating linker script to allocate the full amount of DDR to each. Which needless to say can and will lead to collisions.

To modify the DDR addressed to allocate one part to ARM-core 0 and the other to ARM-core 1, open the linker script for each application (located in the Explorer > application > src > lscript.ld) and change the values for ps7_ddr_0 at the top.

Allocate the lower half of the Zynqberry's DDR to ARM-core 0 and the upper half to ARM-core 1. Leave ARM-core 0's base address at its default value (0x100000) and cut the size down by half to 0xFF800000. Then set ARM-core 1's base address to ARM-core 0's base address plus 0xFF800000 (0x10080000) and it's size is also set to 0xFF800000.

Note: since these are bare-metal applications doing almost nothing but printing a single line to UART, they absolutely do not need anywhere near this much DDR to operate.

Set the AMP Compiler Flag for ARM-Core 1

While both of the ARM cores have the same processing power, there are still factors such as the boot process that require one of the cores to act as the master and the other as the slave.

In the Zynq-7000 ARM-core 0 is the master, thus there are certain operations and peripherals only it will have access to. ARM-core 0 is also responsible for booting up the "slave" ARM-core 1 at the appropriate point in time.

In order for the application on ARM-core 1 to be aware of the fact that it is operating on the slave processor and be provided the appropriate hooks, the Asymmetric Multi Processing (AMP) compiler flag must be set in its BSP.

Return to the platform.xpr file again, navigate to the domain for ps7_cortexa9_1, select the Board Support Package tab, and select Modify BSP settings...

Navigate to ps7_cortexa9_1 > extra_compiler_flags, then add the following flag to the end of the arguments already there:

-DUSE_AMP=1

Click OK then rebuild the platform project (ctrl+B).

Application Code for Dual-Core

Moving on to writing the actual application code that will run on each of the ARM-cores, there are a few more things to take note up with the multi-core setup.

Since ARM-core 0 is responsible for booting up ARM-core 1, there are main two steps it must follow to successfully boot the second ARM-core (per UG585 section 6.1.10):

• Write the memory space base address in the Zynq's DDR (PS7 DDR) for ARM core 1 to 0xFFFFFFF0 (which is 0x10080000 in this project).
• Execute the SEV instruction (an alert to all cores within a multiprocessor system) to wake up ARM core 1 and cause it to jump into its application.

It's also important to disable the cache in the OCM when running multiple processors at once since it's not a shareable resource between the processors.

For safety, executing the ARM data memory barrier instruction right after writing ARM1's base address to 0xFFFFFFF0 is a good practice as it won't allow the processor to move on until the write instruction is completed and the memory space appears as updated.

Finally, the SEV instruction can be executed which acts like a beacon alert across the system to tell all processors present to wake up and jump into their applications.

One more thing: I wanted the UART console to show the "Hello World" print outs from ARM-core 0 and ARM-core 1 in an alternating pattern. So instead of trying to hardcore the sleep times to accomplish that, I used a variable typecast into the shared memory space at 0xFFFF0000 as something each ARM-core could poll the value of to know when the other had finished its print statement.

The resulting code for ARM-core 0:

#include <stdio.h>
#include <sleep.h>
#include "xil_io.h"
#include "xil_mmu.h"
#include "platform.h"
#include "xil_printf.h"
#include "xpseudo_asm.h"
#include "xil_exception.h"

#define sev() __asm__("sev")
#define ARM1_STARTADR 0xFFFFFFF0
#define ARM1_BASEADDR 0x10080000
#define COMM_VAL (*(volatile unsigned long *)(0xFFFF0000))

int main()
{
    init_platform();
    COMM_VAL = 0;

    //Disable cache on OCM    
    // S=b1 TEX=b100 AP=b11, Domain=b1111, C=b0, B=b0
    Xil_SetTlbAttributes(0xFFFF0000,0x14de2); 

    print("ARM0: writing startaddress for ARM1\n\r");
    Xil_Out32(ARM1_STARTADR, ARM1_BASEADDR);
    dmb(); //waits until write has finished

    print("ARM0: sending the SEV to wake up ARM1\n\r");
    sev();

    while(1){
        print("Hello ARM0!\n\r");
        sleep(1);
        COMM_VAL = 1;
        while(COMM_VAL == 1){
        }
    }

    cleanup_platform();
    return 0;
}

The resulting code for ARM-core 1:

#include <stdio.h>
#include <sleep.h>
#include "xil_io.h"
#include "xil_mmu.h"
#include "platform.h"
#include "xil_cache.h"
#include "xil_printf.h"
#include "xparameters.h"
#include "xpseudo_asm.h"
#include "xil_exception.h"

#define COMM_VAL (*(volatile unsigned long *)(0xFFFF0000))
extern u32 MMUTable;

int main()
{
    init_platform();
    print("CPU1: init_platform\n\r");

    //Disable cache on OCM
    // S=b1 TEX=b100 AP=b11, Domain=b1111, C=b0, B=b0
    Xil_SetTlbAttributes(0xFFFF0000,0x14de2);  

    while(1){
        while(COMM_VAL == 0){
        };

        print("Hello ARM1!\n\r");
        sleep(1);
        COMM_VAL = 0;
    }

    cleanup_platform();
    return 0;
}

Note: The function init_platform() needs to be left in both applications at it is responsible for initializing the UART console for each application.

Build the entire workspace by right-clicking <system project name>_system and selecting Build Project.

Create Boot Image

To create a boot image for the Zynqberry that includes the ELF files for both of the ARMs' applications, again right-click on <system project name>_system and select Create Boot Image.

The wizard will auto-fill all of the appropriate settings so just click Create Image:

Unfortunately, I still haven't found a work-around for programming the QSPI flash of my Zynqberry boards in Vitis versons 2021.x and later so my previous workaround is still the only solution I have.