AI at the Edge: Xen Hypervisor on the ZCU102

Virtualization is the concept of abstracting hardware access in a system to be able to isolate the software that wants to run on that hardware and run it as a virtual machine (VM). When a software stack is run as a VM, it believes that it is the only software running on the system and a hypervisor controls what peripherals it can access and when. This isolates the software stack in a way that facilitates better redundancy, security, and over parallel processing in the system since each software stack is running simultaneously.

The hypervisor is an additional layer of low level software (kernel level) that runs at the processor privileged level EL2. The hypervisor itself is very small and can access all of the MPSoC's resources. It then allows for partitions for independent software stacks (Linux, bare-metal application, or RTOS) to run on virtually as a virtual machine, and the hypervisor controls which resources the software stacks can access and when.

The hypervisor's job is to essentially act as a hardware kernel by managing the workload and availability of system resources, loading/booting each of the software stacks in their respect domains, and facilitate any needed communication between them.

This means a design running the hypervisor needs at least two kernels: the hypervisor kernel, the kernel for the software stack that is the master domain (Dom0) kernel with the hooks for the hypervisor, and the kernels for any other user domains (DomU) that will be running on the system.

For this project, the master domain will be Linux, so looking at the boot sequence it is the same for in that the first stage bootloader (FSBL) is kicked off initially to load the hardware bitstream onto the FPGA, then passes off to the second stage bootloader, u-boot. However, instead of u-boot calling the normal Linux kernel to boot, it calls the hypervisor's kernel. The hypervisor kernel is then what calls the Linux kernel for the master domain to boot first, then the master domain launches any other user domains' kernels.

The ZCU102 board has prebuilt images such as board support packages (BSPs) available in Vivado, Vitis, and PetaLinux as a starting point for a user to immediately jump into their own design. This provides a few different avenues to choose from for this project.

If there is any custom logic that you would like to the design on the ZCU102, you'll need to create a Vivado project targeting the ZCU102 board (otherwise you can skip to the PetaLinux steps in this project):

Create a new block design and add the Zynq MPSoC IP block and the desired custom logic (I threw in the a GPIO block for the LEDs on the board for demonstration purposes here):

Run validation on the block design and save it, then create an HDL wrapper for it. Select the option to allow for Vivado to manage and auto-update it.

Since I'm focusing on just getting the VMs running on the Zynq processing core and not working on interfacing with any of the external peripherals not immediately available through the MIO, I'm not going to generate a constraints file for this project.

Run synthesis, implementation, and generate a bitstream. Once the bitstream is generated, select the option to Export Hardware under the File > Export... menu.

Vivado will automatically select a file location to export it to within the current working directory. I personally like to leave this as where I export the hardware platform to, but you can select any folder you desire (that's not in a mapped drive).

Since the master domain for this project is Linux, the next step is in PetaLinux, open a new terminal window and source the PetaLinux tools:

source <PetaLinux installation directory>/Xilinx/PetaLinux/2019.2/settings.sh

Since PetaLinux also has a BSP for the ZCU102, you can create the PetaLinux project using that if you didn't have any custom logic to add (the BSP includes a prebuilt FPGA bitstream/hardware platform):

petalinux-create -t project -s <path to ZCU102 BSP>

Otherwise, create a new blank PetaLinux project:

petalinux-create --type project --template zynqMP --name <project name>

And import your custom hardware platform from Vivado:

petalinux-config --get-hw-description /<path to hardware platform .XSA file>

In the hardware configuration editor (accessed by the following command):

petalinux-config

navigate to the Image Packaging Configuration menu and set the root filesystem type to INITRD. Initial RAMdisk filesystem type is necessary in the way that it loads the temporary root filesystem into memory during the boot process in preparation for the full root filesystem to be mounted and thus provides the intermediate stages for the hypervisor to step in.

The device tree will be located on the SD card in this project so the project needs to be configured to look there for it. Navigate to dtb image settings under Subsystem AUTO Hardware Settings > Advanced bootable images storage settings and set it to primary sd.

While the Xen hypervisor hooks are enabled in the kernel by default, the hooks in the root file system need to be enabled manually. Launch the root filesystem configuration editor:

petalinux-config -c rootfs

Under the PetaLinux Package Groups menu, navigate to packagegroup-petalinux-xen and select the first option:

With the hardware settings configured, the next thing that needs to be modified is the device tree to include the hardware nodes for the hypervisor.

If you chose to not use the BSP for the ZCU102 in PetaLinux, you'll have to create the Xen device tree file xen.dtsi in the <project path>/meta-user/recipes-bsp/device-tree/files/ directory manually. Add the following nodes to it specifying the ZCU102 hardware for the hypervisor:

/ {
    chosen {
        #address-cells = <2>;
        #size-cells = <1>;
        xen,xen-bootargs = "console=dtuart dtuart=serial0 dom0_mem=1G bootscrub=0 maxcpus=1 timer_slop=0";
        xen,dom0-bootargs = "console=hvc0 earlycon=xen earlyprintk=xen maxcpus=1 clk_ignore_unused";

        dom0 {
            compatible = "xen,linux-zimage", "xen,multiboot-module";
            reg = <0x0 0x80000 0x3100000>;
        };
    };
};

&smmu {
    status = "okay";
    mmu-masters = < &gem0 0x874
    /* These are the hardware nodes specific to the ZCU102 */
    &gem1 0x875
    &gem2 0x876
    &gem3 0x877
    &dwc3_0 0x860
    &dwc3_1 0x861
    &qspi 0x873
    &lpd_dma_chan1 0x868
    &lpd_dma_chan2 0x869
    &lpd_dma_chan3 0x86a    
    &lpd_dma_chan4 0x86b    
    &lpd_dma_chan5 0x86c
    &lpd_dma_chan6 0x86d
    &lpd_dma_chan7 0x86e
    &lpd_dma_chan8 0x86f
    &fpd_dma_chan1 0x14e8
    &fpd_dma_chan2 0x14e9
    &fpd_dma_chan3 0x14ea
    &fpd_dma_chan4 0x14eb
    &fpd_dma_chan5 0x14ec
    &fpd_dma_chan6 0x14ed
    &fpd_dma_chan7 0x14ee
    &fpd_dma_chan8 0x14ef
    &sdhci0 0x870
    &sdhci1 0x871
    &nand0 0x872>;
};

Then add the hypervisor's device tree source to the user configurable device tree by editing the file <project path>/meta-user/recipes-bsp/device-tree/files/system-user.dtsi to include the Xen device tree file:

/include/ "system-conf.dtsi"
/include/ "xen.dtsi"
/ { 
};

In <project path>/meta-user/recipes-bsp/device-tree/files/device-tree.bbappend modify it to add the Xen device tree include file to the DTS compiler run when the PetaLinux project builds:

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += "file://system-user.dtsi"
SRC_URI += "file://xen.dtsi"

Build the PetaLinux project to generate the kernel image and the root filesytem for the next step:

petalinux-build

The master domain is responsible for booting the user/guest domains. The user/guest domain needs a copy of the kernel and root filesystem to boot from so to make those files readily avaible to the master domain without having to jump through the hoops of establishing a network connection to download them from elsewhere, a new package can be added to the PetaLinux project to place a copy of the kernel image and the packaged root filesystem in the root directory of the root filesystem itself.

In the <PetaLinux project path>/project-spec/meta-user/ directory, create a new recipe directory. The name of the top level recipe folder follows the format "recipe-<name>", and should be all lowercase with no special characters (this includes underscores):

Within the top level recipe folder, create the component folder(s), again the name should be all lowercase with no special characters (this includes underscores):

Then within the component folder, create a "files" folder and the bitbake recipe (.bb file) to specify how to build this package/recipe will be built into the root filesystem.

In the domu-linux.bb bitbake file, add the following:

SUMMARY = "linux image files for guest domain"
DESCRIPTION = "linux image files for guest domain"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

SRC_URI += "file://rootfs.cpio \
	    file://image.ub \
            file://Image \
"


do_install() {
	install -d ${D}/home/root/
	install -m 0755 ${WORKDIR}/rootfs.cpio ${D}/home/root/
	cp -r ${WORKDIR}/image.ub ${D}/home/root/
	cp -r ${WORKDIR}/Image ${D}/home/root/
}

FILES_${PN} = " \
		/home/root/rootfs.cpio \		
		/home/root/image.ub \		
		/home/root/Image \
"

This tells the recipe to simply place the files in the root directory of the main user in the root filesytem.

Copy the kernel and package root file system from the <PetaLinux project directory>/images/linux directory into the files folder as referenced by the bitbake recipe:

Edit the user-rootfsconfig file in the <PetaLinux project path>/project-spec/meta-user/conf directory to add the following line:

CONFIG_domu-linux

This will make this new recipe visible to the root filesystem configuration editor, but first the package needs to be built:

petalinux-build -c domu-linux

Then launch the root filesystem configuration editor:

petalinux-config -c rootfs

Under the user packages tab, the new package is now an option to be included in the next build of the root filesystem. Enable it, then save and close the root filesystem configuration editor:

Build the root filesystem:

petalinux-build -c rootfs

And finally, build the PetaLinux project one more time:

petalinux-build

Then create the boot binary file (BOOT.BIN) for Dom0 containing the FSBL, FPGA bitstream, and u-boot:

petalinux-package --boot --fsbl <project path>/images/linux/zynqmp_fsbl.elf --fpga <project path>/images/linux/system.bit --u-boot --kernel

Since the ZCU102 is being boot from an SD card for this project (Linux images are too large to fit on the onboard QSPI flash memory), the following components are needed:

• The boot binary image BOOT.BIN
• The Linux kernel image image.ub and Image
• The hypervisor kernel image xen.ub
• The system device tree with the hypervisor hooks added from the previous steps system.dtb
• The RAM disk image rootfs.cpio.gz.u-boot

After the PetaLinux project has finished building, these files can be found in the <project path>/images/linux directory.

Prepare the SD card for the ZCU102 by creating two partitions on it. The first needs to be a FAT32 at least 500MB in size with 4MB of free space preceding it, and the second partition needs to be EXT4 and take up the rest of the space on the SD card (at least 4GB is recommended).

Create mounting points for the SD card on your host system:

sudo mkdir /media/BOOT/
sudo mkdir /media/rootfs

sudo mount /dev/sdc1 /media/BOOT/
sudo mount /dev/sdc2 /media/rootfs

Then copy BOOT.BIN, system.dtb, rootfs.cpio.gz.u-boot, image.ub, and xen.ub to the FAT32 partition:

sudo cp /<petalinux project dir>/images/linux/BOOT.BIN /media/BOOT/
sudo cp /<petalinux project dir>/images/linux/image.ub /media/BOOT/
sudo cp /<petalinux project dir>/images/linux/Image /media/BOOT/
sudo cp /<petalinux project dir>/images/linux/xen.ub /media/BOOT/
sudo cp /<petalinux project dir>/images/linux/rootfs.cpio.gz.u-boot /media/BOOT/
sudo cp /<petalinux project dir>/images/linux/system.dtb /media/BOOT/

And unpack the root filesystem onto the EXT4 partition:

sudo cp /<petalinux project dir>/images/linux/rootfs.cpio /media/rootfs
sudo cpio -idv < ./rootfs.cpio

Unmount the SD card from the system prior to removing it from your host system:

sudo umount /media/BOOT/
sudo umount /media/rootfs/

To boot the ZCU102 from the SD card change the boot mode switch (SW6) to the SD configuration:

Connect a USB cable from the USB UART port of the ZCU102 and open it on your host machine with the serial terminal application of your choice (Putty, TeraTerm, etc). Install the SD card in the ZCU102 and power it on using the main power switch SW1.

To boot the hypervisor from the SD card press any key during initial booting to halt the autoboot process to access the u-boot environment.

Change the u-boot environment to run the hypervisor from the partition ID set in the Xen hypervisor's device tree and then load the Xen kernel and the master domain Linux kernel. Enter the following command and press return to execute this new boot sequence:

mmc dev $sdbootdev && mmcinfo && run uenvboot || run sdroot$sdbootdev; load mmc $sdbootdev:$partid $fdt_addr system.dtb && load mmc $sdbootdev:$partid 0x80000 Image; fdt addr $fdt_addr && fdt resize 128 && fdt set /chosen \#address-cells <1> && fdt set /chosen \#size-cells <1> && fdt mknod /chosen dom0 && fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module" && fdt set /chosen/dom0 reg <0x80000 0x3100000> && fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 dom0_mem=768M bootscrub=0 maxcpus=1 timer_slop=0" && fdt set /chosen xen,dom0-bootargs "console=hvc0 earlycon=xen earlyprintk=xen maxcpus=1 clk_ignore_unused root=/dev/mmcblk0p2 rootwait"; load mmc $sdbootdev:$partid 1000000 xen.ub; bootm 1000000 - $fdt_addr

Once booted up, login with the default password 'root' (unless you changed it in the root filesystem configuration editor).

Now that the master domain (Dom0) is running, subsequent user domains can be booted from this point. The PetaLinux kernel CONFIG_HVC_DRIVER already includes drivers for the hypervisor console within the guest OS (user domain) so that a separate serial console doesn't have to be set up within the guest OS for the hypervisor. This console for the hypervisor is input/output point to point between the user domain (DomU) and the master domain (Dom0). This means a new console for every new user domain must be instantiated.

In this case the user domain guest OS is also a Linux image, so this Linux image (kernel and root filesystem) needs to be copied into the master domain's filesystem. Since these files were included in the PetaLinux build as a recipe they are already present in the root directory of the filesystem on the master domain. So the next step is to extract the root filesystem to a location the user domain can mount to.

One the ZCU102, create a blank 1GB image file to be used as the guest/user domain's file system.

sudo dd if=/dev/zero of=./linuxdomu.img bs=1M count=1024

Then partition the image to use the entire image file as a Linux partition

sudo sh -c 'echo -e "n\np\n1\n\n\nt\n83\nw\n" | fdisk ./linuxdomu.img'

Make a temporary mounting directory for the guest domain's file system and connect it to a loop device:

mkdir -p ./guest0_rootfs
losetup -o 1048576 /dev/loop1 ./linuxdomu.img
mkfs.ext4 /dev/loop1
mount /dev/loop1 ./guest0_rootfs

Copy the packaged root filesystem into the directory and unpack it:

cp ./rootfs.cpio ./guest0_rootfs
cd ./guest0_rootfs
cpio -idv < ./rootfs.cpio

Unmount and detach the guest domain's filesystem from the master domain's filesystem:

umount /dev/loop1
losetup -d /dev/loop1

Finally, create the guest configuration file in the master domain for the user domain (petalinux-guest.cfg):

nano /petalinux-guest.cfg

And add the following:

name = "linuxguest0"
kernel = "/Image"
extra = "console=hvc0 earlyprintk=xenboot root=/dev/xvda1 rw"
memory = 128
vcpus = 1
disk = [ 'disk:/linuxdomu.img,xvda,w' ]
cpus = [0]

Save and close the guest configuration then finally, boot the guest domain (DomU0) using the xl command:

xl create -c petalinux-guest.cfg

Verify that both domains are running with the list command:

xl list

If the guest domain happens to come up in a paused state for some reason, unpause it and connect to its console:

xl unpause linuxguest0
xl console linuxguest0

Keep in mind that in certain applications, there can be a performance penalty of up to 5% within the guest domains.

If you want to shut down the guest domain at any time run:

xl shutdown linuxguest0

And if a guest domain crashes and/or becomes completely unresponsive, you can kill it by running the following:

xl destroy linuxguest0

You can follow the same process starting back at the step of creating a blank 1GB image file to be used as the guest/user domain's file system to create subsequent user domains (just be sure to increment names accordingly).