Hardware components | ||||||
 |
| × | 1 | |||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
This project is part 2 of a 4 part series of projects, where we will progressively create a Vitis-AI and ROS2 enabled platform for ZUBoard:
- Part 1 : Building the foundational designs
- Part 2 : Combining designs into a common platform
- Part 3 : Adding support for Vitis-AI
- Part 4 : Adding support for ROS2
The motivation of this series of projects is to enable users to create their own custom AI applications.
We will also be hosting a webinar series on creating a fun robotics application:
- Webinar I : Teaching the ZUBoard to recognize hand gestures
- Webinar II : Controlling a robot with the ZUBoard
In the previous project ( http://avnet.me/zub1cg-sbc-dualcam-2022.2 ), we learned how to build the foundational designs for ZUBoard:
- zub1cg_sbc_base
- zub1cg_sbc_dualcam
We built an SD image for each design and learned to use each design. Dealing with multiple SD images, however, can be cumbersome.
This project describes how to combine multiple designs into a single platform. This will be accomplished with the "xmutil" platform management utility from AMD.
AMD (Xilinx) introduced XMUTIL with the Kria portfolio. The utility can be used to query the platform status, and manage the accelerated apps, as well as several other features.
Of interest to us in this project are the following commands:
- xmutil listapps
- xmutil loadapp {app}
- xmutil unloadapp
These commands call DFX-MGR under the hood, which is AMD (Xilinx)'s implementation of the open source FPGA MANAGER service in linux.
The following table lists which dfx-mgr-client commands are called by the xmutil utility.
It allows the user to dynamically load hardware designs as full bitstreams or as partial reconfiguration, along with the device tree content that describes this hardware.
Creating the new platformThere are many ways to go about combining the "zub1cg-sbc-base" and "zub1cg-sbc-dualcam" designs into a single "zub1cg-sbc" platform.
I have chosen to do this by combining the petalinux project of the "zub1cg-sbc-dualcam" design with the Vivado project of the "zub1cg-sbc-base" design. The reasons being the following:
- the dualcam petalinux project includes the additional drivers for the MIPI capture pipeline, and is thus a super-set of features that will support both designs
- the base Vivado project has an (almost) empty PL, and is thus easier to remove from the default (empty) device tree content
In order to reuse the petalinux project, we first want to package up the modifications we made (ie. adding jupyter functionality) into a BSP using a meaningful name (ie. zub1cg_sbc_dualcam_2022_2_jupyter.bsp), as follows:
$ cd ~/Avnet_2022_2/petalinux/projects
$ petalinux-package --bsp -p zub1cg_sbc_dualcam_2022_2 --output zub1cg_sbc_dualcam_2022_2_jupyter.bspNext, we want to create a new petalinux project for our new design:
$ cd ~/Avnet_2022_2/petalinux/projects
$ petalinux-create -t project -s zub1cg_sbc_dualcam_2022_2_jupyter.bsp -n zub1cg_sbc_2022_2We want to rename the design name of our new petalinux project, by modifying the following file:
~/Avnet_2022_2/petalinux/projects/zub1cg_sbc_2022_2/project-spec/configs/config
...
#
# Firmware Version Configuration
#
CONFIG_SUBSYSTEM_HOSTNAME="zub1cg-sbc-2022-2"
CONFIG_SUBSYSTEM_PRODUCT="zub1cg_sbc_2022_2"
CONFIG_SUBSYSTEM_FW_VERSION="1.00"
...
#
# Yocto Settings
#
CONFIG_YOCTO_MACHINE_NAME="zub1cg-sbc-base"
...Notice that we set the yocto machine name to "zub1cg-sbc-base".
My first intuition to create the combined petalinux project using the "base" Vivado project was incorrect, since the dualcam needs the PS I2C0 peripheral to be active. The solution to this is to use the "dualcam" Vivado project instead, which has this PS I2C0 peripheral enabled.
Finally, we configure the petalinux project to use the "dualcam" Vivado project, and attempt our first build:
$ cd ~/Avnet_2022_2/petalinux/projects/zub1cg_sbc_2022_2
$ petalinux-config --silentconfig --get-hw-description=../../../hdl/projects/zub1cg_sbc_dualcam_2022_2
$ petalinux-buildBy default, we have our device tree defined for the MIPI capture pipeline.
The next section will purge the device tree.
Purging the Device TreeSince we will be dynamically loading our PL designs, we want to remove all traces of the base and dualcam designs from the default (empty) device tree, and eventually move these to dynamic device tree definitions to each design's firmware overlay.
The first thing to do is configure the project to NOT generate device tree content for the PL. This can be done with the petalinux-config command or in the following file:
~/Avnet_2022_2/petalinux/projects/zub1cg_sbc_2022_2/project-spec/configs/config
...
CONFIG_SUBSYSTEM_REMOVE_PL_DTB=y
...For the ZUBoard's dualcam design, additional device tree content is defined in the following file:
~/Avnet_2022_2/petalinux/projects/zub1cg_sbc_2022_2/project-spec/meta-on-semiconductor/recipes-bsp/device-tree/files/zub1cg-sbc-dualcam/system-user.dtsi
Our new petalinux design is defaulting to the dualcam design's device tree, so we can remove this device tree content:
$ rm -r project-spec/meta-on-semiconductor/recipes-bsp/device-treeThere is also device content in the following file:
~/Avnet_2022_2/petalinux/projects/zub1cg_sbc_2022_2/project-spec/meta-avnet/recipes-bsp/device-tree/files/zub1cg-sbc/system-bsp.dtsi
We want to remove the content that makes reference to peripherals in the PL. This can be done be commenting out the PL definitions (axi_iic_0) as follows:
~/Avnet_2022_2/petalinux/projects/zub1cg_sbc_2022_2/project-spec/meta-avnet/recipes-bsp/device-tree/files/zub1cg-sbc/system-bsp.dtsi
/*
&axi_iic_0 {
#address-cells = <1>;
#size-cells = <0>;
clock-frequency = <100000>;
stts22htr: stts22htr@3f {
compatible = "st,stts22h";
reg = <0x3f>;
};
};
*/We can rebuild the petalinux project (again):
$ petalinux-buildNow that we have our petalinux project set up as a "clean" slate, we can add our two (2) designs as firmware overlays, which we will name:
- {vendor}_{platform}_{design}
- avnet_zub1cg_base
- avnet_zub1cg_dualcam
Petalinux provides a command to create a yocto recipe for these firmware overlays:
$ petalinux-create -t apps
--template fpgamanager -n {firmware}
--enable
--srcuri"{path}/{firmware}.bit
{path}/{firmware}.dtsi
{path}/{firmware}.xclbin
{path}/shell.json"
--forceBefore using this command, however, we need to setup the files required for our firmware.
The bitstream, {firmware}.bit, can be found in the ~/Avnet_2022_2/hdl/projects/ directories.
The dynamic device tree, {firmware}.dtsi, will have to be created from existing content.
The vitis container, {firmware}.xclbin, is not required for these designs, but will be covered in the next project when we add Vitis-AI support to our platform.
The shell.json is a simple xml file describing the overlay type.
We start by copying the.bit and creating the shell.json files for the base design:
$ mkdir -p firmware/avnet_zub1cg_base
$ cp ../../../hdl/projects/zub1cg_sbc_base_2022_2/zub1cg_sbc_base.runs/impl_1/zub1cg_sbc_base_wrapper.bit firmware/avnet_zub1cg_base/avnet_zub1cg_base.bit
$ echo '{ "shell_type":"XRT_FLAT", "num_slots":1 }' > firmware/avnet_zub1cg_base/shell.jsonThen we copy the.bit and create shell.json files for the dualcam design:
$ mkdir -p firmware/avnet_zub1cg_dualcam
$ cp ../../../hdl/projects/zub1cg_sbc_dualcam_2022_2/zub1cg_sbc_dualcam.runs/impl_1/zub1cg_sbc_dualcam_wrapper.bit firmware/avnet_zub1cg_dualcam/avnet_zub1cg_dualcam.bit
$ echo '{ "shell_type":"XRT_FLAT", "num_slots":1 }' > firmware/avnet_zub1cg_dualcam/shell.jsonNow we need to create the device tree content (.dtsi) for our designs.
Use the following content as a starting point.
avnet_zub1cg_base.dtsi
...
/dts-v1/;
/plugin/;
#include <dt-bindings/gpio/gpio.h>
&fpga_full {
#address-cells = <2>;
#size-cells = <2>;
firmware-name = "avnet_zub1cg_base.bit.bin";
resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
}
&amba {
afi0: afi0 {
compatible = "xlnx,afi-fpga";
config-afi = <0 0>, <1 0>, <2 0>, <3 0>, <4 1>, <5 1>, <6 0>, <7 0>, <8 1>, <9 1>, <10 2>, <11 2>, <12 2>, <13 2>, <14 0x000>, <15 0x000>;
resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
reset-names = "pl0", "pl1", "pl2", "pl3";
};
};
&amba {
/* copy content of zub1cg_sbc_base_2022_2/components/plnx_workspace/device-tree/device-tree/pl.dtsi here */
}avnet_zub1cg_dualcam.dtsi
/dts-v1/;
/plugin/;
#include <dt-bindings/gpio/gpio.h>
&fpga_full {
#address-cells = <2>;
#size-cells = <2>;
firmware-name = "avnet_zub1cg_dualcam.bit.bin";
resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
}
&amba {
afi0: afi0 {
compatible = "xlnx,afi-fpga";
config-afi = <0 0>, <1 0>, <2 0>, <3 0>, <4 1>, <5 1>, <6 0>, <7 0>, <8 1>, <9 1>, <10 2>, <11 2>, <12 2>, <13 2>, <14 0x000>, <15 0x000>;
resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
reset-names = "pl0", "pl1", "pl2", "pl3";
};
};
&amba {
/* copy content of zub1cg_sbc_dualcam_2022_2/components/plnx_workspace/device-tree/device-tree/pl.dtsi here */
}Now we can copy the contents of the design specific pl.dtsi from the original petalinux projects:
~/Avnet_2022_2/petalinux/projects/zub1cg_sbc_base_2022_2/components/plnx_workspace/device-tree/device-tree/pl.dtsi
~/Avnet_2022_2/petalinux/projects/zub1cg_sbc_dualcam_2022_2/components/plnx_workspace/device-tree/device-tree/pl.dtsi
For the base overlay, we can also add the content that we removed from the system-bsp.dtsi previously. This can be added at the end of the avnet-zub1cg-base.dtsi file.
&axi_iic_0 {
#address-cells = <1>;
#size-cells = <0>;
clock-frequency = <100000>;
stts22htr: stts22htr@3f {
compatible = "st,stts22h";
reg = <0x3f>;
};
};For the dualcam overlay, we need to add the additional content for the dualcam. This can also be added at the end of the avnet-zub1cg-dualcam.dtsi file:
&axi_iic_0 {
#address-cells = <1>;
#size-cells = <0>;
clock-frequency = <100000>;
stts22htr: stts22htr@3f {
compatible = "st,stts22h";
reg = <0x3f>;
};
};
&amba {
ap1302_osc: ap1302oscillator {
compatible = "fixed-clock";
#clock-cells = <0>;
clock-frequency = <48000000>;
clock-output-names = "ap1302osc";
};
};
...We can now create our firmware overlays, as follows:
$ petalinux-create -t apps \
--template fpgamanager -n avnet-zub1cg-base \
--enable \
--srcuri "firmware/avnet_zub1cg_base/avnet_zub1cg_base.bit \
firmware/avnet_zub1cg_base/avnet_zub1cg_base.dtsi \
firmware/avnet_zub1cg_base/shell.json" \
--force
$ petalinux-create -t apps \
--template fpgamanager -n avnet-zub1cg-dualcam \
--enable \
--srcuri "firmware/avnet_zub1cg_dualcam/avnet_zub1cg_dualcam.bit \
firmware/avnet_zub1cg_dualcam/avnet_zub1cg_dualcam.dtsi \
firmware/avnet_zub1cg_dualcam/shell.json" \
--forceThis will have created new entries in the user-rootfsconfig and rootfs_config configuration files. Add the "xmutil" package to these, as follows:
project-spec/meta-user/conf/user-rootfsconfig
...
CONFIG_avnet-zub1cg-base
CONFIG_avnet-zub1cg-dualcam
CONFIG_xmutil
...project-spec/configs/rootfs_config
...
#
# apps
#
CONFIG_avnet-zub1cg-base=y
CONFIG_avnet-zub1cg-dualcam=y
CONFIG_xmutil=y
...Since we now defined our firmware overlays, we want to include the packages that enable the ap1302 kernel driver and demo scripts. This can be done by adding the following at the end of the following file:
project-spec/meta-on-semiconductor/recipes-core/images/petalinux-image-minimal.bb
...
IMAGE_INSTALL:append:zub1cg-sbc-base += "\
ap1302 \
libdrm \
libdrm-tests \
libdrm-kms \
dualcam-python-examples \
"Note that we are not including the "camera-setup" package, since this does not apply to our common image. We also want to remove this "camera-setup" dependency from the "dualcam-python-examples" package, and replace it with "avnet-zub1cg-dualcam":
project-spec/meta-on-semiconductor/recipes-app/dualcam-python-examples/dualcam-python-examples.bb
...
RDEPENDS:${PN} = "python3 \
python3-numpy \
opencv \
avnet-zub1cg-dualcam \
"
...We can rebuild the petalinux project (yet again):
$ petalinux-buildIn order to verify our petalinux image and base & dualcam overlays, the best way I have found is to execute them.
In order to execute the petalinux image, we first need to program the SD card image to a micro-SD card (of size 32GB or greater).
~/Avnet_2022_2/petalinux/projects/zub1cg_sbc_2022_2/images/linux/rootfs.wic
To do this, we use Balena Etcher, which is available for most operating systems.
Once programmed, insert the micro-SD card into the ZUBoard, and connect up the platform as shown below.
Make sure that the DualCam HSIO jumpers are configured as follows:
- J1 => HSIO
- J2 => MIPI DSI
- J3 – present => VAA connected
- J4/J14 – J4.1-J4.2 => VDD is 1.2V (This needs to be set to 1-2 for 2.8V)
- J5 – 2-3 => Clock is on-board OSC(48 MHz)
- J6 => IAS0
- J7 => IAS1
- J8 – 2-3 => SENSOR1_GPIO1 is 1V8_SP2
- J9 – 2-3 => SENSOR1_GPIO3 is GND
- J10 – 2-3 => SENSOR2_GPIO1 is 1V8_SP3
- J11 – 2-3 => SENSOR2_GPIO3 is SENS2_ADDR
- J12 – absent => SENSOR1_GPIO0/FLASH
- J13 – absent => SENSOR2_GPIO0/FLASH
- J15 => MCU PROG
Press the power push-button to boot the board, and login as "root".
As linux boots, a large quantity of verbose output will be sent to the serial console, ending with the following:
...
*********************************************************************
***
*** Avnet ZUBoard 1CG Out Of Box PetaLinux Build V1.2
*** The PS LED is mapped to 334
***
*********************************************************************
[ OK ] Started Blinky Sample Application.
[FAILED] Failed to start Load Default Camera Configuration.
See 'systemctl status load-default-camera-config.service' for details.
[ OK ] Started Network Name Resolution.
[ OK ] Reached target Network.
[ OK ] Reached target Host and Network Name Lookups.
[ OK ] Started NFS status monitor for NFSv2/3 locking..
Starting Permit User Sessions...
Starting Target Communication Framework agent...
[ OK ] Started Xinetd A Powerful Replacement For Inetd.
[ OK ] Finished Permit User Sessions.
[ OK ] Started Getty on tty1.
[ OK ] Started Serial Getty on ttyPS0.
[ OK ] Reached target Login Prompts.
[ OK ] Started Target Communication Framework agent.
[ OK ] Reached target Multi-User System.
[ OK ] Reached target Graphical Interface.
Starting Record Runlevel Change in UTMP...
[ OK ] Finished Record Runlevel Change in UTMP.
PetaLinux 2022.2_release_S10071807 zub1cg-sbc-2022-2 ttyPS0Login as the "root" user as follows:
zub1cg-sbc-2022-2 login: root
root@zub1cg-sbc-2022-2:~#Notice that, by default, a "Blinky Sample Application" was launched, and will be continually blinking the D9 LED on the ZUBoard.
If we made it this far, our petalinux project is working. We can now verify pour "avnet_zub1cg_base" and "avnet_zub1cg_dualcam" overlays, which are called "apps" by xmutil.
We start by querying which "apps" are present:
root@zub1cg-sbc-2022-2:~# xmutil listapps
Accelerator Accel_type Base Base_type #slots Active_slot
avnet-zub1cg-base XRT_FLAT avnet-zub1cg-base XRT_FLAT (0+0) -1
avnet-zub1cg-dualcam XRT_FLAT avnet-zub1cg-dualcam XRT_FLAT (0+0) -1
root@zub1cg-sbc-2022-2:~#Now that we have verified our common zub1cg-sbc-2022.2 image, and the presence of our avnet-zub1cg-base and avnet-zub1cg-dualcam overlays, we can start verifying these overlays.
Verifying these overlays is an iterative process, due to the complexity of the dynamic device tree content.
Verifying the avnet-zub1cg-base overlayWe start with the simpler of the two overlays : avnet-zub1cg-base.
root@zub1cg-sbc-2022-2:~# xmutil loadapp avnet-zub1cg-base
[ 36.078078] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/firmware-name
[ 36.088209] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/resets
[ 36.098554] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/afi0
[ 36.108050] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_0
[ 36.118054] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_1
[ 36.128076] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_2
[ 36.138094] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_iic_0
[ 36.148021] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/stts22htr
[ 36.157941] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_iic_1
[ 36.167862] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_iic_2
[ 36.177785] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_intc_0
[ 36.187792] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_quad_spi_0
[ 36.198147] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_uartlite_0
[ 36.208504] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/system_management_wiz_0
[ 37.768402] xadc a0090000.system_management_wiz: IRQ index 0 not found
[ 37.810603] zocl-drm axi:zyxclmm_drm: IRQ index 32 not found
avnet-zub1cg-base: loaded to slot 0Note that the following WARNING occurs for the case of a working dynamic device tree, so can be ignored:
OF: overlay: WARNING: memory leak will occur if overlay removedWe can also verify that we can unload the base overlay:
root@zub1cg-sbc-2022-2:~# xmutil unloadapp
[ 50.960995] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/zyxclmm_drm
[ 50.976432] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/axi_quad_spi@a0070000
[ 50.992320] OF: ERROR: memory leak, expected refcount 1 instead of 226, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/interrupt-controller@a0060000
[ 51.009046] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/i2c@a0050000
[ 51.024110] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/i2c@a0040000
[ 51.039171] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/i2c@a0030000
[ 51.054223] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/gpio@a0020000
[ 51.069353] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/gpio@a0010000
[ 51.084479] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/gpio@a0000000
[ 51.099625] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/afi0
[ 51.114232] OF: ERROR: memory leak before free overlay changeset, /axi/axi_quad_spi@a0070000
[ 51.158328] OF: ERROR: memory leak before free overlay changeset, /axi/i2c@a0050000
[ 51.166120] OF: ERROR: memory leak before free overlay changeset, /axi/i2c@a0040000
[ 51.173886] OF: ERROR: memory leak before free overlay changeset, /axi/i2c@a0030000
[ 51.186943] OF: ERROR: memory leak before free overlay changeset, /axi/gpio@a0020000
[ 51.194789] OF: ERROR: memory leak before free overlay changeset, /axi/gpio@a0010000
[ 51.202632] OF: ERROR: memory leak before free overlay changeset, /axi/gpio@a0000000
[ 51.210478] OF: ERROR: memory leak before free overlay changeset, /axi/afi0
remove from slot 0 returns: 0 (Ok)Note that the following WARNING occurs for the case of a working dynamic device tree, so can be ignored:
OF: ERROR: memory leak before free overlay changeset, ...Next, we tackle the more complex of the two overlays : avnet-zub1cg-dualcam.
root@zub1cg-sbc-2022-2:~# xmutil loadapp avnet-zub1cg-dualcam
[ 454.504338] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/firmware-name
[ 454.514467] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/resets
[ 454.525094] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/afi0
[ 454.534595] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 454.547127] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_portsCAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 454.560867] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_port1CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 454.574615] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csirx_outCAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 454.588357] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_port0CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 454.602097] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_inCAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 454.615577] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/misc_clk_0
[ 454.625585] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_frmbuf_wr_0
[ 454.637332] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_csc_0
[ 454.649256] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_portsCAPTURE_PIPELINE_v_proc_ss_csc_0
[ 454.661954] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_port1CAPTURE_PIPELINE_v_proc_ss_csc_0
[ 454.674655] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_outCAPTURE_PIPELINE_v_proc_ss_csc_0
[ 454.687187] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_port0CAPTURE_PIPELINE_v_proc_ss_csc_0
[ 454.699889] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_csc_0CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 454.715197] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 454.727375] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/scaler_portsCAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 454.740594] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/scaler_port1CAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 454.753814] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/sca_outCAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 454.766604] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/scaler_port0CAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 454.779831] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_scaler_0CAPTURE_PIPELINE_v_proc_ss_csc_0
[ 454.794793] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/GPIO_axi_gpio_0
[ 454.805234] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_iic_0
[ 454.815156] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/stts22htr
[ 454.825077] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_intc_0
[ 454.835089] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/system_management_wiz_0
[ 454.846231] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/vcap_portsCAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 454.859276] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/vcap_portCAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 454.872238] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_frmbuf_wr_0CAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 454.887027] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/ap1302_osc
[ 454.897040] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/ias_out0
[ 455.542332] xadc a0030000.system_management_wiz: IRQ index 0 not found
[ 455.561711] zocl-drm axi:zyxclmm_drm: IRQ index 32 not found
avnet-zub1cg-dualcam: loaded to slot 0The last verification step is to check the presence of the /dev/media node.
$ ls /dev/media*
/dev/media0
$ media-ctl -p -d /dev/media0There is a good chance that the first attempt did not work, which will require an iterative approach to verification... especially the device tree content...
Now the fun begins !
It is not required to re-program the SD card image at each iteration. Instead, we can re-build the avnet-zub1cg-base or avnet-zub1cg-dualcam packages, and re-install them on our image with the dnf utility, as shown below:
$ petalinux-build -c avnet-zub1cg-dualcam
$ ls build/tmp/deploy/rpm/zub1cg-sbc-base/avnet-zub1cg-dualcam-1.0-r0.*.*.rpm
avnet-zub1cg-dualcam-1.0-r0.2.zub1cg_sbc_base.rpmThis rpm package can be copied over to the embedded platform (via the SD card image, or via SSH), and installed as follows:
root@zub1cg-sbc-2022-2:~# dnf install rpm/avnet-zub1cg-dualcam-1.0-r0.2.zub1cg_sbc_base.rpm
...
Dependencies resolved.
======================================================================================
Package Architecture Version Repository Size
======================================================================================
Upgrading:
avnet-zub1cg-dualcam zub1cg_sbc_base 1.0-r0.2 @commandline 130 k
Transaction Summary
======================================================================================
Upgrade 1 Package
Total size: 130 k
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Upgrading : avnet-zub1cg-dualcam-1.0-r0.2.zub1cg_sbc_base 1/2
Cleanup : avnet-zub1cg-dualcam-1.0-r0.1.zub1cg_sbc_base 2/2
Verifying : avnet-zub1cg-dualcam-1.0-r0.2.zub1cg_sbc_base 1/2
Verifying : avnet-zub1cg-dualcam-1.0-r0.1.zub1cg_sbc_base 2/2
Upgraded:
avnet-zub1cg-dualcam-1.0-r0.2.zub1cg_sbc_base
Complete!The user can configure the image to automatically boot one of the firmware overlays. The /etc/dfx-mgrd/daemon.conf file indicates which overlay (default_accel) to load at boot in the /etc/dfx/mgrd/default_firmware.
root@zub1cg-sbc-2022-2:~# cat /etc/dfx-mgrd/daemon.conf
{
"firmware_location": ["/lib/firmware/xilinx"],
"default_accel":"/etc/dfx-mgrd/default_firmware"
}This file does not exist by default, but can be created as follows:
root@zub1cg-sbc-2022-2:~# cat /etc/dfx-mgrd/default_firmware
cat: /etc/dfx-mgrd/default_firmware: No such file or directory
root@zub1cg-sbc-2022-2:~# echo avnet-zub1cg-base > /etc/dfx-mgrd/default_firmware
root@zub1cg-sbc-2022-2:~# cat /etc/dfx-mgrd/default_firmware
avnet-zub1cg-baseThe change will take effect at the next boot.
root@zub1cg-sbc-2022-2:~# rebootAfter boot, we start by querying which "apps" are present:
root@zub1cg-sbc-2022-2:~# xmutil listapps
Accelerator Accel_type Base Base_type #slots Active_slot
avnet-zub1cg-base XRT_FLAT avnet-zub1cg-base XRT_FLAT (0+0) 0,
avnet-zub1cg-dualcam XRT_FLAT avnet-zub1cg-dualcam XRT_FLAT (0+0) -1
root@zub1cg-sbc-2022-2:~###Notice that the avnet-zub1cg-base overlay has been loaded.
Executing avnet-zub1cg-baseSimilarly to the previous project, we will create a jupyter notebook that performs a USB camera passthrough. We will add code to explicitly load the desired firmware overlay.
We start by launghing the jupyter-lab server in the serial console, specifying the IP address, as follows:
petalinux@zub1cg-sbc-dualcam-2022-2:$ jupyter-lab --allow-root --ip 10.0.0.179 &
[1] 879
[I 2023-03-20 16:07:40.475 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-03-20 16:07:40.548 ServerApp] Writing Jupyter server cookie secret to /home/petalinux/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2023-03-20 16:07:40.649 LabApp] JupyterLab extension loaded from /usr/lib/python3.9/site-packages/jupyterlab
[I 2023-03-20 16:07:40.649 LabApp] JupyterLab application directory is /usr/share/jupyter/lab
[I 2023-03-20 16:07:40.681 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-03-20 16:07:40.686 ServerApp] Serving notebooks from local directory: /home/root
[I 2023-03-20 16:07:40.686 ServerApp] Jupyter Server 1.13.5 is running at:
[I 2023-03-20 16:07:40.687 ServerApp] http://10.0.0.179:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
[I 2023-03-20 16:07:40.687 ServerApp] or http://127.0.0.1:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
[I 2023-03-20 16:07:40.687 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).On the PC side, the jupyter notebook can be accesed by copying the link that contains the token. For example, in my case this was:
http://10.0.0.179:8888/?token=6dae07dd4168bf26850904cadd43526fe146fcc59de4b384
http://10.0.0.179:8888/?token=6dae07dd4168bf26850904cadd43526fe146fcc59de4b384In the "Notebook" section of the "Launcher", click on the "Python3 (ipykernel)" button to create a new jupyter notebook.
We will start by explicitly loading the firmware overlay we want to use : avnet-zub1cg-base.
Create code cells and copy-paste the following code snippets:
import subprocess
import re
import osThe following code will query the platform name using the hostname utility.
def get_platform_name():
proc = subprocess.run(['hostname'], capture_output=True, encoding='utf8')
for line in proc.stdout.splitlines():
platform_name = re.search('(.+?)-sbc-(.+?)',line).group(1)
return platform_name
platform_name = get_platform_name()
print(platform_name)The following code will unload the current firmware overlay (if present):
cmd = 'xmutil unloadapp'
print(cmd)
os.system(cmd)The following code will load the base firmware overlay.
cmd = 'xmutil loadapp avnet-'+platform_name+'-base'
print(cmd)
os.system(cmd)Finally, we can paste the same code as in the previous project, to perform the live USB camera feed in the jupyter notebook:
import matplotlib.pyplot as plt
import cv2
import numpy as np
from IPython.display import display, Image
import ipywidgets as widgets
import threading
# Stop button
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Display function
# ================
def view(button):
cap = cv2.VideoCapture(0)
display_handle=display(None, display_id=True)
i = 0
while True:
_, frame = cap.read()
frame = cv2.flip(frame, 1) # if your camera reverses your image
_, frame = cv2.imencode('.jpeg', frame)
display_handle.update(Image(data=frame.tobytes()))
if stopButton.value==True:
cap.release()
display_handle.update(None)
# Run
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()Save the jupyter notebook with a meaningful name (ie. usbcam_live_view.ipynb), then click the "Run" icon on each of the two code cells.
You will see a live video feed in the jupyter notebook, as shown below:
Click the "Stop" widget to stop the live feed.
Known IssuesThe current version of this project has the following known issues:
- avnet-zub1cg-dualcam does not enumerate /dev/media node
Issue resolved on 2023/04/20. The issue was caused by the PS I2C0 peripheral not being active. The solution was to reference the dualcam hdl project (instead of the base hdl project) when creating the petalinux project.
ConclusionI hope this tutorial helped to understand the benefits of using "xmutil" and the dynamic configuration of the PL to create a single SD image for your custom AI applications on the ZUBoard.
If you would like to have the pre-built petalinux BSP or SDcard image for this common platform, please let me know in the comments below.
Revision History2023/04/20
Resolved following issue:
- avnet-zub1cg-dualcam does not enumerate /dev/media node
The issue was caused by the PS I2C0 peripheral not being active. The solution was to reference the dualcam hdl project (instead of the base hdl project) when creating the petalinux project.
2023/04/11
Added registration link to webinar series
http://avnet.me/ZU1-Robotics-webinar-series
2023/03/27
Preliminary Version
Comments