Easy AI with Python and PYNQ

Introduction

The following instructions document a novel method to immediately get started using Xilinx Vitis AI v1.1 hardware accelerated machine learning inference. It also enables Python control and execution of the Vitis AI Xilinx Deep Learning Processing Unit (DPU). The VART api is also now supported.

Get started with Vitis AI on either the Ultra96 (v1 and v2), ZCU104 or ZCU111 edge platforms in just a handful of simple steps. For advanced users here are some benefits to using Vitis AI with PYNQ versus the standard PetaLinux approach:

• There are no initial Xilinx tool installations nor any docker images to download before taking it for a spin!
• You can switch between different inference models on the fly!
• You can control and interface with the DPU using Python (or C/C++)!
• You do not have to rebuild the SD image each time to run a different inference model!

There is a one time modest amount of local compile time involved, however it is much shorter than any known prior techniques that assemble an appropriate platform for Xilinx Vitis AI.

With the included PYNQ Vitis AI Jupyter Notebooks you will immediately be able to use the following inference accelerators:

• YOLOv3
• Inception
• Resnet50
• Classic MNIST
• (You may also create your own custom inference models)

All under Pythonic control or C/C++ if needed!

The Open Source PYNQ framework has always enabled easy integration with the AARCH64 Ubuntu application universe. Now hardware accelerated Vitis AI with PYNQ too!

It gets even better! Many other free pre-trained Caffe, Tensorflow, Darknet and PyTorch Vitis AI models from the Xilinx AI Model Zoo can now easily be ported to run on PYNQ enabled boards too. You can even build your own custom inference model with Xilinx Vitis AI! https://github.com/Xilinx/AI-Model-Zoo

Cautionary note: If you follow these instructions, some PYNQ Python libraries and other libraries may be updated. If you rely on older PYNQ or Xilinx AI DPU versions you may want to backup your existing PYNQ image first or try this on a new installation of the latest version of PYNQ.

Before getting started, get the right version of PYNQ software on your dev board

Your PYNQ board should be running the latest version of PYNQ! See some tips at the bottom to help you upgrade if needed.

What are you waiting for: get started!

If this is your first time using or installing PYNQ, see Notes at the end for links to each board's different PYNQ SD card image and how to get it running. With a compatible board already running PYNQ, follow these steps!

Step 1 - Access the PYNQ Jupyter command console

With PYNQ up and running and the board itself connected to the Internet through a wired LAN or Wifi, from a PC that can access the LAN the board is on, use a web-browesr (preferably Firefox, Chrome or Safari) and login to the PYNQ Jupyter server. Then open a Jupyter command root console tab in your web-browser.

Tip: you can login to the Jupyter server running on the PYNQ board by entering the IP address of the PYNQ board that is on your WLAN or LAN in your PC web-browser. For the screenclip this was done for an Ultra96 which was connected to a PC through USB OTG, the Ultra96 had a USB to Ethernet adapter attached to provide Internet to the Ultra96:

You may be asked to enter a password, which is all lower-case: xilinx

After the password entry you should see (this example shows an Ultra96):

Once you login to the Jupyter notebook server through your web-browser, use the mouse and click on the New button (circled in red above) and select to create a new root console terminal:

This will open a command console with root privilege, in this console you will soon enter the installation commands:

Step 2 - Update PYNQ and install Vitis AI

Note: for this to work your dev board MUST have an Internet connection! (See far below for some tips and instructions if you don’t know how). I do not recommend using U96 v2 wifi, because it is very slow. I recommend using a board's onboard Ethernet or a USB to Ethernet adapter for the U96 v2.

The hardest part for this step is to be patient!!! This step will directly compile the Vitis AI and Xilinx XRT drivers and libraries on the PYNQ board. It will take from half to a whole hour. It depends on the speed of your SD card. It is worth the wait and only needs done once!

Enter the installation and compilation commands exactly as shown. Enter each of the 3 lines one at a time in the Jupyter command root console and WAIT for them to complete:

git clone --recursive --shallow-submodules https://github.com/Xilinx/DPU-PYNQ
cd DPU-PYNQ/upgrade
make

Step 3 - Install the Python DPU package

In the command root console enter the following and wait for it to complete:

pip3 install pynq-dpu

Step 4 - Install YOLOv3, Resnet50, MNIST and Inception Jupyter Notebooks

In the command console enter the following, make sure you enter a space and final '.' as shown:

cd $PYNQ_JUPYTER_NOTEBOOKS
pynq get-notebooks pynq-dpu -p .

Step 5 - Run the YOLOv3 notebook

Here is the beginning of the PYNQ Python YOLOv3 notebook, you will find dpu_yolo_v3.ipynb under the new folder pynq-dpu:

You select a Jupyter cell with a mouse and use <SHIFT><ENTER> to execute that cell. If you are not familiar with how to execute a Jupyter Notebook, see the tutorials here: https://jupyter.org

This was run on an Ultra96 and I customized it to use a different input image than what is included:

Here is the output for the end of the notebook:

Step 6 - Enjoy!

Nuff said :-)

Rebuilding the DPU for different AI Models

A future tutorial may cover this, for now please read the instructions in the PYNQ DPU Vitis AI Github: https://github.com/Xilinx/DPU-PYNQ

Appendix

Extra tips for how to get PYNQ running on a compatible Xilinx MPSoC board

Compatible dev boards are Ultra96 v1 and v2, ZCU104 and ZCU111.

These are some additional and optional tips to help you get PYNQ installed and up and running with an Internet connection.

If you don’t already have the latest version of PYNQ running on your board, install it. The oldest version of PYNQ that has a chance of working is v2.5. Older versions may work too but it is strongly advised that you use the latest.

For Ultra96, download the SD image from github. Be careful to install the correct image for either board version of Ultra96 v1 or v2. If you have an industrial Ultra96 use the v2 image. The U96 setup instructions pdf on github explains how to tell the difference. https://github.com/Avnet/Ultra96-PYNQ/releases

For ZCU111 and ZCU104 you can find the latest images here: https://pynq.io

Sorry, at this time the PYNQ Z1/Z2 boards do not yet support Vitis AI!

For this project to work, the PYNQ board must have access to the internet through a network connection!

If you can, establish and use a wired LAN connection. For the Ultra96, plug-in a USB to Ethernet adapter and simply power up the board. As long as your LAN has a DHCP service it will automatically obtain an IP address. If you cannot identify the IP address of the USB to Ethernet adapter, you can plug-in a usb cable to the Ultra96 micro usb OTG socket and the other end to a PC, the IP address for the micro usb OTG from the connecting PC will always be 192.168.3.1.

You can then use either 192.168.3.1 or the acquired USB to Ethernet address to proceed. Other PYNQ supported boards will have slightly different means of connecting the board to the internet, plan accordingly. The key is to allow the board to have internet access and to also know what its’ local IP address is. For Ultra96 v1 and v2 the github setup instructions provide additional details.

You may also do this through an Ultra96 wifi connection but this will run a bit slower. To connect to wifi you will need to use the USB OTG connection to a PC and then you may execute the included Jupyter Notebook. You would follow the steps above to login to the Jupyter Notebook through 192.168.3.1 and run the Jupyter Notebook entitled common/wifi.ipynb

If you are new to Jupyter Notebooks you can find more information here (just fyi, PYNQ also comes enabled with the Jupyter Lab IDE): https://jupyter.org

Pseudo Glossary

PYNQ Main Website

https://pynq.io

PYNQ Community Forum

https://pynq.io/discuss

PYNQ DPU Github

https://github.com/Xilinx/DPU-PYNQ

Vitis AI Info

https://github.com/Xilinx/AI-Model-Zoo

https://github.com/xilinx/vitis-ai

https://github.com/Xilinx/Vitis-AI/tree/master/Vitis-AI-Library

Ultra96 release images, installation instructions and repo to re-build PYNQ and customize it (for advanced users)

https://github.com/Avnet/Ultra96-PYNQ

https://ultra96-pynq.readthedocs.io

ZCU111 PYNQ Github (for advanced users who want to customize)

https://github.com/Xilinx/ZCU111-PYNQ

Main PYNQ and ZCU104 PYNQ Github (for advanced users who want to customize their PYNQ bsps)

https://github.com/Xilinx/PYNQ