DRP-AI dashboard for vision-AI processing on RZBoard V2L

In this project we will step through the process of how to:

• Program RZBoard eMMC Flash with a new pre-built Linux image that includes the Renesas usb_camera_http_demo application (to enable evaluation of the performance of different AI models, using USB webcam video, processed on DRP-AI core, with result reporting in a web browser
• Setup and use the PC based application webpage, to dynamically switch between different vision-AI models (to perform object detection, object classification, and pose-estimation AI functions) utilizing the DRP-AI neural network acceleration core on the RZ/V2L Renesas processor.
• Evaluate performance metrics and visualizations that are rendered onscreen in the application's informative internet browser-based UI.

Focus area for the AI accelerated RZ/V2L apps processor is to provide low-power, cost-effective, vision-AI functionality. It's DRP-AI core offloads compute-intensive AI inference tasks from the CPU cores, for significant acceleration and power efficiency.

Four different AI models translated to run on the DRP-AI core will be exercised:

(The easy procedure to translate onnx AI models to run on RZ/V2L, is covered in the Run Machine Learning on RZBoard project on hackster.io by Monica Houston, as well as in the Renesas RZ/V2L AI Implementation Guide)

What is DRP-AI?

DRP-AI is the AI-accelerator core on the RZ/V2L device, it is composed of:
DRP (Dynamically Reconfigurable Processor), it excels at parallel operations
AI-MAC (AI Multiply-Accumulate), accelerates Multiply-Accumulate operations

DRP-AI Dashboard Demo Prerequisites

Install Tera Term serial console application fromhttps://osdn.net/dl/ttssh2/teraterm-4.106.exe/

Download and unzip the following two software zip files from Avnet:

• RZBoard_dashboard_HTTP_dark.zip(306 KB)
• RZBoard_dashboard_eMMC.zip(1.34 GB)

The DIP switch on RZBoard shall remain set as follows throughout this exercise:

A fly-lead is needed to strap BOOT2 (if / when reflashing the bootloader image)

Most USB webcams are supported, you can use 1 of the 2 listed here, or try using the USB webcam that you already have

RZBoard Hardware Test Setup

With RZBoard powered OFF, connect the following cables:

Note regarding RZBoard HDMI output!

• Connection of RZBoard HDMI output (to show Wayland desktop) is not required for the webpage based DRP-AI USB camera HTTP demo application. This will however be useful if you also plan to run "standalone" DRP-AI demos (also in the filesystem) and/or Gstreamer exercises (described elsewhere)

Note regarding USB-Serial J19header fly-lead connections!

• Tabled here are wire colors and RZBoard J19 header connections when using the following popular PL2303TA based USB-Serial adapter cable

• Open Windows Device Manager to: a) check the software driver for this serial cable correctly loadedb) note the COM port reported for this serial cable.
• Open the Tera Term console PC application, from the menu then use - Setup/Serial, then set it use this COM port (115200 8N1) - Setup/Save, to save the Tera Term serial settings- Make sure the console does not show “Disconnected”!

Note regarding Ethernet IP addresses!

• The RZBoard eMMC Flash programming is scripted for the following:
  RZBoard eth0 IP address must be configured as 192.168.1.99
  Laptop/PC Ethernet adapter IP address must be 192.168.1.88
• The DRP-AI USB camera HTTP demo application however has:
  RZBoard eth0 IP address hard-coded as 192.168.1.11
  ie. After completion of eMMC programming and reboot of RZBoard, you need to manually set eth0 IP address to 192.168.1.11 before running the demo application (the step for this is described later)
• Rather than a direct Ethernet connection with the host computer, an optional more versatile network setup for running this application, is to have RZBoard (and your host computer) connected via your network router. For this to work however, the DHCP server in your router must be configured to assign IP address 192.168.1.11 to RZBoard. This is necessary because the embedded application is hard-coded with RZBoard at this specific IP address. (Access this from the network router's admin screen --> LAN --> DHCP Server)

RZBoard_dashboard_HTTP

Adapted from the usb_camera_http_demo_webpage (developed by Renesas), this dashboard webpage template gets served from your development PC. It displays two video feeds, plus various measurements and AI results that are streamed from RZBoard via the 1G Ethernet interface. Download and unzip this webpage (using default unzip folder name) onto your test computer.

Download link: RZBoard_dashboard_HTTP_dark

RZBoard_dashboard_eMMC

An Ubuntu 20.04 host development environment is notrequired for this! For simplicity Windows.bat files are used to reflash the eMMC with:

• Bootloader files (using 115200 baud USB-serial interface)
• Linux system image file (using the faster 1G Ethernet interface)

Programming is scripted with two separate.bat files, as different boot modes must be setup and typically only the Linux image needs to be updated.

Download and unzip this large zip file (use default folder name on test computer) that includes the flash-programming tools plus prebuilt RZBoard Linux 5.10 image (which includes the usb_camera_http_demo embedded application)

Download link: RZBoard_dashboard_eMMC

The files that are transferred from the PC and programmed into RZBoard eMMC are summarized in the table below:

Download the RZBoard-Linux-Yocto-UserManual-v2.1 document from the RZBoard product page for a detailed step-by-step procedure of eMMC flash programming. The key steps however are summarized here as follows.

1) Prior to launching either of the.bat files, make sure of the following:

• RZboard is powered down!
• All Tera Term windows are closed!
• USB serial cable from PC has TX, RX, GND fly-leads connected to correct J19 header pins and the COM port has enumerated on the PC
• Ethernet cable from PC is connected to RZBoard and the Laptop/PC's Ethernet adapter IPv4 properties set a static IP address of 192.168.1.88
• The config.ini file has been edited to list: - the correct COM port for the attached USB-Serial cable- the correct name for the.wic Linux image file (if an updated version is used)

2) To reflash the bootloader images, set the bootmode to SCIF serial mode: This done by setting BOOT2=1 by strapping J19-pin1 to +5V(ie. connect fly-wire from J19-pin1 to J1-pin2 on the 40pin header)

With RZBoard powered OFF, from Windows file explorer run flash_bootloader.bat (this launches the applicable Tera Term macro using the edited config.inisettings) Choose eMMC as the flash to program, the macro then waits for power up...Press and hold S1 for 2 seconds to power-on RZBoard, the macro will now proceed. Wait for this to complete (<5 min).Make sure to shutdown the Tera Term window after this has completed

2) To reflash the Linux System image, set the bootmode to eMMC mode:ie. Set BOOT2=0 by removing fly-wire previously used (in reflash of bootloader)

With RZBoard powered OFF, run flash_system_image.bat (this launches a different Tera Term macro, using same saved config.ini settings)

Power-on RZBoard. A blue window should open and an Ethernet connection established within 30 seconds

Wait for the macro to complete (you shall see multiple blocks of data getting sent) with the sequence finishing in 5 to 10 minutes. After finishing, press any key to exit the BAT script. You will also then need to manually exit the Tera Term console.

Setup and Run the RZBoard embedded application

(ie. usb_camera_http_demo)

• Press and hold S1 button switch to power-up RZBoard
• Monitor the U-boot and Linux console boot messages for any errors
• Now execute the following commands from the Linux command prompt (after entering login root and password avnet)

$ ifconfig eth0 192.168.1.11
$ ls -l

• Change directory to usb_cam_http then launch the embedded application:

$ cd usb_cam_http/
$ ./usb_camera_http_demo

Setup and run the PC based application webpage

• Make sure the development PC's Ethernet adapter IP address is configured for same network sub-LAN 192.168.1.xx as used by RZBoard (if Ethernet adapter was manually set to 192.168.1.88, then use this setting)
• Open the RZBoard_dashboard_HTTP folder on your test computer
• Double-click on the index.html file to load the webpage from local drive, into your internet browser (Chrome or Edge browser recommended)

Evaluate onscreen performance metrics and visualizations

• In the browser view, from the "Select Model" drop-down menu, launch the execution of RZBoard application code for one of the provided AI models:

• If this does not render at first try, close the browser, reopen index.html, then again select the model to exercise, from the drop-down list
• Keep monitoring the serial console to view status of the embedded application running on RZBoard, once it is running with the selected model.The console log should report information similar to the following...

• Once this embedded application is running a selected model, the browser screen will populate with the following (see numbers on the image below):
  1) AI model selector (and model ID confirmation)
  2) Live capture video panel with FPS info
  3) Processed video panel with FPS plus performance times
  4) Overlay graphics (when applicable)
  5) Bounding Box
  6) CPU utilization chart
  7) Detection Info (eg. classification, confidence, identity, coordinates)
  8) Processing Time (DRP-AI and CPU processing time)

Notes regarding PC based application webpage "dashboard"

• Framerate of USB camera capture video is set for 30 fps (L-panel)
  Framerate of AI inference video depends on the AI model (R-panel)
• Processing time varies significantly according to which model is used
• < Further notes to be added... >

Reference Links

(Download links for files that target the Renesas RZ/V2L EVK board)

RZ/V2L DRP-AI USBCamera HTTP Demo Pre-Built version

RZ/V2L DRP-AI USBCamera HTTP Demo SD Image version