Run Machine Learning on the RZBoard

Intro

Many specialized accelerators for machine learning require you to only use models from their model zoo or go through many hoops to translate a model to their special format.

Luckily, I found that translating models into the proprietary format for the RzBoard was surprisingly easy, and took me less than half a day, including installation time, to get from the onnx model that I wanted to run to an app incorporating the model running on their board.

This tutorial will walk you through the necessary steps. If you get stuck at any point, please reach out in the comments here. Also know that Renesas has a wealth of information on the RZ/V2L documentation page.

Prerequisites

You'll need an Ubuntu build machine. Ubuntu 18.04 is recommended, although I successfully used 20.04. The following prerequisites are required as well:

Be sure to use Python 3.6. I found that using the translator with python 3.8+ gave me this error: "TypeError: Shared Cython type cython_function_or_method has the wrong size, try recompiling")

Install the DRP-AI Translator

Download and extract the DRP-AI Translator from Renesas' RZ/V2L documentation page.

While you're at it, download and extract the RZ/V2L DRP-AI Support Package [V7.00] as well, as we'll need the demo applications they provide to set up a sample machine learning application.

Install dependencies, including pip:

sudo apt-get update
sudo apt-get install -y libgl1-mesa-dev
sudo apt-get install -y python3-pip
pip3 install -U pip

Note: you'll likely also want to add pip to path, E.G. export PATH="$HOME/[USER]/.local/bin:$PATH"

Now install the DRP-AI translator:

chmod +x DRP-AI_Translator-v1.60-Linux-x86_64-Install 
./DRP-AI_Translator-v1.60-Linux-x86_64-Install

Once the DRP-AI Transator finishes installing, the following directories will be generated in your current directory:

Download or build the RzBoard's Yocto image

Boot the Yocto Image

Download the RzBoard's Yocto image. Flash the image to microSD card using a program like Balena Etcher.

Use a known good microSD card such as Kingston or Samsung. Unfortunately, Sandisk and Kexin SD cards won't work.

Connect to the RzBoard

1. Set the Boot Mode Switch (SW1) to eMMC with SW1 1=ON and 2=OFF. Connect the FTDI cable to the 4-pin header as shown:

2. Connect UART debug port to USB-to-Serial module, and connect the USB to Serial module to USB port of a PC acting as a host terminal.

3. Open the terminal software (Tera Term is Recommended), choose the correct COM port number and apply the following configuration.

• Baud rate: 115200

• Data bits: 8

• Parity: None

• Stop bits: 1

4. Connect a Micro to HDMI cable to the Mini HDMI connector. Connect the other end of the cable to a HDMI display panel.

5. Attach a 5V, 2A DC power supply to the USB Type-C connecter.

6. Press and hold the Power ON/OFF Button (S1) for more than 2s to power on RzBoard, The LED_RGB will flush blue.

7. As the board boots up, you will see 4 penguins appear in the upper left-hand corner of the monitor, and then you will see the Linux terminal icon on the top left and the timer on right top corner. Congratulations, you are up and running!

Note: the default login is "avnet" and the default password is "root."

TRANSLATE A MODEL

The RzV2L Translator can translate onnx models into RZ/V2L compatible models.

There is a model zoo with a variety of onnx models on onnx's github. Today, I'll be translating resnet50v2. Download it and unzip it to get started.

If you have a Tensorflow model, Renesas has instructions for converting it to onnx format prior to in their AI IMPLEMENTATION GUIDE Appendix REV.7.00 using tf2onnx.

Create a pre and postprocessing definition file

The first step in translating a model to use on the RzBoard V2L is to create pre and postprocessing definitions and address map definition files for your model. You can find these in the /UserConfig/sample folder:

1. Copy one of the examples found under./UserConfig/sample, e.g. prepost_resnet50v1.yaml, and rename it prepost_resnet50v2.yaml (Note: this naming is important, as the translator uses it to find the files during translation).

2. Save it in./UserConfig.

3. You'll need to define the following in this file:

• input_to_pre: [...] - defines the input to the preprocessing part. This will depend on the input to your network from your camera or image sensor, or if you're just testing the application by using a bitmap.
• input_to_body: [...] - defines the input to the neural network. You'll need to change the input name and shape. Find these by looking at the model in software like netron (described below).
• output_from_body: [...] - defines the neural network's output. You'll need to change the name, shape and channel.
• output_from_post: [...] - defines the output from the postprocessing part.
• preprocess: [...] - defines the preprocessing. Change "dest" to your neural network's input name, and change shape_out to its shape.
• postprocess: [...] - defines the postprocessing. Change "src" to the name of your neural network's output name (see the below section if this is confusing).

Use Netron to find your model's input and output names

Open Resnet50v2 (or whatever model you're using) in netron. Here, you can see that the input's name is data, so "dest" should be "data." The output name is resnetv24_dense0_fwd, so "src" should be "resnetv24_dense0_fwd."

Create an address map definition file

Specify the address map to place the DRP-AI object files in the memory space by using "Element-Space" and "Sub-Space". "Element-Space" means the minimum composition unit of the memory space. A set of "Element-Space" is called "Sub-Space". We'll use the default memory config.

1. Copy one of the examples found under./UserConfig/sample, e.g. addrmap_in_resnet50v1.yaml, and rename it addrmap_in_mobilenet_v1_224_quant.yaml

2. Save it in./UserConfig.

Run the translator

Run the translator with this command:

cd drp-ai_translator_release 
./run_DRP-AI_translator_V2L.sh [prefix] -onnx [path]

• [prefix] is the name of the translated model, e.g. "resnet50_v2". The translator generates a a folder named /output/[prefix].
• [path] is the path to the onnx model, e.g../onnx/resnet50_v2".onnx

When the translator completes, if the translation is succesful you'll find a number of files under output/[prefix]. The following diagram explains the output:

BUILD AND RUN YOUR ML APP

Within the DRP-AI_support_package_v7.0.0 that we downloaded earlier, extract and navigate to the folder "rzv2l_drpai-sample-application". Within this folder, you'll find a number of sample applications. Each sample application contains three folders: src, exe, and etc.

If you want run the application with a camera, copy app_resnet50_cam and rename it app_resnet50v2_cam.

If you don't have a webcam or simply want to test the model using a bitmap image, copy app_resnet50_img.

Note that this will depend on how you translated the model in the previous step.

Drag and drop your translated model, e.g. "output/resnet50v2" into the copied exe folder, e.g. app_resnet50v2_cam/exe. Rename it resnet50_cam and overwrite the existing model.

Drag and drop the pre and post processing definition files you created earlier into app_resnet50v2_cam/etc.

If you would like to make any changes to the application, open sample_app_resnet50_cam.cpp in the src directory in a code editor. For instance, I added a timer to print the time that it took my app to run.

Compile your app

1. Set the SDK environmental variable:

source /opt/poky/3.1.5/environment-setup-aarch64-poky-linux

2. cd into the app_resnet50v2_cam/srcand run the compilation command:

cd $APP_WORK/app_resnet50v2_cam/src
make

After compilation, a generated binary file with a name like "app_resnet50_img" will appear in the src directory. Move this into the exe directory.

Zip the exe folder containing the app:

tar -zcvf resnet50_cam_exe.tar.gz /app_resnet50v2_cam/exe

Move the app to the RzBoard

Connect an ethernet cable to the board's ethernet port. Open a terminal on the RzBoard and run "ipconfig" to get the IP address.

Copy the folder containing the app to the RzBoard from your host PC (optionally, you could also use a USB stick to move the resnet50_cam_exe.tar.gz zipped directory from the host computer to the RzBoard):

cp resnet50_cam_exe.tar.gz avnet@[IP ADRESS]:~/root

On the RzBoard, unzip the folder:

tar -xzvf resnet50_cam_exe.tar.gz

Run the app on the RzBoard

Execute the application:

cd /home/root
./app_resnet50_img

Note: you may need to change the file permissions on RzBoard to allow the app to run:

chmod +x app_resnet50_img

And that's all! You should see that your app has successfully run on RzBoard. Congratulations!