I created this project as part of my efforts to benchmark Tensorflow Lite on the MaaXBoard. I wanted to compare the MaaXBoard's performance running inference to similar developer boards, so I chose the Google Coral and Raspberry Pi 3 Model B+.
- Instructions for benchmarking the Raspberry Pi 3 Model B+ are here.
- Instructions for benchmarking the MaaXBoard are here.
- Final benchmarking article is here.
- Alasdair Allan's original big benchmarking roundup is here.
In addition to running the benchmark, this guide walks you through getting started with Google Coral. There are already a couple great setup guides for the Coral dev board, including the official getting started guide at coral.ai and Alasdair's guide on Hackster. However, I ran into several problems while using OSX Catalina to connect to the dev board, so I figured I should document them here so other people won't spend hours looking for a workaround like I did.
This guide is specifically for Mac, so if you're on Linux, you should probably follow one of the guides I just mentioned. Windows isn't compatible at all. If you've already set up your Coral dev board before, skip to step 6.THE BENCHMARK
The Coral dev board is picky about the models it can run. It can only run Tensorflow Lite models that have been pre-compiled for the edge TPU.
MobileNet V.1 and V.2
Luckily for us, there are pre-converted models that meet all of the requirements, like these versions of MobileNet v1 and MobileNet v2. Even luckier - Alasdair Allan created a script to benchmark MobileNet last year, so I can run the exact same script and compare my results to his.
I wanted to see what the Coral dev board was capable of, so I chose one more model to benchmark - EfficientNet-Lite.
EfficientNet-Lite is a version of the EfficientNet model that was created by Google for use on edge / mobile. It's specifically tuned for Google Coral's TPU, so I wanted to try that as well to see what kinds of speeds and accuracy I could achieve.1. SETUP YOUR MACBOOK
The Coral dev board doesn't have a GUI. In order to work with it, you'll have to install a couple tools to connect to it.1.a Install Screen
Screen is the recommended serial console program. It can be installed with brew. Check t
1.b Install Fastboot
brew install screen
The fastboot protocol is a mechanism for communicating with bootloaders over USB or ethernet. It is designed to be very straightforward to implement, to allow it to be used across a wide range of devices and from hosts running Linux, macOS, or Windows.
Download Google's Platform-Tools (includes fastboot): Mac
Google Corals instructions say to do it like this:
mkdir -p ~/.local/bin
sudo mv ~/Downloads/platform-tools/fastboot ~/.local/bin/
However, because Catalina blocks 3rd party developers, it has to be done like this:
brew cask install android-platform-tools
fastboot version 29.0.6-6198805
Installed as /usr/local/bin/fastboot
It will give you a warning, so go to under Security and Privacy settings, enable this 3rd party developer, and then click "open" when the popup returns.
Check that it's installed:
For Mac compatibility, the version must be 28.0.2 or higher.1.c Install Mendel Developer tool on your Mac
The Mendel developer tool (mdt) is supposed to let you transfer an SSH key to the board. You can install it with pip.
$ pip3 install --user mendel-development-tool
As you'll see later, mdt doesn't work correctly on OSX Catalina, so the workaround is detailed in step 5.
On mac, Mendel requires Silicon Lab's CP210x USB to UART Bridge Virtual COM Port (VCP) driver for Mac. I believe that the problem is with the driver. Hopefully they update it soon! Maybe they already have. Install the latest version just in case.2. FLASH THE BOARD
Now is the moment you've been waiting for! You get to use your board.
First, check to make sure your boot switches are set to "on, off, off, off" as pictured:
Connect micro-USB to your board, and the other end to your computer.2.a Initiate fastboot mode
Checkyour serial number. If your serial number starts with "9410" or higher, then it was manufactured on April 10th, 2019, or later, so you can skip this step and go to step 2.b (because your board automatically boots into fastboot mode).
[PHOTO OF SERIAL #S]
In a terminal on your mac, use screen to connect to the board.
screen /dev/cu.SLAB_USBtoUART 115200
Immediately plug the board into power (power is the USB-C port closest to the audio input).
You should see something like this after it starts up (if you don't, press enter):
Welcome to your new Coral EdgeTPU Development Board!
To get started, you need to download and flash the latest firmware.
Please follow the instructions at g.co/coral/setup.
The board is now in fastboot mode2.b Execute the flash script
Plug another USB-C cable into the board's data port (USB-C closest to HDMI) Open a new terminal window (ctrl-n) and run "fastboot devices"
curl -O https://dl.google.com/coral/mendel/enterprise/mendel-enterprise-day-13.zip
unzip mendel-enterprise-day-13.zip && cd mendel-enterprise-day-13
Sending 'bootloader0' (991 KB) OKAY [ 0.057s]
Once flashing is done, shut down the Google Coral:
sudo shutdown now
Then unplug your USB-C power cable and USB-C data cable.
NOTE: When you're done using the Coral dev board, always use this command to shut the board down. Don't simply unplug the board without shutting it down first, since that could corrupt the image.3. CONNECT TO WIFI ON YOUR BOARD
With the USB-B pluggd in, from terminal, run:
screen /dev/cu.SLAB_USBtoUART 115200
Then immediately plug a micro-B USB cable into the MaaXboard's micro-USB port. The system of the dev board will start loading, the login prompt will come up. Password and username are both "mendel."3.a Use nmcli to connect to your wifi
nmcli dev wifi connect 'YOUR SSID' password 'YOUR PASSWORD' ifname wlan0
Device 'wlan0' successfully activated with 'c473dd7c-e7ca-4827-821f-efdaa893cc6d'.
Note: dual band routers won't work.3.b Make it more permanent by changing the etc/hosts file
sudo nano /etc/wpa_supplicant/wpa_supplicant.conf
You can certify the connection with
nmcli connection show
nmcli dev wifi
(connected network will have a * next to it):
5. CONNECT TO YOUR BOARD VIA MDT5.a Connect to the board's shell via mdt
sudo apt-get update
sudo apt-get dist-upgrade
mdt is supposed to be a user-friendly version of SSH that will automatically generate a public key and share it with your PC over USB. Other ways of connecting to the board, like regular ssh, are disabled, so if mdt isn't working it makes it difficult to share files, etc with the board from your mac. Read on to learn about the problem I ran into and how I re-enabled SSH.
At this point, Google's instructions say to plug the USB-C cable into the data port (the USB-C port closest to the HDMI port) and unplug the USB-B cable. On a Linux machine, and on older versions of Mac, the UART device shows up as a network connection. That's what the Silicon labs driver you installed earlier is intended to help with. Open a new terminal and run:
If you're able to connect to your device, congrats! The issue has been fixed. Skip to step 6. If not, you'll get the error:
Unable to find any devices on your local network segment.
This is a problem on macOS Catalina. In spite of having the Silabs driver installed, it doesn't "see" USB connection as a network connection. Mdt is only able to see the Coral dev board wifi, but when you try to connect via WIFI using mdt (
mdt shell [WIFI IP]), it won't exchange keys, and gives the message "Cowardly refusing to attempt to push a key to a public machine."
To see what was wrong with the USB connection, I opened a new terminal and ran:
sudo ifconfig -a
I realized that the USB to UART actually shows up as two different devices with two different IP addresses.
As you can see in the above screenshot, my USB1's IP address is 192.168.101.2, and USB2's IP address is 192.168.100.2, but running the command mdt shell with the explicit IP address can't connect to either of them:
Note - if you run mdt and get the error "command not found: mdt" make sure mdt is added to your path.THE FIX FOR OSX CATALINA
I finally found part of the fix in this blog post and the rest of the fix in this video:
Try connecting to your Coral Dev Board using the WIFI IP address you found earlier (mine was 192.168.1.28):
mdt shell 192.168.1.28
You'll get this error:
"Cowardly refusing to attempt to push a key to a public machine."
In another terminal window on Mac, generate a new key:
save its as
Copy your key to the mdt config on your macbook:
cp mdt.key ~/.config/mdt/keys
View your recently created public key:
If you're still connect to the Coral dev board via screen, open that terminal, otherwise reconnect with screen /dev/cu.SLAB_USBtoUART 115200 (you may need to unplug and re-plug USB-B, unplug the data USB-C if it's plugged in, and don't forget to press "enter").
Connected to the Coral dev board. In the "mendel" directory, type:
Create a new file within the.ssh folder called authorized_keys:
Paste in the public key you just created with ssh_keygen and save the file with ctrl-X (or however you like to do it). Get your board's IP address:
ip addr | grep wlan0
Now if you type:
You should be able to SSH into the board.5.b Edit the Google Coral's ssh_config:
The above fix depends on your Coral dev board's IP address in /.ssh/known_hosts staying the same. Every time your board's IP address changes you'll have to create a fresh public key, so to make it easier for you to connect, it's easiest to edit the dev board's ssh_config file.
(DISCLAIMER: note that allowing ssh with only password, as I am describing here, is not as safe as using keys, if you do it, please be aware of the risks in case you have sensible data or security is an issue for you.)
In your screen terminal:
- Open ssh_config
- In the file, change
- Then change
- Restart the
sudo service ssh restart
Now you should be able to copy files back and forth using scp.
Note: if you ever upgrade your board's distro, you'll be asked whether or not you want to upgrade sshd_config. Select "keep the local version currently installed."
Get your python 3 version on the dev board:
My Python version is 3.7 so I chose the cp37 version for aarch64 linux.
pip3 install https://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp37-cp37m-linux_aarch64.whl
Ok, all of that was leading up to running our benchmarks. The first benchmark we'll run is MobileNet.7.a Download the files
The original files Alasdair created were hosted on dropbox here. Because of changes to Tensorflow v.2, I've updated
benchmark_edgetpu.py. It's in the code files for this project. The project code also includes two other benchmarking scripts. The only thing it doesn't contain is the EfficientNet-Lite model, but that we can download from the Google Coral github.
You can download the files from Hackster with curl:
7.b Run benchmark_edgetpu.py
curl -L https://hacksterio.s3.amazonaws.com/uploads/attachments/1143279/CORAL.zip > files.zip
Benchmark for MobileNet SSD v.2 on Coral dev board:
python3 benchmark_edgetpu.py \
--model ssd_mobilenet_v2/tflite_for_edgetpu/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite \
--label ssd_mobilenet_v2/tflite_for_edgetpu/coco_labels_for_edgetpu.txt \
--input fruit.jpg \
--output out.jpg \
Elapsed time is 25.81631173300002 ms
apple score = 0.98046875
banana score = 0.93359375
Benchmark for mobilenet SSD v.1 on Coral dev board:
python3 benchmark_edgetpu.py \
--model ssd_mobilenet_v1/tflite_for_edgetpu/tflite_graph_1556320556_edgetpu.tflite \
--label ssd_mobilenet_v1/tflite_for_edgetpu/coco_labels_for_edgetpu.txt \
--input fruit.jpg \
--output out.jpg \
Elapsed time is 18.869002893000015 ms
apple score = 0.7890625
banana score = 0.69921875
dining table score = 0.53125
Wow. Not only does inference correctly classify the apple and banana, it also correctly recognizes that they are sitting on a table!7.c Running the MaaXBoard's benchmark on Coral
When benchmarking, it's sometimes hard to balance between keeping all things equal, and trying to make the benchmarks imitate actual use cases. Because the edgetpu is able to use the edgetpu detection engine, benchmark_egetpu.py makes use of it. This makes it slightly different from the benchmarking code for the Raspberry Pi and MaaXBoard, which I decided to try running their benchmarking code on MaaXBoard.7.d Install OpenCV on Coral dev board
The other platforms use OpenCV to resize images, so I decided to try using OpenCV on the Coral DevBoard too. Thankfully, Coral dev board now supports OpenCV.
- Install OpenCV
sudo apt install python3-opencv
- Test installation
7.e Run benchmark_coral.py
python3 -c 'import cv2; print(cv2.__version__)'
Benchmark for Mobilenet SSD v.2 on Coral dev board:
python3 benchmark_coral.py --model tflite_for_edgetpu/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite --label tflite_for_edgetpu/coco_labels_for_edgetpu.txt --input fruit.jpg --output out.jpg --runs 1000
Elapsed time is 15.911620770000352 ms
apple score = 0.953125
banana score = 0.83984375
Benchmark for mobilenet SSD v.1 on Coral dev board:
python3 benchmark_coral.py --model tflite_for_edgetpuv1/tflite_graph_1556320556_edgetpu.tflite --label tflite_for_edgetpuv1/coco_labels_for_edgetpu.txt --input fruit.jpg --output out.jpg --runs 1000
Elapsed time is 12.535867665999831 ms
apple score = 0.80078125
apple score = 0.515625
Copy the file off your Coral dev board so that you can check it out:
scp mendel@[IP ADDRESS]:CORAL/out.jpg .
This version was even faster than the original version. However, the accuracy was much worse, especially for MobileNet v.1. As you can see, mobilenet v.1 misclassified the banana as an apple:
EfficientNet-Lite was released in March. It's a smaller version of the full EfficientNet, which was published a year ago. EfficientNet-Lite is an image classification model aimed at mobile applications, and specifically tailored to the edge TPU found on the Coral dev board (and many Android phones).
Get the model. There's a 300px version that I used for this benchmark. The "L" stands for large, as in image size. There are three different versions of this model available in the Google Coral model zoo that take 3 different image sizes:
curl -OL https://github.com/google-coral/edgetpu/raw/master/test_data/efficientnet-edgetpu-S_quant_edgetpu.tflite > efficientnet-edgetpu-S_quant_edgetpu.tflite
The corresponding imagenet_labels are in the zip folder provided under the code section (ImageNetLabels.txt).8.a Run benchmark_coral_efficient.py
Benchmark_coral_efficient.py is the same benchmark as the previous ones, except that it is written to do classification with the Classification Engine.
Benchmark the TPU optimized model
python3 benchmark_coral_efficient.py --model efficientnet-edgetpu/efficientnet-edgetpu-L_quant_edgetpu.tflite --label efficientnet-edgetpu/ImageNetLabels.txt --input fruit.jpg --runs 1000
Elapsed time is 36.041603453999414 ms
hook, claw score = 0.2421875
banana score = 0.22265625
screwdriver score = 0.06640625
The accuracy is not perfect (banana comes in 2nd with a score of 0.22) but it's blazing fast for a classifier that can recognize 1, 000 images.
Benchmark the lite0 model
I also discovered that there is a version of EfficientNet-Lite that's not specialized for the edge TPU, so I decided to try running this one to see how it performed. It's quite a bit smaller than the other version - only 5.17MB: https://tfhub.dev/tensorflow/lite-model/efficientnet/lite0/int8/1
python3 benchmark_coral_efficient.py --model efficientnet-edgetpu/efficientnet_lite0_int8_1.tflite --label ImageNetLabels.txt --input fruit.jpg --runs 1000
The results from this version are actually quite a bit slower, in spite of the model being smaller. However, it's more accurate! The only thing it sees in this image is a banana:
Elapsed time is 180.288102787 ms
banana score = 0.72265625
Here's a quick preview of speed vs price for the Google Coral and its competitors. If you want to know how the MaaXBoard and Rasbperry Pi performed, the full benchmarking article is here.