Bee Wilkerson's Machine Learning Shootout Shows You the Model You Need on Your Raspberry Pi 4
"The model almost matters more than the framework for performance," Wilkerson notes in a detailed investigation you can try for yourself.
Engineer Bee Wilkerson has some information that will prove useful to anyone planning to use a Raspberry Pi 4 single-board computer for machine learning at the edge: an impressive shootout of different frameworks and models for computer vision work, with and without the benefit of a Google Coral Edge TPU accelerator.
"I build bots to train and entertain my canine housemate," Wilkerson explains of the project's origins. "Robotics automation routines like tracking and following your pet around the house require that we be able detect when a dog or cat is in the field of view (FOV) and, even more important, where in the FOV — how many degrees off center — the detected object is. The higher the performance of object detection correlates to a higher chance of successfully tracking and following an object."
While it's entirely possible to roll a randomly-chosen framework and model onto a Raspberry Pi 4 and achieve usable results, Wilkerson was looking for hard data on getting the best performance — and set about building a benchmarking framework which compares multiple models and frameworks. The key metric: performance on-device, rather than based on offloading the work to a more powerful external system.
"It's true that for most applications around the house you could probably just set up a base computer that was much more powerful. All the computer hardware on the bot would do is send sensor data and video back and receive motor commands in," Wilkerson admits. "'But that's not cool!' you yell? I'm with you. I want it to be fully autonomous and self contained."
The benchmark results themselves are, perhaps, surprising: "As it turns out," Wilkerson writes, "the model almost matters more than the framework for performance! In case you are wondering if upgrading to a [Raspberry Pi 4 Model B] w/8GB will improve your performance, the answer is no. Put that money towards a Coral [Edge] TPU."
That Coral Edge TPU, a USB-accessible accelerator for machine learning work launched by Google as part of the broader Coral range, is also the only way Wilkerson found of getting the benchmark workload — object recognition on a VGA-resolution video feed — above 15 frames per second, achieving an average of just under 24 frames per second when using a quantized SSD MobileNet V1 model via TensorFlow Lite. The fastest accelerated performance, meanwhile, reached a little over 14 frames per second using the same model — while the YOLOv5 model running via PyTorch struggled to average over a single frame per second.
Wilkerson's work has been published on GitHub, along with a shell script for running the benchmarks yourself if you want to double-check the results.