Software apps and online services
When I was a kid I had really bad reactions to poison ivy: painful, itchy rashes that would require visits to the doctor for strong medication. Poison ivy can be tricky to identify and is found in places where it can be difficult to get mobile phone reception (forests, hiking trails, remote areas). Wouldn't it be cool to have an edge device that could identify poison ivy (or other bad stuff) that is low power and doesn't require an internet connection?
I started tinkering around with TinyML in the past year and recently bought a Pico4ML from Arducam. I was intrigued because it came with a Raspberry Pi RP2040 MCU, monochrome camera, small LCD display, microphone, and an IMU all in one small form factor. I slowly started playing around with the demos to understand what it could do. I also started tinkering with Edge Impulse creating image, audio, and motion classification models for a variety of MCUs.
When the "Eyes on Edge: tinyML Vision Challenge" was announced I decided to try to put my new-found skills to the test!
I started with Edge Impulse and taking photos to populate my image classification model. I developed 3 classes: poison ivy, plants that weren't poison ivy, and unknown (photos of random stuff that weren't plants). I followed the guidance at Edge Impulse (https://docs.edgeimpulse.com/docs/image-classification) on how to develop the model. I used my daughter's small digital camera to take photos and did bulk uploads of the images to my Edge Impulse project.
After that I designed the impulse. To align with the constraints of the Pico4ML, I selected 96x96 monochrome images and used the default image and transfer learning blocks suggested by Edge Impulse.
I selected MobilenetV1 0.2 for the model, again because of the memory constraints of the Pico4ML. While not ideal, and not the most accurate, it was good enough for the application and the hardware. Twenty neurons seemed to yield the best result. I also selected data augmentation for a more diverse dataset.
With these parameters selected, I trained the model. Results below:
Not too bad for a couple hundred images! This is certainly not the best model, and there is some confusion on plants that look like poison ivy but aren't. I think with additional quality images of poison ivy and similar plants (ie thorn bush leaves and blackberry leaves) the model could be improved even further.
Once the model was built and tested, I exported it to a C++ library to integrate into my project. Now, I didn't get this done in 1 iteration. There was a lot of experimenting in number/quality of photos to use and the model type (the first model I used was too large for the Pico4ML and it was running out of memory). I didn't realize how many iterations I would have to go through!
I did my development on my Raspberry Pi 4B, because that seemed to be the easiest interface. I worked off of the code/instructions from the Pico4ML github. I used the person_detection demo as a starting point so I could understand how images were received and how they were passed to the ML algorithm. This is where I spent *a lot* of time, and I thank the Edge Impulse team for sticking with me and answering questions for me on their forum.
The demo used a tensorflow lite micro model, so the interface was very different from the Edge Impulse model in terms of setup and inference. For those that are legit coders, it's probably not a big issue, but for someone who hasn't coded in over a decade, it was a long slog to get the Edge Impulse model working properly with the hardware (see note above how my original model was too big for the hardware and was causing memory issues).
Once I was confident I had it working, my live results were not as impressive as I had hoped, so I continued to refine the model until I felt it was "good enough". The photo below shows the image received from the camera, and the probability that the image is poison ivy. If the probability is greater than 60%, the number turns red: beware!
This was a fun exercise and I really feel like it is at the bleeding "edge" of TinyML. The Pico4ML is low-power, so I'm able to power it off of a simple 5V mobile phone battery charger. My USB cable has an on/off switch, so it's a simple click on, classify the image, then click off. It's very effective for remote areas where power and internet are not readily available.
Future extensions of the project could be to classify other "bad" stuff that you would want to know about in remote areas while hiking of camping: "is this berry poisonous? what about this mushroom?" and as MCUs become more powerful better models with better input can be used (with color images instead of monochrome for example). Other improvements could be multiple classes (instead of the binary classification I have here) and eventually object detection!
This is my first project submitted to hackster.io, and I hope you enjoyed it!