AVR IoT Bird Watch

Note:

I failed to submit my project in time as I had an issue with my machine learning model setup requiring me to restart and I wasn't able to get it before the deadline. I will send back the associated board but posting my in progress contest sheet and I will continue with my bird feeder model with a different setup once I return the board. Apologies again, I really thought I could get it in time but I misjudged the compression process especially with regard to the additional birds (I'm thinking that perhaps I'm struggling to go down further as I may be losing too much contest with the full 540 birds).

Story

I started this project with an intent to create a machine learning device to alert when dogs entered the dog park such that a community could get an idea of what dogs existed. I was granted a board by AVR for this purpose and tried my best to accomplish it. Unfortunately I ran into several issues during my time trying to build a model for that though as the dataset the primary dataset for dog related classification, the Stanford dogs dataset, has quite a few issues. For example I noticed during testing that the dataset contained random color pop and black and white images: https://github.com/Timo614/machine-learning/blob/main/stanford_dogs_b_and_w.ipynb

Given this I switched gears to classifying birds as I found a dataset that was of high quality making the process easier: https://www.kaggle.com/datasets/gpiosenka/100-bird-species One thing to note: the dataset includes some humans in a folder called "Looney Birds" which should be excluded to avoid any issues there.

Model Compression Experimentation

I knew the birds model would be too large for TinyML if I used a larger base model like EfficientNetB0 but I also wanted to be able to classify a larger number of them. As such I decided I would go forward with the EfficientNetB0 approach and worry about / figure out a way to compress the model after it was done. After several months of work and experimentation I devised a way to compress the model without losing much accuracy.

Before shrinking at 224x224 with heatmaps: https://github.com/Timo614/machine-learning/blob/main/birds-tflite-test.ipynb

Post shrinking to 96x96 with heatmaps: https://github.com/Timo614/machine-learning/blob/main/birds-tflite-test-96-x-96.ipynb

I realized though The caveat here is unfortunately I used the splits from the original model such that there are only 5 of the birds in the test and validation groups. My model is accurate to the validation and test sets (the latter which was kept pure) but given how few birds there are to verify against I can't know for sure if it would be a good model.

As such I began the process anew with a proper 80/20 split. I was able to compress the image size down to 80x80 this time while keeping 90% accuracy albeit it's still a new process for me so I may not have it refined well. After that I began the process of shrinking the model bulk. It is proceeding but unfortunately is taking far too long and given I just restarted my process this week I wasn't able to get it done in time for this project.

Problem

As birds migrate it's not always clear what birds may be present in a given location or city at a given time. Knowing what birds are visiting may help plan for and protect endangered species and their habitats by allowing researchers the informatin they need to act quickly.

Generally speaking bird classification is not something that is often done on a tiny microcontroller like the Seeed Xiao ESP32S3. There are many reasons for this but primarily:

• Image size affects the RAM of the program space so larger images require larger space and as such they can quickly outgrow the device capabilities
• Parameter counts affect the size of the end model and these devices have limited file storage space
• Most base models are designed for larger devices

By performing classification at the edge like in this bird classification model one can reduce how often the end device needs to speak with a central server. A device could operate in a remote area gathering data for long periods of time only needing to be collected when it pings that it is becoming full or that a specimen of importance was identified.

Solution

I've created a bird classification model for the purposes of identifying when birds come to bird feeders. The idea for this particular project is to pair it with a AVR-IoT to be able to send these notifications out. I chose a bird feeder as the enclosure for this device as I felt it would give the highest chance of attracting a bird for taking the photo with. A commercial product using the Xiao ESP32S3, AVR IoT, and the bird feeder model would likely be much slimmer and designed for mounting in the wild such that it could capture a more diverse body of birds than those that eat at a feeder.

Hardware

Connections

The connection for this project is fairly straightforward. The Xiao ESP32S3 is powered by the AVR IoT which has its power given by the lithium ion battery. The Xiao ESP32S3 communicates over serial with the AVR IoT allowing it to send the MQTT message in response to inferencing results. To make the process easier I used a Xiao Expansion board which allowed me to plugin my Xiao ESP32S3 and then use solderless breadboard wires to make my connections.

Software

This project uses two boards as mentioned previously the Xiao ESP32S3 and the AVR IoT. Arduino is used to setup both boards. They serve different purposes, however.

In general this is a pretty simple project in terms of the Arduino code.

The Xiao ESP32S3 has a tilt sensor which it uses to determine if there is movement on the device. This could be wind, a bird landing, or perhaps something else entirely. After this, inferencing is triggered and a list of prediction scores are returned. The top score from this list is examined and if the probability of it being that bird surpases 50% a photo is stored to the SD card and a serial message is triggered with the bird name to the AVR IoT.

The AVR IoT is simply setup to wait for serial messages containing a bird name and then promptly reporting this to a waiting AWS server which logs the time, location, and bird type for further analysis.

Xiao ESP32S3

To begin with you'll need to get the Xiao board configured for use with Arduino. A complete setup guide for the board is outside of the scope of this guide and would serve to get out of date fairly quickly in the event of updates. As such please refer to this [setup guide](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/#software-preparation) instead.

Bird feeder setup

http://www.kerinlovett.com/strokes/2024/3/31/bird-feeder-id

My wife Kerin worked on the bird feeder for me designing a neat bird design and coloring it. She picked it up from Michaels where you can find them for a fair price.

With the feeder now setup with the AVR IoT device, battery, and camera it can be left to its devices and examined for further use as MQTT messages come in.

Future / Next Steps

Unfortunately most of the time for this project has gone into my bird model so I wasn't able to create a nice experience for the post detection scenario. Ideally I think having a community based map where multiple users all send their detection data could be useful and help with understanding bird movements across the world.

In addition it is unfortunate that my updated model was not ready in time for the project. I regret not realizing that my initial setup was not ideal for a machine learning project as it was one of my first models.