Solar LoRa'r is the evolution of my previous bird detection model and setup. The core of the project was to create a mostly compact circuit board and shrunken TinyML model that could be used to detect many birds at high accuracy at the edge. I believe it is important for researchers to be able to detect bird migration patterns and further study them in their environments. Their environments, while often encroached upon by humans, do not always have readily available access to WiFi networks. It's a custom circuit board I designed with the goal of providing researchers a tool to create a network of edge AI cameras they can then remotely collect information on and routinely access for associated images.
NoteUnfortunately I was unable to complete the entire project in the timeframe available. I retrained the model from scratch to the shrunken model but I was unable to modify the shrunken model to remove the normalization layer built into EfficientNetB0 such that I could completely int8 quantize it. Given the model is 1.5MB and the Grove Vision AI V2 requires models of <= 1MB it's too large to fit otherwise and it would require quantization to run on the edge device as well. I'm including everything up to this point but I'm aware it won't win the competition given this unfortunate state. I did start retraining on EfficientNetLiteB0 as a base but in the twelve hours since I realized the quantization issue I only managed to get 80% accuracy across the 523 outputs so I'd likely need another few days of training to reclaim the 90% or so the other shrunken model showed (it's a large dataset and my method, while faster now than before, does take a small amount of time).
The timeline for this competition was a bit aggressive so despite my focus on trying to get everything together I failed to meet that window. I could have just used my older model but I wanted to ensure that I showed the process worked and that large output models like this could be made for edge devices so I kept the focus on that.
Things done well:
- I tested the board across all of the association functionality needed here confirming at least in theory things would have worked out had I been able to get the model quantized
- I designed and printed an enclosure for the device trying to minimize water leakage
- I did train a shrunken model that was very close to working here and my process involved keeping the holdout from the start to avoid any leakage proving it does do a good job
Imagine if a researcher could setup a base station at their cabin and then enter a location of interest setting up multiple Solar LoRa'r devices. This would create a network of devices designed to be eyes for them at all times.
The device triggers on motion waking from deep sleep, activating its boost circuit for 5V, and then its Grove Vision AI sensor over i2c. It takes a photo, classifies it, and returns data about its detection. If the likelihood of a detected bird is high enough it then saves the bird's image to its onboard SD card and triggers a LoRa transaction over its onboard radio sending an alert to the base station.
The researcher can see these alerts allowing them to see updates as they come in and high level trends. Later this researcher can visit the installation locations and uses a magnet card to trigger a reed switch on the board which triggers another interrupt but this time puts the device in bluetooth sync state allowing them to download images the device had captured to a mobile app.
Solar LoRa'rThe solar LoRa'r is a custom PCB designed with that user story in mind. With the help of a NextPCB sponsorship my vision was fulfilled giving me this very interesting board. NextPCB made the whole process very easy with multiple confirmations as issues arose with the PCBA process. For example I had several footprints in my BOM which were custom from my EDA software and the team reached out to ask me about my intent. They also procured the Xiao NRF52840 devices ordering them from Seeed which I didn't expect. The PCB came exactly as I had designed and I was able to get up and running very quickly.
I opted to use Arduino for programming as it made interfacing with the Seeed Grove Vision AI V2 module a lot easier. Zephyr was struggling with the LoRa modem I selected while the Arduino library worked without issue once I sorted out a hardware issue mentioned in the errata below.
The board consists of several components:
- Xiao NRF52840 - the microcontroller from Seeed Studio NRF52840 MCU
- E22-900M22S - the LoRa modem utilizing the SX1262 transreceiver
- DW06D - battery protection IC
- BQ24210 - solar charging IC
- MT3608 - step up converter IC
- SY6280AAC - power distribution switch
- MCP23008 - 8 bit pin expander
- TF card slot - SPI communication with an SD card
- HY2.0 "Grove" connectors for I2C to interface easily with Seeed boards
- Reed switch - magnet based input
- AM312 pinout - for connecting a PIR device
- On / off switch for battery and DW06D reset button on battery change or power state toggle
- Test pins: 3V3, 5V, VBAT, VSolar and their associated GND pins located across the board near their components
I've designed several PCBs but this was the first I've used LoRa, the SD card, Battery ICs, Solar Charging ICs, and the pin expander. So I was nervous a bit going into this project but luckily it mostly turned out alright with minimal errata needing modifications for a v2.
Given the power elements were new to me I relied on DFRobot's excellent schematic for their DFR0059 to get an idea on how to best connect the DW06D, MT3608, and the SY6280AAC -- I believe this contributed heavily to reducing issues with my schematic as while I did consult the manuals I was able to confidently know my structure was used in a battle tested board. I didn't find a good source for comparing my schematic of the E22-900M22S and BQ24210 so for those I was a bit more unsure of but luckily my parsing of their datasheets was sufficient.
I'm still learning PCB design and I'm self taught yet I feel like this one was a huge success and I added quite a few new tricks to my capabilities. I'm very grateful of NextPCB giving me the opportunity as I don't have the financial freedom to experiment at this level normally.
If I ever have a chance for a future revision I'd consider using the nRF52840 directly as I feel like the Xiao version while great for prototyping made things a bit more difficult given the lack of pins (the plus one may be helpful here though for that) and it would make positioning elements of the board a bit easier as I could center the module (vs needing to route everything from a corner).
Errata:
There were unfortunately several issues with the PCB caused which will need to be corrected in future revisions. Given I had applied late in the contest toward the last week I only had a little over a week to design the board from scratch so I made a few minor mistakes.
- For the battery input I setup the pins in reverse of what all battery packs I've purchased to date use. To fix this I desoldered the battery terminal (unfortunately cutting traces in the process I had to fix) and used a different component where its power pin aligned with the footprint.
- For the LoRa busy pin -- I used the MCP pin expander initially which is over i2c. It's too slow though for the busy pin so I had to repurpose the RX pin from the Xiao board. Unfortunately making UART and serial debugging an impossibility. I'd likely want to use the NRF52840 directly or perhaps try out the Xiao's new plus line in future revisions as while the Xiao board is awesome the lack of pins can be a problem.
- I didn't add a pull-down resistor to the EN pin of the boost enable MCP output. The MCP is left floating during boot so this could likely be a problem. I noticed it before I started using the component.
- I created a mosfet based power system for the SD card to minimize power but I noticed in practice it still produced a running voltage even when I didn't provide power. It appears the SD card was being powered via parasitic power from the SPI lines. I'd like to deal with this issue in a future revision if possible.
- AM312 is missing proper silkscreen notices for which pin relates to each header forcing user to refer to board design / needs additional labeling.
- I noticed the boost circuit still passes on the existing battery voltage when not enabled (just not the 5V) which in turn means my i2c circuit is also being powered at this time. I likely should create a p channel mosfet sort of setup where the 5V i2c is completely cutoff when the boost is not enabled to avoid unintentionally still powering anything attached.
Notes:
- Not a problem per se but I did notice I had placed the SD card capacitors pretty far away from the power pin. In the future I'd like to reposition those so they are closer.
- Additionally not a problem (from what I can tell / my use so far) I noticed I should have routed the SPI lines under the board vs directly under the LoRa module to ensure no issues. To date I haven't had issues but it was not ideal how I routed it.
- The cutouts for the VBAT soldering on the microcontroller work but are very difficult to work with. I struggled to solder to the pin. In retrospec putting some solder on the VBAT pads prior to affixing the microcontroller may be a good idea. Given I had soldered my board prior I used a resistor and its leads to push down the solder paste which I then used my heat system to bring up to temperature. I found this allowed me to avoid flooding the cavity while ensuring the pin has enough solder around it to make a good connection.
This video compilation showcases my testing procedure. For this project I created several sketches to test specific functionality (referenced with more detail below in the software section). Additionally I tested some elements in isolation prior to using with the full set: like testing the battery prior to soldering pins on the microcontroller to power via it and so on.
Software for the BoardInitially I had hoped to use Zephyr RTOS for the board. I did create a custom overlay for it and began testing components such as the SD card. Unfortunately the LoRa modem was not easy to get working there for me. I may have just failed to use the right configuration but it was a struggle. Initially I also foolishly had used the pin expander for the LoRa busy and RXEN lines which caused issues. I realized with the Arduino example I didn't need to use RXEN at all given I was relying on the DIO2 to TXEN pin setup. For the LoRa busy pin I just rerouted to the RX pin on the Xiao (making my UART input useless but allowing me to start sending transactions).
For the board setup my approach was to create individual Arduino sketches for each piece of functionality, get them working, and then combine them as part of a final sketch.
As such I created test scripts using a Grove 1.12" OLED i2c display which helped to ensure everything on the board was properly setup as I tested each piece individually. I've included these in the Github repository under the test-scripts directory in case elements of the setup need to be further debugged. Use the 3V3 i2c as the 5V is for the Grove AI v2 sensor.
- arduino-boost.ino - tests boost functionality with 60s off and 60s on
- arduino-interrupt.ino - tests MCP interrupts for AM312 and reed switch and MCP interrupt line on D1 pin
- arduino-lora.ino - tests sending a TX on interval, useful with another device listening or a Flipper Zero for monitoring 915MHz
- arduino-sd.ino - tests connecting and reading file system state from SD card
Given I was unable to get the model working and quantized I didn't end up building a working final script that combined all elements. I'd have probably needed another few days to wrap things up but given the project already had an extension I wasn't expecting further delays here. My goal was to additionally build out a bluetooth app to pull down images and a base station app to show as images are taken for given registered cameras.
EnclosureFor the board enclosure I designed a 3D model with various elements to make the printing process easier.
- The bottom half contains screw holes which allow the board to couple securely with the top half. It also contains built in standoffs allowing the SoRa LoRa'r to be fastened securely to the enclosure.
- The top half was designed for heat set insert use allowing the device to be opened and closed more easily.
- A PG7 cable gland was used to allow the solar panel wiring to exit the device
- A hole for the AM312 exists allowing it to poke out of the enclosure, the hat from the device is then attached securing the device and keeping the enclosure sealed
- A cavity exists for an acrylic circle to be inserted (and sealed with epoxy
- Additionally a 3D print element exists for fastening a camera board to the cavity allowing the camera to avoid reflections
The device has a sleek look while maintaining its core functionality. Compared to my previous setup with the other birdhouse project I did last year the addition of the custom PCB helps to reduce component count.
Model CreationThe model utilizes my model shrinking technique I've perfected over the past few years. It took me a little over a week to train the model and distill it into the shrunken model. I used Tensorflow for training the model on Google Colab T4s and later on my local machine once my resources were exhausted (ensuring I used the same splits though I had initially started with).
You can see the compressed EfficientNetB0 version on Kaggle as a notebook. This version is 1.5MB and unfortunately is unable to quantize to int8 quantization. It does demonstrate the high accuracy across the holdout sets and my split methodology. In the future I'll post the EfficientNetLiteB0 version as a subsequent notebook version showcasing the int8 quantization on that model but it's outside the scope of this competition as it wasn't finished in time.
My methodology was to:
- Create a splits.json file that contained the chosen splits and was reused across all training sessions reducing any chance I'd unintentionally modify the splits midtraining and allow leakage in.
- Create new splits from the training data as the dataset I used had only a few validation and test images and I wanted to ensure the model generalized well across the various birds.
- Limited the subset of training and validation to be an equal number of birds across each species to reduce any one species from overwhelming the training. Placed extra images in the test set for confirmation purposes post training.
- Model was trained using EfficientNetB0 and then knowledge transfer distilled into a partial model using teacher/student distillation and further refined with my model shrinking technique.
- After training completed the hold out test set was checked for accuracy, additionally the validation and test sets from the original split (unused during training as additional hold outs) were checked. Then quantization was performed and the various data subsets were used to check the post quantization accuracy drop. Note the test set was not touched prior to this point and was not used for any future fine tuning to avoid leakage.
Note:
- Unfortunately Gerry made the dataset private at some point as you can see though via the wayback machine it was CC0 public domain released so I am using the local copy with that license. It does appear another user has uploaded the dataset to Huggingface it's possible to experiment with this dataset still for others just less convenient than on Kaggle.
- Gerry included a joke category of humans and had one bird of the 525 with extreme image dimensions for most images so I used a 523 subset of the 525.
- I previously trained the same model for my 97% F1 score model but wanted to restart the process as I've refined the process and that model was rather large at 544 thousand parameters. Given the interest of time I stopped training this model when its accuracy hit 96.20% accuracy on the validation subset while the other model I trained for a much longer time until validation plateaued.
Demonstration: This work demonstrates 523 bird species getting detected by the AI. With a subset specific to a region smaller hardware could be used but the goal was to demonstrate using a large selection here allowing researchers to monitor across significantly varied species without having compromise in terms of model size.
Whyismyapproachnovel? Most models in the TinyML space compromise on either model size or accuracy.
What's Next?While this project has come to a close there are many ways it could be improved further.
- Fixing errata for the board / switching to the nRF52840 directly or the Xiao plus model for more pins
- Adding additional sensors -- light intensity, temperature, etc. and useful helper components (RTC with battery)
- Enhanced bluetooth and base station applications
- Wrapping up work around EfficientNetLiteB0 training and its quantization
- Creating a full working prototype as I failed here to do that in the time frame
- Relay units to listen for messages and rebroadcast for longer range support
- Gerry for providing the initial dataset compiling this wide selection of high quality images
- NextPCB for the sponsorship of the PCB
- DFRobot for the awesome schematic references
- Thank you to my kids for recording with my camera as I balanced my multimeter leads for my dev testing videos
Teacher model trained from EfficientNet-B0 - Apache v2.0 License, original weights from TensorFlow Hub
Changes:
- Replaced classified head
- Fine-tuned on Bird-523 dataset
BIRDS 525 SPECIES (Huggingface Mirror) - Creative Commons Zero License
Changes:
- Removed "LOONEY BIRDS" - joke human category
- Removed "PLUSH CRESTED JAY" - folder containing odd sized images
I started shrinking models almost two years ago. Without going into too much detail there was a TinyML UN ITU competition which I had heard about and it got me interested at seeing if I could derive a means to use a fine tuned model on TinyML hardware. At the time I was relatively new to machine learning and I failed miserably but became obsessed with shrinking models. I then spent two months focusing on the problem until I had a breakthrough. I've since improved the process.
There was a period where I was trying to drum up interest in the work so had created several open source models (I failed to drive any interest in the work as I don't have a background in academia and the community has insular tendencies).
EfficientNet shrinking:
- https://www.kaggle.com/code/timothylovett/flower-102-tinyml-70k-params - a recent demonstration utilizing the Oxford 102 flower dataset and getting near MobileNetV2 levels of accuracy on holdout with only 70k parameters
- https://www.kaggle.com/code/timothylovett/birds-523-shrunken-model-500k-params-97-29-f1 - my previous work with 500k parameters on the dataset but less rigorous split methodology
- https://www.kaggle.com/code/timothylovett/plant-disease-shrunken-efficientnet
- https://www.kaggle.com/code/timothylovett/caltech-vehicles-tinyml-sized-f1-100-grayscale - an experiment in further shrinking a trained model across the RGB channels where color is less likely pivotal to the accuracy
- https://www.kaggle.com/code/timothylovett/f1-97-tinyml-animals
ViT shrinking:
- https://www.kaggle.com/code/timothylovett/99-f1-shrunken-vit-model - experiment in ViT shrinking proving the transformer architecture could be shrunk as well with the approach
Custom model shrinking:
Future:
- Currently training a Tiny LLM using llama-like architecture on SimpleBooks vocab so hoping to try my hand at shrinking that time permitting as I feel like there will be more interest if I showcase the technique works on that domain.
Comments