Unlocking Vision Magic: Object Detection with TinyML

Welcome to our two-part blog series where we delve into the world of object recognition using the Nicla Vision board. In this first part, we'll go through the process of setting up our Nicla Vision board, collecting the dataset, and labeling the data. Then, in the second part, we'll see the training process and learn how to load our trained model onto the Nicla Vision board seamlessly.

Before going deep, we will first understand some basics like Firmware. Firmware is the low-level software that controls the hardware of a device. It's like the brain of your device, telling it how to function and interact with the world around it. Since we will be using edge impulse for the training process, we need to replace the existing firmware with firmware that can interact with edge impulse. This dedicated firmware can be downloaded from the following link: https://cdn.edgeimpulse.com/firmware/arduino-nicla-vision.zip

Once the firmware has been downloaded and the folder has been unzipped, the user should connect the Nicla Vision device to their PC with a USB cable. When the Nicla Vision device starts glowing, the user can run the bash file found within the unzipped folder. This bash file contains the scripts and commands to configure and initialize the Nicla Vision device. Running the bash file initiates the required actions or configurations for the Nicla Vision device.

Next, launch the OpenMV IDE and insert the following code. This code is designed for capturing images from the Nicla Vision board, which will subsequently be utilized to construct the dataset.

import sensor, image, time
sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.set_vflip(True)
sensor.set_hmirror(True)
sensor.skip_frames(time = 2000)
clock = time.clock()
while(True):
clock.tick()
img = sensor.snapshot()
print(clock.fps())

In the IDE, find the "connect" symbol in the lower-left corner. Click it, and a window will appear, allowing you to install the latest firmware. You can skip this step if you've already installed the firmware. However, if an update is necessary, select the "Install the latest firmware (v X.X.X, X-varying)" option but avoid choosing "Erase internal file system."

To execute the code, the user should press the triangular button located in the lower-left corner. This action will start the camera, and the captured frames will be displayed in the IDE.

Now, we will go through the process of dataset creation. Create an empty folder on the user's PC, naming it "Dataset" or a preferred name. With the snapshot code actively running, navigate to the "Tools" menu, choose "Dataset Editor, " then select the new "Dataset." folder created.

A file chooser dialog on the left will appear. Select the dataset directory, and the file will open in the code editor. Close this, as a capture script is already configured from the test.

Within the OpenMV IDE, the user can initiate the creation of a new class by selecting "New class" from the left side of the screen and providing a name for the class, for example, "pen." Subsequently, choose the "pen" class from the left side of the screen.

Click on the camera symbol located on the left edge of the screen to capture data. Multiple data points should be collected with the "pen" object within the camera's field of view. If a new class, such as "eraser, " needs to be created, the user should follow the same process mentioned above.

Edge Impulse:

In the process of setting up an Edge Impulse project, the user should access the dashboard and opt for the Bounding Box labeling method. It's worth noting that there's no need to establish a connection between the Nicla Vision device and the PC for this step.

10. Navigate to the Data Acquisition section and upload data from the "Dataset" directory. Ensure that the data is uploaded to folders for each class. Additionally, choose the preferred Training-Testing Split ratio, with 80% for training data and 20% for testing data.

11. Within the Data Acquisition page, proceed to the Label Queuing section, where bounding boxes should be drawn around the objects in the training data.

12. Moving on to the Impulse Design section on the left side, select "Create Impulse" and include Edge Impulse's Processing and Learning blocks.

13. Under the "Image" category on the left side of Edge Impulse, make sure to save the parameters.

14. In the "Object Detection" section, initiate the training process.

15. Finally, visit the Deployment section and search for the OpenMV library, then proceed to download it.

Creating the Dataset:

16. After downloading the "OpenMVlibrary, " it's essential to unzip the folder and copy the "labels" and "trained.tflite" files. These folders likely contain information about the labels and the tensorflowmodel.

17. If the Nicla Vision device is currently connected to the PC, it should be safely disconnected and then reconnected. Upon reconnection, a Nicla Vision pop-up window will appear. Within this pop-up, the user will need to paste the two previously copied files, ensuring that the Nicla Vision is equipped with the required data for object detection.

18. Now create a new file for the object detection code to be placed. Copy the "detection.py" code from the "OpenMVlibrary" folder that was previously unzipped and paste it in the new file.

19. Finally, run the code within the OpenMV environment. When the Nicla Vision Camera scans its surroundings, it will be capable of identifying pens and erasers. The results of this object detection process can be viewed in the serial terminal of OpenMV, providing real-time feedback on what the camera sees and recognizes. This system is now set up for object detection with the Nicla Vision device.