Monitoring retail checkout lines with Renesas RZ/V2L kit

No one likes waiting in lines! Research shows that people wait in lines three to five days a year. Long queues at supermarkets cause shopper fatigue, reduce the customer experience and lead to carts abandonment.

Existing surveillance cameras can be used in stores near the queue area. Using Computer Vision models, the camera will be able to understand how many customers are in a queue. When the queue reaches a certain threshold of customers, this can be identified as a long queue, so staff can take action and even suggest that another counter should be opened.

For this project, I developed a solution that can identify when a long queue is forming at a counter. If a counter is seen to have 51 percent of the total customers then the system flags this as a long queue with a red indicator. Similarly the shortest serving counter is indicated with a green indicator signaling that people should be redirected there.

I used Edge Impulse to train a YOLOv5 model that can detect people. You can find the public Edge Impulse project here: Person Detection with Renesas DRP-AI. To add this project to your Edge Impulse projects, click “Clone this project” at the top of the window. Later, I deployed this model to the Renesas RZ/V2L Evaluation Board Kit and created a Web application that uses the camera feed and model results.

Data preparation 📂

For my dataset, I sourced images of people from public available datasets such as Kaggle. This ranges from images of people in retail stores and various environments.

In total, I had 588 images for training and 156 images for testing.

With 744 images it would be tiresome to draw bounding boxes and give a name for all instances where a person appears. Edge Impulse offers various AI-assisted labelling methods to automate this process. In my case, I chose YOLOv5 and it was able to annotate more than 90% of people. To use this feature, in the Labelling queue you can select "Classify using YOLOv5" under "Label suggestions".

Impulse Design 🧠

An Impulse is a machine learning pipeline that indicates the type of input data, extracts features from the data and finally a neural network that trains on the features from your data.

For the YOLOv5 model that I settled with, I used an image width and height of 320 and the "Resize mode" set to "Squash". The processing block was set to "Image" and the learning block set to "Object Detection (Images)".

Under "Image" in Impulse design, I set the color depth of the images to RGB and the next step was to extract features.

Here on the features tab we can see the on-device performance for generating features during the deployment. These metrics are for the Renesas RZ/V2L(with DRP-AI accelerator). The Renesas RZ/V2L Evaluation Board Kit was recently supported by Edge Impulse. This board is designed for vision AI applications and it offers a powerful hardware acceleration through its Dynamically Reconfigurable Processor (DRP) and multiply-accumulate unit (AI-MAC).

Currently, all Edge Impulse models can run on the RZ/V2L CPU which is a dedicated Cortex A55. However, so that I can benefit from the DRP-AI hardware acceleration, I chose a YOLOV5 model. Note that on the training page you have to select the RZ/V2L (with DRP-AI accelerator) before starting the training in order to tell the Edge Impulse studio that you are training the model for the RZ/V2L. This can be done on the top right in the training page or by changing target device in the Dashboard page.

I used 100 training cycles with a learning rate of 0.015. It is however advised to train a YOLOv5 model using more than 1500 photos per class and more than 10, 000 instances per class to produce a robust YOLOv5 model.

After the training process, I got a precision score of 88%. Precision is the number of True Positives divided by the number of True Positives plus the number of False Positives.

Testing the YOLOv5 model

After training the model, I did a test with the unseen(test) data and the results were impressive, 85% accuracy.

In this test sample which has a precision score of 64%, we can see that the model was able to detect 6 out of 8 people in the image.

Deploying the model to Renesas RZ/V2L Evaluation Kit 📸

The Renesas RZ/V2L Evaluation Kit comes with the RZ/V2L board and a 5-megapixel Google Coral Camera. To setup the board, Edge Impulse has prepared a guide that shows how to prepare the Linux Image, install Edge Impulse CLI and finally connecting to Edge Impulse Studio.

To run the model locally on the RZ/V2L we can run the command edge-impulse-linux-runner after edge-impulse-linux which lets us log in to our Edge Impulse account and select the cloned public project.

We can also download an executable of the model which contains the signal processing and ML code, compiled with optimizations for the processor, plus a very simple IPC layer (over a Unix socket). This executable is called .eim model.

To do a similar method, create a directory and navigate into the directory:

mkdir monitoring_retail_checkout_lines && \
cd monitoring_retail_checkout_lines

Next, download the.eim model with the command:edge-impulse-linux-runner --download modelfile.eim

Now we can run the executable model locally using the command:

edge-impulse-linux-runner --model-file modelfile.eim

We pass the directory and name of the downloaded file modelfile in the command.

Now, we can go to the URL provided and we will see the feed being captured by the camera as well as the bounding boxes if present.

A smart application to monitor checkout lines 🛒🚶🏻‍♀️

Using the eim executable and Python SDK for Edge Impulse for Linux, I developed a Web Application using Flask that counts the number of people at each queue and computes a distribution of total counts across the available counters which enables identifying long and short queues.

The application shows which counter is serving more than 51 percent of the total number of people with a red indicator. At the same time the counter with the lowest serving customers is shown with a green indicator signaling that people should be redirected there.

For the demo I obtained a public available video footage of a retail store which shows 3 counter points. A snapshot of the footage can be seen below.

The application obtains the bounding boxes coordinates ( x and y) for each detected person using the Edge Impulse Python SDK. From the video footage I identified where the checkout lines for the 3 cashiers appear in the x coordinate. These form 3 regions of interest. Next, the Python script checks how many bounding boxes' x coordinates appear in each region of interest and increments the person count accordingly.

Note that if you are using another surveillance footage then you will need to identify the location of the region(s) of interest.

You can clone the public GitHub repository to your RZ/V2L board. The installation steps are in the repository. You can run the Flask application or the binary executable built with PyInstaller for AARCH64 platform.

Afterwards, I projected the demo footage from a monitor to the 5-megapixel Google Coral camera attached to the RZ/V2L board. The colored circles on the right are responsive and they can be red, green or orange. Red implies that the cashier has 51% of the total number of people in the queues while green shows the cashier with the lowest customer(s). Orange shows the cashier who has a count that is in the middle of the highest and lowest. This enables us to easily identify when long queues are forming at a cashier.

Conclusion

For a more detailed explanation of this project please check the documentation on Edge Impulse. The documentation also has the inference classification times that I obtained between using the RZ/V2L CPU (Arm Cortex-A55 cores) and RZ/V2L DRP-AI accelerator.

Implementing Computer Vision models comes with its challenges. Some challenges can be identify accompanying guests, distinguish customers who are in the queue and not shopping, among others.

A future work for this project can be to predict queueing times for different days and particular times of the day.

With Edge Impulse, developing ML models and deploying them has always been easy. Knowledge on machine learning and microcontrollers is not known by everyone but the Edge Impulse platform provides easy, smarter and faster tools that we can use to create build edge ML solutions quickly.