AI-powered refrigerator stock detection & alert system

Hello everybody!

Almost every single time I get home from shopping for groceries, I realize I didn't buy eggs because I was sure I already had enough, or I buy another bottle of ketchup to add to the two already at the back of the fridge.

I'm Hirao, and I work at Sony Semiconductor Solutions. The Raspberry Pi AI Camera was launched in September 2024, and it makes it easy and convenient to utilize AI models in edge environments. I wondered if I could use it to solve my own grocery shopping problem with a refrigerator stock management solution, and I hope that other people might find it useful too, or at least a good example of how to set up an AI vision project of your own.

For this project, I developed a stock management application for the refrigerator using the Raspberry Pi AI Camera and a Raspberry Pi Zero 2 W (I call it Pi Zero 2 W below). I used the Pi Zero 2 W because it's tiny and space is limited. The Raspberry Pi AI Camera works with all Raspberry Pi models.

In the application, I use two types of AI model. The first is the OSS model published as open source by the Raspberry Pi Foundation, and the second is a custom model created with Brain Builder for AITRIOS, provided by Sony Semiconductor Solutions to meet unique user needs. Brain Builder is a development tool that allows users to easily create AI models with just GUI operations, even without specialized knowledge in AI and with minimal coding experience. For more details about Brain Builder, please check the official site.

The application detects the produce in the refrigerator and sends stock information to me via email at regular intervals.

Setting up the project

Write the Raspberry Pi OS image

You can write the Raspberry Pi OS image to the SD card using the Raspberry Pi OS Imager (download them here). Raspberry Pi OS Imager is compatible with Windows, MacOS, Debian, and Ubuntu.For a detailed step-by-step procedure, see Getting started with your Raspberry Pi.

Set up the Raspberry Pi AI Camera

The contents of the Raspberry Pi AI Camera box are shown in the image.

• Raspberry Pi AI Camera
• mipi DSI cable for display (Standard - Standard 20cm)
• Use this cable for connecting to Raspberry Pi 4 series.
• mipi DSI cable for display (Standard - Mini 20cm)
• Use this cable for connecting to Raspberry Pi 5 or Pi Zero series.
• Small focus adjustment tool
• The tool can be inserted and twisted to adjust the focus.

Connect the Raspberry Pi AI Camera to the Raspberry Pi as shown in the image.

WARNING Note that the camera cables have front and back sides. Pay attention to this when connecting the cable.

After connecting the hardware, please proceed with the setup according to the steps in the official documentation.

Verify operation

To confirm that everything is connected correctly, you can run a sample script. If you want to try out the object detection sample script, run the following code:

rpicam-hello -t 0s --post-process-file /usr/share/rpi-camera-assets/imx500_mobilenet_ssd.json --viewfinder-width 1920 --viewfinder-height 1080 --framerate 30

The output should be as follows. You can confirm that the apple in the image has been correctly detected.

Implementation

Overview

We can use an object detection model to count the number of bottles in the refrigerator, and create a system that sends stock information by email at a set time every day. I personally love coffee and always keep bottled coffee, but I often forget to buy more... so let’s create a system that reminds us to buy it when we're running low!

Raspberry Pi AI Camera capture script

For object detection, we can use one of the sample models for object recognition installed during the setup of the Raspberry Pi AI Camera, called nanodet_plus_416×416.rpk. You can view the list of models in the usr/share/imx500-models directory.

This model was trained on the coco dataset, and one of its labels is bottle. We will implement the counting of the number of bottles using this label.

For the object detection code, we can fetch a sample script for object recognition published by the Raspberry Pi Foundation from GitHub. First, clone the script with the following command:

git clone https://github.com/raspberrypi/picamera2.git

You can verify operation with the following command.

python picamera2/examples/imx500imx500_object_detection_demo.py --model=/usr/share/imx500-models/imx500_network_nanodet_plus_416×416.rpk

The output should be as follows. You can confirm that the detection of the bottles is working correctly. (It even manages to detect the table underneath it!).

Of course, multiple detections are also possible.

WARNING On the Pi Zero 2 W, keeping the preview displayed may slow down operation. In that case, change picam2.start(config, show_preview=True) to picam2.start(config, show_preview=False).

Analyzing the detection results

Before implementing, let’s examine the output from the object detection model of the Raspberry Pi AI Camera. The output from the camera is received by the np_outputs of the parse_detections function, and the results are stored in last_detections.

last_detections contains data for each detection sample as objects of the Detection class. The data members of the Detection class are as follows:

• category: The category of the detection sample
• conf: Confidence score of the detection result
• box: Position information (x, y, w, h) of the detection result

So then the detection of a bottle can be determined by the presence of category = bottle.

Stock information retrieval system

In general, when you open a refrigerator door, the light turns on, and after closing it, it goes out after a certain period. Taking this into consideration, if the light is off, there is a possibility of incorrect AI inference due to insufficient light. However, during the period just before and immediately after closing the door, the refrigerator's light is on, and this time, from just before the door closes until the light goes out, provides the best shooting environment. Also, since the stock situation generally does not change while the refrigerator door is closed, it makes the most sense to regard the inference result just before the most recent opening and closing as the latest and most up-to-date stock information.

So, how can we obtain information about the door's opening and closing? One method could be connecting an external sensor, such as an accelerometer, to the Raspberry Pi, but in this project, we can do it using only the Raspberry Pi AI Camera. Specifically, we will analyze the pixel values of the images captured by the camera to determine the opening and closing state.

Method for detecting door opening and closing

First, photograph the refrigerator in both open and closed states and measure the average pixel values for each. Based on this, we can set an appropriate threshold and determine the open and closed states using the following logic:

• If the average pixel value is above the threshold → the door is open (and the light is on)
• If the average pixel value is below the threshold → the door is closed (and the light is off)

Then, we will update the latest stock information based on the object detection results from the images of the last few frames just before the average pixel value falls below the threshold (that is, just before the door is closed, when it is still bright).

The implementation is as follows. Please modify the following two points in the script according to your own environment:

• num_frames: The number of frames to retain for detection. Adjust this value if the detection accuracy is poor.
• img_threshold: The threshold for light on/light off, based on the average pixel values.

from collections import deque
import threading

detection_label = "bottle"
num_frames = 30 # Number of frames to retain
check_detections = deque(maxlen=num_frames) # Store detection data for the past num_frames frames
latest_frame = 0 # Store the average pixel value of the latest frame (for calculation)
current_status = None # Status used for notifications
img_threshold = 45 # Threshold for average pixel value

def check_outofstack():
    """
    Check if objects are out of stock based on detection history.
    Updates status when stock level changes are detected.
    """
    global current_status
    
    if latest_frame > img_threshold:
        Append_last_detections()
    else:
        if len(check_detections) != 0:
            current_status = max(sublist.count(detection_label) for sublist in check_detections)
    
# Check for objects in detection area
def Append_last_detections():
    """
    Append detected objects to the detection history queue.
    Extracts labels from current detections and adds them to the history.
    """
    global check_detections
    tmp_list = []
    detections = last_detections
    labels = get_labels()
    if detections == []:
        check_detections.append([])
        return 
    else:
        for detection in detections:
            tmp_list.append(labels[int(detection.category)])
        check_detections.append(tmp_list)
        return

Email notifications

For the second part of the project, let's implement a mail notification feature for our convenience. With this feature, we can receive the latest stock information from the refrigerator via email at the time we set, every day. This allows us to check stock information even when we're not at home, such as after work. The delivery time can be set by modifying delivery_time. Also, the sender's email address is assumed to be from Gmail.

IMPORTANT:To use the email feature in this application you will need to install Schedule using pip as follows:

pip install schedule

Or create a virtual environment with uv to help manage all your requirements without affecting your system packages.

Implementation of email notification

Using smtplib and email packages from Python's standard library, we can implement a simple email sending function. Additionally, we can use the schedule library for periodic execution. The content of the email can be modified as needed.

Please change the following three points in the script according to your environment:

• sender: The sender's email address (this implementation assumes Gmail).
• password: The app password for Gmail (refer to the official site for instructions to create an app password).
• receiver: The recipient's email address.

import schedule
import time
import smtplib
from email.mime.text import MIMEText
from email.utils import formatdate

# Email settings
delivery_time = "18:00"
sender = "Sender's_Email_Address (gmail)"
password = "Your_own_Password"  # For Gmail, use app password
receiver = "Recipient's_Email_Address"

def send_email():
    # Create email content
    if current_status == None:
        subject = "AICamera_demo_Stock Notification"
        body = "No stock information detected"
    elif current_status == 0:
        subject = "AICamera_demo_Stock Notification"
        body = "Out of stock!! Let's buy more！！！！"
    else :
        subject = "AICamera_demo_Stock Notification"
        body = "There are " + str(current_status) + " items remaining in stock"
    
    msg = MIMEText(body)
    msg['Subject'] = subject
    msg['From'] = sender
    msg['To'] = receiver
    msg['Date'] = formatdate()
    
    # Send email
    try:
        with smtplib.SMTP_SSL('smtp.gmail.com', 465) as smtp:
            smtp.login(sender, password)
            smtp.send_message(msg)
        print(f"Email sent successfully: {time.strftime('%Y-%m-%d %H:%M:%S')}")
    except Exception as e:
        print(f"Error occurred: {e}")

def schedule_task():
    while True:
        schedule.run_pending()
        time.sleep(1)

The code created this time can be obtained here.

Creating an AI model using Brain Builder for AITRIOS

Data collection

In this section, we will create a model to count the number of eggs in the refrigerator using Brain Builder. The training data for the eggs will be collected using the Open Images Dataset V7. A sample of the collected data is as follows. Since the training of the object detection AI model in Brain Builder requires about 150 images per label, collect 150 images for this project, and compile them into a zip file.

Sample dataset

Installing Brain Builder

Brain Builder can be downloaded from here. There is a free trial period that includes object detection (Detector) AI models, so let's start by trying that. After installation, please follow the steps in this guide to set up your environment.

Creating a project

Brain Builder manages datasets and AI models as projects. So, let’s create a new project.

• Select Create Project in the top right of the screen. A pop-up will appear, so enter any project name (this project uses Egg_Detector).

• Select Create Dataset in the top right of the screen. Set any dataset name and choose Detector as the model type.

• A dashboard will appear, where you can select Upload Data in the top right, choose the prepared zip file via Browse, and then select Start Upload. This completes the project creation and data preparation.

Training

To train a Detector model, you need to label the training data. Labeling can be performed within Brain Builder. The flow from labeling to training is as follows:

• Select the Annotation tab, and add a new class using the + button in the top right, naming it as desired (this project uses Egg).

• Create bounding boxes on the objects detected in the images displayed on the screen, and select the label. Switch images using the ＜＞ in the top right while labeling all of the training data.

• Select the Training tab and train the AI model based on the labeled data. During training, you can choose from parameters: Quick, Balanced, or Thorough.

After selecting, you can start training from the bottom right Train.

• When training is finished, you can check the training results on the Evaluation tab.

You can also verify the inference results for each evaluation data in the verification images at the bottom of the screen. After checking several samples, it seems that the model is recognizing objects correctly.

Downloading the AI model

• Select the Export tab and download the AI model you created in zip format from Download AI Model.

• Extract the zip file and check its contents. The two files to be used are as follows. These two files are used on the Raspberry Pi AI Camera, so please transfer them.
• packerOut.zip
• labels.txt

Package processing

To run the created AI model on the Raspberry Pi AI Camera, convert it to .rpk format. For conversion steps, refer to the instructions here. In this project, it is saved as network.rpk.

Capture script

The code to run the AI model created with Brain Builder can be downloaded from GitHub, published by us at Sony Semiconductor Solutions. Clone the script with the following command:

git clone https://github.com/SonySemiconductorSolutions/aitrios-rpi-model-zoo.git

Navigate to aitrios-rpi-model-zoo/models/object-detection/brainbuilder and place the previously created .rpk file there. Then, you can verify operation with the following code:

python app.py --model=network.rpk

The execution result will be as follows. Confirm that the eggs are being detected correctly.

With the operational confirmation done, let’s modify the script to notify the stock information. Since the content is similar to the OSS model, I will skip that part. The final code can be obtained here.

Experiments

Results using OSS model

Let's run the code created and verify the operation. I prepared several bottled drinks in the refrigerator and adjusted the Raspberry Pi AI Camera according to their placement. By the way, the operating temperature of the Raspberry Pi ranges from 0 to 80 degrees, so it can operate normally inside a refrigerator.

Inside the refrigerator, I arranged the items as follows.

Object Detection Results

For the case where there are 3 bottles, the execution result shows that the individual count is correctly detected just before the refrigerator door closes. (It also manages to detect the refrigerator itself!)

Notification Results

Next, let's check the notifications. I modified the code so that a notification would come with each open and close of the refrigerator for debugging purposes. The debug code is published as app_debug.py in the GitHub repo. I conducted three notification verifications while changing the number of bottled drinks in the refrigerator from 4 to 2 and then to 0. In all three cases, the detection results were correctly notified.

When there are 4 items in stock

When there are 2 items in stock

When there are 0 items in stock

Results using the Brain Builder model

Next, let's verify the operation of the model created with Brain Builder for AITRIOS. I prepared some eggs in the refrigerator and changed the position of the Raspberry Pi AI Camera accordingly.

Object Detection Results

For the case where there are 3 Eggs, the execution result shows that the individual count is correctly detected just before the refrigerator door closes, similar to the bottle case.

Notification Results

The execution results are as follows. I changed the number of Eggs from 3 to 0 for confirmation. As with the bottle, the notifications were also sent correctly!

When there are 3 items in stock

When there are 0 items in stock

Conclusion

In this project, I created a stock management application for the refrigerator using the Raspberry Pi AI Camera and a Raspberry Pi Zero 2 W. Detection of bottle can be easily accomplished with the OSS model, so please give it a try. Additionally, when it comes to refrigerator stock management, it isn't limited to bottled drinks or eggs but also other items like milk, condiments, and so on, which may vary from person to person and home to home. I hope you’ll try creating AI models using Brain Builder in your own projects!

When in trouble

If you encounter any issues while reading the article, please feel free to comment on this article. Also, please check the support site below.Please note that it may take some time to respond to comments.

• Support Site

If you have any issues regarding AITRIOS that are not related to the content of the article, please contact us using the link below.

• Contact Us

If you have questions related to Raspberry Pi, please check and utilize the forum below.

• Raspberry Pi Forums

Want to learn more

Experiment further with the Raspberry Pi AI Camera by following the Get Started guide on the AITRIOS developer site.

Code

OutofStock detection system with Object detection