This Raspberry Pi-Powered Smart Fridge Keeps an Eye on Ingredients with OpenAI's GPT-4 with Vision
Just opening the fridge updates as list of its contents, while a Telegram bot provides a field-expedient user interface.
Pseudonymous maker "mimobeano" is putting a little artificial intelligence into the kitchen, adding OpenAI's GPT-4 with Vision machine learning model to somewhere unexpected: the refrigerator.
"I'm working on a project to know what food is in my fridge from anywhere, so I can use it up before it expires," mimobeano explains of the project's origins. "Originally, this was my Master's project in 2022. I took 2000+ photos and fine-tuned an object detection model to make something that only really worked for 12 items. But now I have GPT-4 [with] Vision, I thought I'd give the project a big upgrade."
GPT-4 with Vision adds computer vision capabilities to the company's GPT-4 generative pre-trained transformer machine learning model, giving it to the ability to ingest imagery and provide responses to questions. "The model is best at answering general questions about what is present in the images," OpenAI explains of its capabilities. "For example, you can ask it what color a car is or what some ideas for dinner might be based on what is in you fridge, but if you show it an image of a room and ask it where the chair is, it may not answer the question correctly."
In the case of mimibeano's fridge, the imagery is of perishable goods. "When the fridge door is opened, it changes the state of a button connected to [a Raspberry Pi 4 Model B] and a photo is taken with the [Raspberry Pi] HQ Camera and wide-angle lens," the maker explains. "When the user wants to know what's in the fridge, the most recent photo is sent to the API, which returns the list of foods it can see in the image.
"I call the OpenAI API [Application Programming Interface] only when the user wants to know what they have, so I can be super efficient on spending," mimobeano adds. "Since I couldn't be diddled to build a UI [User Interface], I used a Telegram event to trigger this and I host the bot on the Raspberry Pi. At the moment, I check the '/ingredients' request is coming from me to stop anyone else using the bot. It also gets recipe suggestions from GPT-4 with '/recipes'."
More details are available in mimobeano's Reddit post; source code has not yet been published.