Calorie Loaf: DIY AI Calorie Tracker for Your Kitchen

Ever since I started taking fitness seriously, I've paid more attention to what I eat: the calories on my plate, not just my workouts.

Every meal meant the same routine. Take out the phone, open the app, scan or search the food, pick a portion, log it. Doing that for every single meal gets old fast, and it's exactly the kind of friction that makes people give up on tracking.

So I started wondering why I needed my phone for this at all. I'm a hardware engineer, so why not build a dedicated device for it? It would sit on the kitchen counter like a toaster. You put your plate under it, press one button, and the meal gets logged.

So that's what we built. Using it is simple: place your plate under the camera and tap one button. The device recognizes the food, estimates the calories, protein, carbs, and fat, and adds the meal to your daily log.

Everything runs on the device itself. The camera, two neural networks, the meal database, and the dashboard all run on a D-Robotics RDK X5, housed in a 3D-printed enclosure shaped like a loaf of bread. There's no cloud service and no subscription, and your food photos stay at home.

Demo Video

The Brain: D-Robotics RDK X5

Every device like this needs a brain that can run computer vision without leaning on the cloud. For this project, we went with the D-Robotics RDK X5, a compact single-board computer built specifically for edge AI and robotics.

• CPU: Octa-core Arm Cortex-A55 @ 1.5 GHz
• BPU: Dedicated neural accelerator, up to 10 TOPS
• GPU: 32 GFLOPS
• RAM: 4 GB or 8 GB versions
• Camera: Dual MIPI CSI interfaces

The number that matters most here is the BPU. It's a separate AI accelerator that runs neural networks on the board itself, fast enough that a measurement feels instant. As we found out later, it can also keep **two** detection models loaded at once, and the whole vision pipeline ended up being built around that.

The dual MIPI CSI ports were the other reason we picked it. They let the board drive a binocular stereo camera natively, which opened up some interesting ideas we'll get to shortly.

Setting Up the Board

Getting the board running was quick. We downloaded the RDK X5 OS image from the D-Robotics Developer Community, flashed it to a microSD card with balenaEtcher, put the card in the board, connected a display, and powered it on.

A few moments later, we were looking at the RDK X5 desktop. Everything was up and running, so it was time for the first AI test.

First AI Test: YOLO from the RDK Model Zoo

Before pointing anything at food, we wanted to be sure the AI side of the board worked end to end. For the first test, we went with YOLOv9.

This turned out to be easier than expected. D-Robotics' RDK Model Zoo has the whole Ultralytics YOLO lineup, from YOLOv5 right up to YOLO26**, already converted into ready-to-run BPU binaries. There's no model conversion or quantization to do yourself. Getting YOLOv9 onto the X5 took one command to download the model and one to run it:

git clone https://github.com/D-Robotics/rdk_model_zoo
cd rdk_model_zoo
git checkout rdk_x5 # the X5 has its own branch
cd samples/vision/ultralytics_yolo
bash download.sh # pulls precompiled *_640x640_nv12.bin models

The `_nv12` in the model names matters. NV12 is the camera's native output format, so camera frames go straight into the BPU with no colour conversion. The food pipeline later in this write-up uses the same trick.

The board processed video in real time, drawing labels, bounding boxes, and confidence scores on every detected object. That told us the hardware was ready for the real job.

The Camera: RDK Stereo Camera Module

For the camera, we went with the D-Robotics RDK Stereo Camera Module (GS130W). It's a binocular stereo camera with two sensors in a single module, and it plugs straight into the X5's two MIPI CSI ports. We connected it, tested both camera feeds, and both worked.

• Sensors: 2× SmartSens SC132GS, 1.3 MP, global shutter
• Field of view: 157.2° diagonal / 96.8° horizontal / 115.6° vertical
• Max frame rate: 120 fps
• Baseline: 80 mm (79.2 mm in the factory calibration)
• Extras: synchronized dual exposure, external trigger, near-infrared sensitive (850/940 nm)

The global shutter captures the whole frame at once, so there's no smearing if the plate is still moving slightly when you press the button. And having two lenses 80 mm apart meant we could, in theory, measure depth and work out the actual volume of the food rather than only recognizing it. That's the plan for the next version of this project.

Once we started writing code against the camera, a few quirks showed up:

• The sensors are mounted in portrait: Native resolution is 1280×1088 on its side, so every frame gets rotated 90° before anything else touches it.
• The hardware scaler is picky: It only accepts certain fixed downscale ratios and rejects anything else with "Invalid size", even properly aligned sizes. Half resolution (544×640) works, so that's what the pipeline captures at.
• The lens is very wide: At 157° diagonal, a plate 20 - 40 cm away fills only part of the frame. For experimenting, our live-preview tool has a `--zoom` option that crops the centre and scales it up, like a longer lens.

Food Segmentation: Going Beyond Bounding Boxes

With the board and camera sorted, we moved on to the actual food recognition.

Normal object detection gives you a bounding box, a rectangle drawn roughly around each object. Segmentation goes a step further and marks the exact pixels that belong to each item. Rice, chicken, and bread on the same plate each get their own mask, which gives a much clearer picture of what's really there.

One model wasn't enough

The obvious starting point was YOLOv8n-Seg, a segmentation model that comes precompiled for the X5's BPU. It produces proper pixel masks, but it's trained on the COCO dataset, and only ten of COCO's 80 classes are food. Four of them made it into our prototype: banana, apple, orange, and pizza.

Four foods aren't much of a calorie tracker, so we needed something that knew more foods. That's where YOLO-World came in. It's an open-vocabulary detector: instead of a fixed list of classes, you give it text prompts like "fried egg" or "rice, " and it looks for those. There's no retraining involved. The catch is that on this BPU build, it only outputs boxes, not masks.

So neither model could do the job on its own. YOLOv8-Seg had the masks but not the foods, and YOLO-World had the foods but not the masks. We decided to run both on every frame and combine the results:

PROTOTYPE_COCO_CLASSES = {"banana", "apple", "orange", "pizza","donut"}
PROTOTYPE_WORLD_CLASSES = [
    "fried egg", "sausage", "bacon", "bread",
    "rice", "grilled chicken", "cheese",
]

Together, that gives 12 food classes. Adding a 13th to the YOLO-World side is as simple as adding a string to that list, as long as there's a matching text embedding in `config/food_vocabulary_embeddings.json`. That file already has embeddings for 115 words, all 80 COCO classes plus 35 extra food terms, so there's plenty to choose from. YOLO-World has room for 32 prompts. We started with 42 candidates and trimmed down to 8, because every extra prompt costs compute and memory on every frame.

Turning boxes into masks

That still left YOLO-World's foods with only rectangles. Rather than train a new segmentation model, we reached for a classic image-processing algorithm: OpenCV's GrabCut. You give it a rectangle, and it separates the object from the background inside it based on colour:

pad_x, pad_y = box_w * pad_ratio, box_h * pad_ratio   # 8% outward padding
rect = (rx1, ry1, rx2 - rx1, ry2 - ry1)
cv2.grabCut(image_bgr, mask, rect, bgd_model, fgd_model,
        iterations, cv2.GC_INIT_WITH_RECT)
fg_mask = np.where((mask == cv2.GC_FGD) | (mask == cv2.GC_PR_FGD), 1, 0)

We pad the box outward by 8%, so GrabCut has a strip of real background to learn from. It isn't perfect. On flat, uniform food like a slice of cheese, it sometimes gives up and returns nothing, and on tiny regions it can throw an error. In both cases, the code simply falls back to using the box itself as the mask. The masks come out in the same format as YOLOv8-Seg's, so the rest of the pipeline doesn't care which model found the food.

Cleaning up duplicates

Running two models on the same image created a new problem: sometimes both of them found the same apple. We clean that up in two passes. First, if a YOLO-World box overlaps a YOLOv8-Seg detection by more than 50%, the YOLO-World one is dropped, because a trained mask is more trustworthy than a GrabCut guess. Then a second pass sorts everything by confidence and removes anything that overlaps a higher-scoring item by more than 30%:

def nms_items(items, iou_thres=NMS_IOU_THRES):
    kept = []
    for item in sorted(items, key=lambda it: it["score"], reverse=True):
        if not any(boxes_iou(item["box"], k["box"]) > iou_thres for k in kept):
            kept.append(item)
    return kept

First Real Test

With both detectors wired up, we put a real meal in front of the camera and ran the pipeline on the RDK X5. And it actually worked. The masks landed on the food, the labels were right, and both models finished in about a quarter of a second.

To make testing like this easier, we wrote a small live-preview tool, food_seg_web_preview.py. It streams the camera feed to a browser with masks and labels drawn onto every frame, which made it easy to adjust lighting and camera position. Bright violet means a YOLOv8-Seg mask and deep purple means YOLO-World + GrabCut. We picked violet because almost no food is that colour, so the overlay never blends in.

python3 vision/food_seg_web_preview.py --video-index 1 --zoom 1.5 --detect-every 2

From Detection to Calories

Knowing what is on the plate is only half the job. To get calories, you also need to know how much of it there is.

Counting servings

The approach that ended up in the final build is deliberately simple: every detected item counts as one typical serving. A banana is a medium banana (118 g), a slice of bread is 30 g, and a portion of rice is 158 g

UNIT_GRAMS = {
"banana": 118, "apple": 182, "orange": 131, "pizza": 107,
"fried egg": 60, "sausage": 75, "bacon": 12, "bread": 30,
"rice": 158, "grilled chicken": 120, "cheese": 30, "avocado": 200,
...
}

It can't tell a small apple from a big one, so it's a starting point rather than the final answer. Counting needs no depth at all, so it works reliably every time. Measuring each item's real volume with the stereo camera is what the next version will add.

Looking up the nutrition

From there it's a table lookup. Each food has its calories, protein, carbs, and fat per 100 g, roughly following USDA reference values

# name -> (density_g_per_cm3, kcal_per_100g, protein_g, carb_g, fat_g)  [per 100g]
"banana":          (0.95,  89,  1.1, 23.0,  0.3),
"grilled chicken": (0.85, 165, 31.0,  0.0,  3.6),
"bacon":           (0.70, 541, 37.0,  1.4, 42.0),

The table covers 45 foods, well beyond the 12 the camera currently recognizes, so adding a new food later usually doesn't need new nutrition data. The density column isn't used yet. It's there for the volume-based estimation coming in the next version, where grams = volume × density.

Keeping the models warm

There was one more practical problem to solve. Loading both models onto the BPU takes several seconds, and nobody wants to wait that long every time they press a button. So the vision code runs as its own long-lived background service, measure_server.py. It loads the models once when it starts and then sits waiting. Each time the dashboard asks it to measure, it only has to capture a frame and run inference.

A measurement comes back as a simple JSON report

{
"mode": "count",
"total_kcal": 199.7,
"items": [
{"label": "banana", "score": 0.81, "grams": 118, "kcal": 105.0,
"protein_g": 1.3, "carb_g": 27.1, "fat_g": 0.4},
{"label": "apple",  "score": 0.64, "grams": 182, "kcal": 94.6,
"protein_g": 0.5, "carb_g": 25.5, "fat_g": 0.4}
]
}

If the camera spots something that isn't in the nutrition table, it still shows up in the report with kcal: null, so the dashboard can say what it saw instead of quietly ignoring it.

Building the Dashboard

With the AI side working, we started on the web dashboard, where everything from the device comes together. It shows the detected food with its estimated calories and macros, keeps a log of everything you've eaten that day, and tracks your progress against daily goals.

To show it on the device itself, we used a 4-inch Waveshare display connected to the RDK X5 over HDMI. Because it's a web app, the same dashboard also opens on any phone or laptop on the same network.

Using it

Here's what a typical meal looks like. The Home screen has one big amber Measure button. Tap it, and a scanning animation plays while the RDK X5 captures the plate and runs both models. A moment later, the app jumps to the Log screen, where the new items appear in a "Just added" card. The ring at the top shows how many calories you have left for the day and turns from amber to red if you go over. Below it, three bars track protein, carbs, and fat, followed by the list of everything you've eaten today. Anything the camera can't recognize can be added by hand.

The History screen charts your daily calories over the last 30 days along with your 7-day average, and Goals is where you set your calorie and macro targets.

The front end uses Alpine.js and Chart.js straight from a CDN, with no build step, so it loads quickly on the board.

Under the hood

The software runs as two processes. The dashboard is a small Flask app with a SQLite database, and it talks to the vision service over HTTP:

┌───────────────────────────────────────────────┐
│  Browser (kiosk on the 4" display, or LAN)    │
│  Alpine.js UI: Home · Log · History · Goals   │
└──────────────────────┬────────────────────────┘
                       │ HTTP :5000
┌──────────────────────▼────────────────────────┐
│  web/app.py  (Flask)                          │
│  • UI + REST API • meals.db (SQLite)          │
│  • background capture jobs                    │
│  • auto-launches measure_server.py            │
└──────────────────────┬────────────────────────┘
                       │ HTTP POST /measure :8081
┌──────────────────────▼────────────────────────┐
│  vision/measure_server.py  (warm BPU models)  │
│  YOLOv8-Seg + YOLO-World/GrabCut → nutrition  │
└───────────────────────────────────────────────┘

Splitting it this way had a bonus: the web app never touches the board-specific camera and BPU libraries, so we could develop the dashboard and run its tests on an ordinary laptop.

The database is just two tables, one for meals and one for goals:

CREATE TABLE IF NOT EXISTS meals (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp TEXT NOT NULL,
photo_path TEXT,
food_name TEXT NOT NULL,
calories REAL NOT NULL,
protein_g REAL, carb_g REAL, fat_g REAL,
confidence REAL,
source TEXT NOT NULL DEFAULT 'vision' -- 'vision' or 'manual'
);

CREATE TABLE IF NOT EXISTS goals (
id INTEGER PRIMARY KEY CHECK (id = 1),  -- single-row settings table
daily_calorie_goal REAL NOT NULL DEFAULT 2000,
protein_g REAL, carb_g REAL, fat_g REAL
);

For anyone who wants to build on it, here's the API the dashboard uses:

| Method | Route | Purpose |
|---|---|---|
| `GET` | `/api/meals?date=YYYY-MM-DD` | Meals for a day |
| `POST` | `/api/meals` | Manual entry |
| `DELETE` | `/api/meals/<id>` | Delete a meal |
| `GET` | `/api/history?days=30` | Daily totals |
| `GET` / `PUT` | `/api/goals` | Read/update goals |
| `POST` | `/api/capture` | Start a measurement → `202 {job_id}` |
| `GET` | `/api/capture/<job_id>` | Poll: `processing` / `done` / `error` |

Designing the Enclosure in Fusion 360

At this point, the software was working, but it was still a bare board with cables everywhere, which isn't something you'd keep on a kitchen counter. Next up was the enclosure.

We wanted it to look kitchen-friendly, so we took inspiration from the shape of a loaf of bread and went with a beige and chocolate-brown colour scheme.

We designed it in Fusion 360 around the actual hardware. The camera sits on top of a stand, pointing down at the plate. The 4-inch display faces the front, and the inside has room for the RDK X5 in its case plus cable routing. We split the body into front and back halves with a separate top plate, so each part prints cleanly, and the electronics are easy to get in and out.

3D Printing

We printed every part separately on a Bambu Lab A1: the front body, back body, top plate, camera stand, and camera holder, in beige and chocolate brown.

Before the final assembly, we did a quick fit check, placing the display, camera, and other hardware into the printed parts to make sure everything lined up. Everything fit, so it was finally time to put it together.

Assembly

We started by attaching the RDK Stereo Camera Module to the camera holder with M3 screws.

Then we fixed the camera holder onto the camera stand.

Next, we fitted the 4-inch Waveshare display into the front portion of the enclosure.

There wasn't enough room inside for straight connectors, so we used right-angle HDMI and USB adapters while installing the RDK X5 in its case. This is also the moment to connect the display's HDMI cable and the camera ribbon cables.

With everything connected, we joined the front and back halves.

Finally, we placed the top plate and mounted the camera assembly on the main body.

And that's it: our AI calorie tracker is ready.

Conclusion

This project started with a small annoyance, getting the phone out at every meal, and ended up as a standalone device that sits on the counter and does one job well.

Under the hood, the RDK X5's BPU keeps two detection models running side by side to cover each other's gaps, a classic algorithm (GrabCut) fills in the missing masks, and a small web dashboard on a 4-inch display makes it a one-tap appliance. The bread-loaf enclosure helps it look like it belongs in the kitchen.

A special thanks to D-Robotics for providing the RDK X5 kit for this project.

If you build one, let us know which foods you add to the menu!