PAANI: The On-Device AI Brain for River Robots

Why we built PAANI

A GPS waypoint tells a robot where to go. It does not tell it whether a boat, pier, or riverbank is in the way.

We built PAANI to explore that missing step: turn a camera image into a navigation advisory that a builder can inspect. It identifies supported objects, estimates which parts of the image are water, and recommends STRAIGHT, LEFT, RIGHT, or STOP. A dashboard shows the evidence behind the result.

River monitoring is our first use case. A mobile robot could eventually carry water-quality instruments to more locations, but it first needs a way to interpret its surroundings. PAANI focuses on that perception and guidance layer. Water-quality measurement would require additional sensors.

What the demo does

Our setup combines an Arduino UNO Q and USB camera with a simulated boat running on an Intel NUC.

The camera observes recorded river footage. UNO Q runs the AI models and guidance software locally. The NUC runs ROS 2 and Gazebo to simulate the vessel, water dynamics, GPS, and IMU. A browser dashboard displays the camera view, model outputs, and navigation advisory.

UNO Q handles perception and guidance; the NUC handles simulation. The recorded river view is independent of the simulated boat's position.

This setup lets us test the software and inspect its decisions before moving to a physical boat. Because the footage does not change with the simulated boat's movement, completing a simulated route does not prove that the robot avoided objects in the footage.

Hardware and software

The demonstration uses:

- Arduino UNO Q, 4 GB version.

- USB webcam, USB hub with power delivery, power bank, and suitable data/power cables.

- Separately powered Intel NUC for simulation.

On UNO Q, the Linux processor runs Docker, ROS 2 Jazzy, ONNX Runtime, the dashboard, and logging. The STM32 microcontroller runs the Arduino sketch for guidance LEDs and communication-watchdog indication. Arduino App Lab supports the app/sketch workflow, and Arduino Router Bridge connects the Linux and MCU sides.

Linux runs the AI and robotics software. The MCU provides status indication. Physical GPS/IMU extensions shown in the diagram are not required for this demo; thrust commands go to Gazebo.

Inference currently runs on the CPU through a custom ROS/Docker integration. GPU/NPU acceleration and App Lab AI Bricks are not part of the demonstrated deployment. See the Arduino integration guide

How an image becomes an advisory

PAANI uses two models because finding objects and finding open water are different tasks.

Both models use a 320 × 256 deployment input and run sequentially through ONNX Runtime 1.20.1. They use FP32 precision and occupy about 14.82 MB combined on disk. That file size is not their runtime memory requirement.

Camera frame → detection and segmentation → timestamped evidence → tracking → corridor rules → explained advisory. The branches show two model roles; the current worker executes them sequentially.

The software divides the image into overlapping left, center, and right corridors. It combines water coverage, detected objects, uncertainty, and a legacy motion-urgency term into a risk score. Tracking adds evidence about whether an object persists across observations.

Rules then decide which advisories are allowed. For example, persistent occupancy can block a turn even when that side has the lowest numerical risk. The dashboard explanation reflects the final decision after those rules run.

There is no LLM or third neural model in this pipeline. Tracking and decision logic use Python/NumPy and explicit rules. Builders can inspect the detector runtime, segmenter runtime, and guidance policy. Deployment artifacts are listed under pinned models.

Seeing why the robot turns—or stops

The dashboard helps trace a questionable advisory back to its source: the water mask, an object detection, track history, or a decision rule.

Historical offline host replay of WaterScenes image 01703. PAANI advises RIGHT, with left/center/right risk scores of approximately 0.498 / 0.511 / 0.457.

Historical offline host replay of image 47357. PAANI advises STOP despite a lower right-corridor score because occupancy and permitted-turn rules also affect the decision.

These examples used repeated still images and controlled timestamps on a Windows host. Their detector threshold was 0.25, compared with 0.40 on the board. They illustrate the dashboard and policy, rather than real-world motion performance.

PAANI also stops when perception data becomes stale. A separate diagnostic estimates whether an object's apparent size suggests approaching or receding motion. At the current two-second inference interval, it cannot gather its required evidence window, so "unknown" is the expected board result. It does not measure distance or closing speed, and its labels do not change the navigation risk score. See motion estimator's limits.

Training for river scenes

We trained task-specific models using two datasets:

- Detection: WaterScenes, with 10,000 selected training images, 10,824 official validation images, and 5,412 official test images. Source labels were grouped into our four detection classes. Training ran for 60 epochs.

- Segmentation: MaSTr1325, split into 1,127 training and 198 validation images. Labels were mapped to water, obstacle, and sky. Training ran for 50 epochs.

Both runs used an NVIDIA RTX A2000 12 GB GPU. Training used PyTorch; the detector also used Ultralytics, and the segmenter used torchvision. We exported the selected models to ONNX for deployment on UNO Q.

Detector accuracy and loss across all 60 recorded epochs. These measure model training, not navigation success.

Segmentation validation mIoU and training loss across 50 epochs. A validation-loss series was not recorded

We also tried INT8 quantization for segmentation. It made the model smaller, but recorded validation mIoU dropped from approximately 0.9750 to 0.7303, so we kept FP32.

The segmentation split is image-based, with no independent test set or guaranteed separation between related sequences. Similar scenes may make its validation score optimistic. Dataset and software licences also apply when reusing the work.

Detection mAP measures object-detection quality; segmentation IoU measures overlap with labelled regions. Neither is a boat-navigation success rate.

The board figures come from a five-minute CPU benchmark with 151 distinct results. Sub-second processing time does not mean video-rate inference, and container memory includes more than model weights. Power consumption and battery endurance were not measured.

Recreate the demonstration

Start with the system README and its camera-simulation workflow.

• Prepare the hosts Connect UNO Q and the NUC to the configured network. Use your development computer for SSH and the dashboard.
• Get the source Match revisions on both hosts and preserve your local configuration.
• Prepare models and camera Load and verify the pinned models on UNO Q, then select the actual camera-device path.
• Start Gazebo on the NUC Confirm that the lake, vessel, and numbered waypoints appear.
• Start perception and navigation on UNO Q Check that the dashboard updates the camera view, learned mask, and advisory. Navigation requires fresh visual and simulated-sensor data.

Use the camera-simulation guide for troubleshooting.

What still needs work

The most useful next steps come from the demo's limits:

• Reject unusable images A black image has produced a mostly-water mask and STRAIGHT advice. Stale-data detection does not catch this failure.
• Test a coupled scene Camera input needs to respond to the boat's motion before avoidance can be evaluated meaningfully.
• Validate beyond the training data There are no dedicated litter, log, or vegetation detection classes. The person class mainly represents people aboard vessels, not a validated person-in-water rescue detector.
• Prepare for physical operation Add and validate sensing, motor control, and hardware safety measures, then measure power and endurance.

Build on PAANI

PAANI gives makers a starting point for experimenting with local robot vision, and researchers a way to inspect how model outputs affect navigation rules.

The camera interfaces, timestamped evidence, dashboard, and logging can be reused while adapting the models and controller. Canal inspection, reservoir surveys, or aquaculture monitoring are possible directions. Each would need its own testing; a ground robot would also need different traversability logic.

Our next goal is to connect what the camera sees to the robot's actual movement—and use the same visible evidence trail to understand where it succeeds and where it needs improvement.

PAANI provides a reusable local AI foundation for robots whose decisions developers can understand, test and improve.