AirDesk 3D: Minority Report Hand Control on the BPI-AI2N

Introduction

A few weeks ago I had a free evening and was watching the movie Minority Report and wondering what project I could create for the Renesas BPI-AI2N board from Banana Pi that had recently turned up as a surprise at my door.

This board is a little different to my normal FPGA developments but as I do embedded software as well I thought it would be fun to try creating a simple project.

I just needed an idea, then the protagonist of the movie started controlling his computer using his hands. It looks very cool and futuristic even more so back in the early 2000s when this film was originally made.

It struck me that this would be an interesting project to recreate using our hands to be able to interact with the GUI on our computer.

Creating this solution would require a range of applications from image processing to machine learning and then tying it back to the GUI and control all fun things to do.

Firstly, however, we had to set up the BPI-AI2N board and its carrier card.

Set Up Board

The board comes with the BPI-AI2N core, mounted on a carrier card which provides USB, power, terminal and ethernet connectivity. For a display we also get a second board which connects to the DSI output and converts this to HDMI. This connects over a flexi and can be seen on the left side of the image.

This HDMI output board folds under the board and we can mount it under the carrier card using the mounting screws provided.

However, there is one more step which must be followed before we power the system; that is we must fit the heat sink and fan on top of the BPI-AI2N carrier card to ensure when we are running the Linux OS we are not overheating the processor.

The final system looks as below.

Now we are ready to get started with installing the operating system on to the SD Card.

Install the OS into the a SD Card

To get started with creating the operating system, you can go to the Banana Pi Docs page for BP-AI2N. This will enable you to be able to download either a Yocto or Armbian image for your SD Card.

I went with Armbian. Once the file was downloaded, I unzipped it onto my machine and then used Balena Etcher to write the image to the board.

Once the image was ready, I inserted the SD Card, connected up the board to power, termina, a webcam and ethernet and watched as the board booted.

The first request you see will be asking you to set a root password, it will then walk you through a few different set up options.

This includes your command shell.

Followed by setting up the user account you normally work with.

Once the account was set up the next stage is to set the location.

Finally we are able to start working with the terminal.

if we want to make any changes then we can run the command.

armbian-config

With this we can see the system configuration and make changes as necessary.

I wanted to make sure I was connected over Ethernet so I then used PowerShell to ping the board.

With network connectivity confirmed the next step is to connect over the network and start development of our application.

To get started I am going to install a x11vnc server so I can see the desktop remotely also if required.

Resultant

With the VNC server available I could develop on the board over the network from the comfort of my desk, which made the iterative bring up of the application much less painful.

The Plan

With the board up and running, it was time to think about the application itself, which I called AirDesk 3D. The idea is a spatial desktop served from the BPI-AI2N as a web page. The browser shows floating panels, and instead of a mouse we use our hand. An open hand hovers and moves the cursor, closing the hand into a fist grabs the panel underneath it, moving the closed fist drags the panel around, and opening the hand again releases it. A broad lateral swipe switches workspace.

To achieve this we need two sensors. A Logitech C920 webcam provides the RGB video for the hand tracking AI, while an Onion Tau LiDAR camera provides an optional metric depth channel, so the palm position also gets a real Z distance in metres. The Tau is a depth camera not an RGB camera, which is why the separate C920 is required for the hand landmarks.

The processing chain runs entirely on the BPI-AI2N. The C920 is captured at 1280 x 720 in MJPEG mode, a 320 x 180 copy is passed to the hand tracking AI, the resulting landmarks drive a cursor and gesture state machine, and the optional Tau path attaches a metric depth. Everything lands in a thread safe state store which is published by a small HTTP server, so any browser on the LAN can be the display. The application runs as a systemd user service so the whole thing survives reboots and USB glitches.

Under the heat sink the RZ/V2N gives us four Arm Cortex-A55 cores, 8 GB of LPDDR4x and up to 15 sparse TOPS of DRP-AI acceleration, which is a very capable platform for this kind of edge vision application.

Hand Tracking with MediaPipe

For the hand tracking I used the pretrained palm detection and hand landmark models which ship with MediaPipe 0.10.18. MediaPipe first finds a palm, then regresses 21 hand landmarks inside that region, and in video mode it reuses the previous hand region until tracking is lost, which reduces how often the palm detector needs to run.

I want to be clear about what was and was not done here. No custom neural network was trained for this project, no dataset was collected or labelled, and this version of the pipeline runs on the Cortex-A55 cores using the XNNPACK CPU delegate rather than the DRP-AI accelerator. Porting a compatible network through the Renesas DRP-AI toolchain is the obvious next optimisation, and would be the correct route to substantially higher inference throughput, but it would be wrong to claim the current version is DRP-AI accelerated. Even on the CPU we achieve roughly 4 to 6 processed AI frames per second, which turns out to be perfectly usable once the filtering described below is in place.

The detector configuration is deliberate in every choice.

self.hands = mp.solutions.hands.Hands( static_image_mode=False, max_num_hands=1, model_complexity=0, min_detection_confidence=0.55, min_tracking_confidence=0.55, )

Setting static_image_mode to False enables temporal tracking between video frames, max_num_hands of one avoids any ambiguity over which hand controls the desktop, model_complexity of zero selects the lighter landmark model for the Arm CPU, and thresholds of 0.55 reject weak guesses without making it too hard to reacquire the hand.

One capture detail worth knowing: the C920 in this build reports a hardware minimum zoom_absolute of 100, so software cannot zoom out any further. If 1280 x 720 does not include enough of the room for the full arm movement area, the answer is to move the camera farther away, not to fight the driver.

From Landmarks to a Cursor

MediaPipe returns normalized landmark coordinates, so although inference runs on the 320 x 180 copy, the same results can be drawn accurately over the full 1280 x 720 frame for the annotated video stream.

My first attempt used the index fingertip, landmark 8, as the cursor. This works nicely with an open hand but jumps the moment the finger folds into a fist, which is exactly when we grab a panel. The fix is to use the palm centre instead, calculated as the mean of the wrist and the four finger MCP knuckle landmarks.

palm_points = (0, 5, 9, 13, 17) palm_x = sum(landmarks[i][0] for i in palm_points) / len(palm_points) palm_y = sum(landmarks[i][1] for i in palm_points) / len(palm_points)

The palm centre barely moves as the hand opens and closes, which makes closed hand dragging much more stable. The normalized coordinates are scaled to the browser workspace and passed through a low pass smoothing filter with a coefficient of 0.34, which removes landmark jitter without making the cursor feel disconnected from the hand.

Grab and Release

There is no separately trained gesture classifier in this project either. The open or fist state is calculated geometrically from the landmarks. For each of the four non thumb fingers, the code compares the wrist to fingertip distance with the wrist to knuckle distance.

tips = (8, 12, 16, 20) mcps = (5, 9, 13, 17)
openness = mean( distance(wrist, fingertip) / distance(wrist, finger_mcp) for each finger )

An extended finger produces a large ratio and a folded finger a small one. The important part is that a single threshold does not work, the value hovers around it and the interface chatters between grabbed and released. The deployed state machine therefore uses hysteresis with a dead band between the two thresholds.

An openness below 1.28 for two consecutive AI frames declares the hand closed and emits a grab, an openness above 1.58 for two consecutive frames declares it open and emits a release, and anything in between keeps the previous state. Requiring two consecutive frames adds roughly 0.3 to 0.5 seconds of deliberate dwell at the measured inference rate, which in practice feels intentional rather than laggy. If the hand is lost for 0.9 seconds the grab is cancelled safely so a panel is never left stuck to a hand that has left the frame.

Adding Real Depth with the Tau LiDAR

The Tau is entirely optional, the RGB AI alone supplies the X and Y position and the hand state. What the Tau adds is a genuine metric Z, turning the project into a proper RGB and depth fusion demonstration.

The Tau delivers a little endian 160 x 60 array of uint16 distances over its USB serial port at around 23 frames per second. Distances between 250 and 1500 mm are accepted as the working range. On startup, or whenever calibration is requested, the application records 45 empty scene frames, roughly two seconds worth, and takes a per pixel median to learn the background. A pixel is then marked as foreground when it is at least 65 mm closer than that learned background. A 3 x 3 morphological opening and a 5 x 3 closing clean the mask, connected components are extracted, and components touching an image edge, smaller than 9 pixels or larger than 1900 pixels are rejected. The survivors are scored by depth, size, fill, centre position and continuity with the previous frame.

That is considerably more machinery than my first attempt, which simply selected the nearest global depth percentile. In live testing that naive detector locked solidly onto static clutter at the extreme left of the depth image and refused to let go. Background subtraction, coherent component selection and explicit edge rejection removed that failure completely, and it was a good reminder that the simple version of a depth tracker rarely survives contact with a real room.

Fusing RGB and Depth

To fuse the two sensors, the RGB palm coordinates are mapped into normalized Tau coordinates. The mapping is currently a simple crop exposed as environment variables in the service file, with the default being the full frame.

Environment=AIRDESK_RGB_LEFT=0.0 Environment=AIRDESK_RGB_TOP=0.0 Environment=AIRDESK_RGB_RIGHT=1.0 Environment=AIRDESK_RGB_BOTTOM=1.0

At the projected Tau position the application samples a 17 x 11 depth patch, keeps depths inside the working range, finds the nearer portion of the patch using the 35th percentile plus 55 mm, and returns the median of that nearer surface. This biases the answer towards the hand rather than the wall behind it. A set of temporal rules then keeps the fused Z honest: RGB observations older than 0.48 seconds and Tau frames older than 0.25 seconds are not fused, the last good depth is held for up to 0.75 seconds, the Z value is low pass filtered with a coefficient of 0.30, and an isolated jump of more than 0.34 metres is rejected as an outlier.

One design decision matters more than all of these numbers. In an early version, noisy depth generated its own grab and release events and the interface was unusable. The final version completely suppresses depth generated gestures. Depth only ever supplies the Z coordinate, and grab and release come exclusively from the debounced open and fist AI transition. If the Tau is disconnected the cursor and fist controls carry on working, only the metric Z is lost.

Because the Tau has a much shorter vertical field than the C920, physical alignment matters. Mount the Tau close to the webcam, aim both at the same interaction area, and recalibrate with an empty scene.

The AirDesk Interface

The browser interface is served by the application itself at port 8080, so opening http://BOARD_IP:8080/ from any machine on the LAN gives the spatial desktop. The server also exposes a small set of useful endpoints: /api/state returns the pointer, landmarks, gesture, sensor and performance telemetry as JSON, /camera.mjpg is the live annotated RGB stream with the skeleton drawn over it, /api/calibrate requests a fresh empty scene Tau calibration and /health is a simple health response.

The interface shows the live annotated camera view with the hand skeleton, confidence and hand state overlaid, the Tau depth field, and an edge core panel with the live frame rates, UI latency and memory figures. The panels themselves are the things you grab and move with your fist.

Installing and Running on the Board

The application lives in /home/adiuvo/airdesk3d, with the Python package in airdesk3d/, the static browser files in web/ and service scripts in scripts/.

One quirk of the specific Armbian image used is that it did not include ensurepip, so python3 -m venv could not create a virtual environment without pulling in another OS package. Rather than requiring sudo, the dependencies are installed into a project local vendor directory.

cd /home/adiuvo/airdesk3d
python3 -m pip install --target ./vendor 
'mediapipe==0.10.18' 
'TauLidarCamera==0.0.5' 
'TauLidarCommon==0.0.2' 
'pyserial==3.5'

MediaPipe brings its compatible NumPy, OpenCV contrib, protobuf and TensorFlow Lite dependencies with it into vendor, and keeping that directory first on PYTHONPATH stops those versions colliding with anything else on the board. It is also worth checking the USB devices enumerate as expected, the C920 appears as 046d:082d on /dev/video0 and the Tau as 0483:5740 on /dev/ttyACM0, and adding your user to the video and dialout groups if necessary. A powered USB hub is recommended for stable camera and Tau power.

To run manually:

PYTHONPATH=/home/adiuvo/airdesk3d/vendor:/home/adiuvo/airdesk3d 
python3 -m airdesk3d 
--mode auto 
--host 0.0.0.0 
--port 8080 
--tau-port /dev/ttyACM0 
--camera-device /dev/video0

For unattended operation the application is installed as a systemd user service, which restarts on failure and recovers dropped USB devices.

[Unit] Description=AirDesk 3D RGB-AI and Tau fusion service After=network-online.target graphical-session.target Wants=network-online.target
[Service] Type=simple WorkingDirectory=/home/adiuvo/airdesk3d ExecStart=/home/adiuvo/airdesk3d/scripts/run-board.sh Restart=on-failure RestartSec=3 Environment=PYTHONUNBUFFERED=1 Environment=PYTHONPATH=/home/adiuvo/airdesk3d/vendor:/home/adiuvo/airdesk3d Environment=TF_CPP_MIN_LOG_LEVEL=2 Environment=AIRDESK_RGB_LEFT=0.0 Environment=AIRDESK_RGB_TOP=0.0 Environment=AIRDESK_RGB_RIGHT=1.0 Environment=AIRDESK_RGB_BOTTOM=1.0
[Install] WantedBy=default.target

It is enabled in the usual way.

systemctl --user daemon-reload systemctl --user enable --now airdesk3d

Using it is then straightforward. Open the interface in a browser, keep hands out of the Tau field and hit /api/calibrate to learn the empty scene, then place an open hand in the C920 view, move the palm over a panel, close your fist to grab it, drag, and open your hand to release.

Testing and Results

Because the gesture logic is full of thresholds and timing rules, I backed it with focused unit tests, five temporal gesture tests, three background hand tracker tests, two open space and edge clutter tests, three RGB and depth fusion tests, three production filtering tests and three open and fist state machine tests. All 19 passed on the board itself.

A coordinated live hand test then sampled the RGB, Tau and HTTP paths together, 120 samples in all. An AI hand was present in 60 of them, a live pointer in 57, and 30 samples achieved direct RGB to Tau fusion. The maximum AI confidence reached 99.8 percent, the fused depth ranged from 0.42 to 1.47 metres, and both sensors stayed online for all 120 samples. The Tau loop processed at roughly 23 frames per second while the full resolution camera and AI path ran at roughly 6 frames per second.

[IMAGE: AirDesk UI screenshot mid gesture - hand skeleton with confidence and hand distance visible]

Subjectively, the combination of the palm centre cursor, the smoothing and the hysteresis is what makes it feel usable. Grabbing a panel, dragging it across the workspace and dropping it works reliably, and watching your own hand drive the interface with a real depth readout attached is a genuinely pleasing Minority Report moment.

It is worth saying that all of the video is processed locally on the BPI-AI2N, the application does not record camera frames, and the browser only receives the stream while it is connected. This is, of course, an experimental human interface project, not a certified control system.

Wrap Up

This project demonstrates a complete edge vision interaction loop on the BPI-AI2N: USB camera acquisition, pretrained on device AI, landmark derived gesture recognition, optional RGB and depth fusion, temporal filtering, a real time web interface and persistent Linux service operation. The most important lesson was that AI landmark detection alone is nowhere near enough for a usable interface. Getting from a working demo to something you can actually drive required a stable palm coordinate, explicit open and fist hysteresis, lost hand handling, depth outlier rejection and recoverable USB workers.

There is plenty of scope to take it further. Replacing the full frame RGB to Tau mapping with a measured homography calibration, porting a compatible palm and landmark network through the Renesas DRP-AI toolchain to unlock the board's 15 TOPS, and adding an on screen calibration wizard are all on the list. In the meantime, all of the code is available in the project repository, and I hope it provides a useful starting point for your own gesture controlled creations.