Introduction
Steps
1. Uno Q Setup and Arduino AppLab
2. Collecting Data ( Images
3. Labeling
4. Train, Build, Deploy
5. Test on Uno Q
6. Build Your System: Uno Q + usb camera + robotic arm + program

Published August 18, 2026 © GPL3+

Hand Gestures Control for Robotic Arm

Basic implementation of hand gesture detection in robotics, featuring real-time object tracking and intuitive robotic arm movement.

BeginnerFull instructions provided3 hours40

Things used in this project

Hardware components

Arduino UNO Q

USB-C Power Adapter

eg. 27W Pi5 Power Adapter

USB-C to USB A hub with PD

eg. Official Arduino USB C hub (8 in 1)

USB Camera

eg. Logitech C920/ 922/ 270 ..

PC/ laptop

Your existing computer

Robotic arm with gripper (3 DoF)

with 3 servos

Breadboard (generic)

Jumper cables

6V Power Supply

eg. 4x 1.5V battery

Software apps and online services

Arduino App Lab

Edge Impulse Studio

Story

Demo Video

Introduction:

This project utilizes a local network setup where an Arduino Uno Q is connected to a USB camera and a robotic arm, with programming performed via the Arduino AppLab on our computer/laptop. The ease of using the I/O pins on the Arduino Uno Q, which are identical to those of a standard Arduino Uno, makes this project accessible and easy for anyone to carry out.

In this relatively simple project, the system will translate the position of hand detected into movements of two servos that control the pan-tilt mechanism on robotic arm, while gesture types (for example, open and close) will control the servo on the claw/gripper. The system can use an existing model (Hand Gesture) available in Arduino AppLab, or a custom model created in Edge Impulse Studio integrated within AppLab. The model will be deployed in Python code for the program logic and then passed to the Arduino Sketch to drive the servos accordingly.

This project is intended as a basic implementation of object detection in robotics, featuring real-time object tracking and intuitive robotic arm movement. It is a foundation for applications in more complex robotic systems.

Easy to get components

Steps:

1. Uno Q Setup and Arduino AppLab

For the initial setup, install the Arduino App Lab on your computer and connect the Arduino Uno Q via a USB-C connection (PC-hosted mode). Once AppLab is opened, the Uno Q device will appear; select it, create an access password, and configure the Wi-Fi network so that we can use it in local network mode for convenience. After completing this process, disconnect the Uno Q and connect it to a power adapter. Open the Arduino AppLab on your computer, and the Uno Q will appear for you to access. To familiarize yourself with this ecosystem, you can select 1 project from Examples and press Run, which will upload and execute the chosen program on the Uno Q.

For the next step, we will create our project by duplicating "Detect Objects from Camera" from the Examples, because it’s using similar bricks. Open it and click Copy and edit (in the upper right corner), then rename it as desired, for example: "Custom Hand Gestures Detection." Next, select the AI models tab and click Train new AI model, which will allow you to access Edge Impulse Studio using your Arduino account (sign in or create one if you don't have one yet). Then, you will be redirected to the Edge Impulse Studio by clicking Start to Train your AI model. We will discuss the stages within Edge Impulse Studio in the next step below.

1 / 5 • Connect Uno Q to App Lab

2. Collecting Data ( Images)

Before we start uploading the photos, we will create three labels/classes: a closed hand (neutral), and a scissor-like hand gesture (open and close). Capture these photos using your camera or your laptop's built-in webcam. I recommend taking a variety of 30–40 photos for each type, which should be sufficient for this project. Save them in folders, and we will continue in the Edge Impulse Studio.

For those who are not familiar with Edge Impulse Studio, choose Images project option, then Object detection. In Dashboard > Project Info, choose Bounding Boxes for labeling method and Uno Q for target device. Then in Data acquisition, click on Upload Data tab. Choose you saved folder then upload.

1 / 2

3. Labeling

The next step is labeling, now click on Data Acquisition, click on Labeling queue tab, then start drag a box around an object and label it and Save. Repeat.. until all images labelled.

After labeling, it’s recommended to split the data into training and testing sets, around an 80/20 ratio. If you haven't done this yet, click on Train / Test Split to automate this process.

1 / 2 • Labeling

4. Train, Build, Deploy

Once your labelled dataset is ready, go to Impulse Design > Create Impulse, and set the image width and height (eg. 320x320). Choose Fit shortest axis, then select Image and Object Detection as the learning blocks, and click Save Impulse. Next, navigate to the Image Parameters section, select RGB as the color depth, and press Save parameters. After that, click on Generate, where you’ll be able to see a graphical distribution.

Now, move to the Object Detection section and configure the training settings. Select GPU and set the training cycles to around 100, learning rate to 0.001 and medium for model size. Choose YOLO-Pro or MobileNetV2 SSD FPN-Lite as the NN architecture. Once done, start training by pressing Start, and monitor the progress.

If everything goes well; eg MAP@50 = 1.0 or the precision result is more than 90%, proceed to the next step. Go to the Model Testing section, click Classify all, and if the MAP@50 = 1.0 or the accuracy is around 90%, you can move on to Deployment. Select deployment target Arduino Uno Q, select quantized (int 98) then click Build. Next step will be on Arduino AppLab.

1 / 5

5. Test on Uno Q

After the model has been built, return to the Arduino AppLab; you will find the custom model we built in Edge Impulse Studio earlier—click the Download icon and select it. Now, our custom model is ready to use. To test it, don't forget to connect the USB-C to USB-A hub and the USB camera to yout Uno Q. Then, click Run in AppLab; the upload and execution process on the Uno Q will begin. If everything goes smoothly, the WebUI-HTML program will display live object detection inference in your computer's browser. Try the 3 types of gestures from our model to ensure it is working properly.

6. Build Your System: Uno Q + usb camera + robotic arm + program

To control the robotic arm using hand gestures, you must ensure that everything is properly connected as shown in the diagram above. Additionally, before creating your Python or Sketch programs, you need to download several Sketch Libraries. Click Add Sketch Library, then search for and download them one by one: Servo 1.3.0, MsgPack 0.4.2, DebugLog 0.8.4, ArxContainer 0.7.0, and ArxTypeTraits 0.3.1 (or their latest updated versions).

Below is a screenshot of the Python code. It is important to adjust/calibrate the servo angles to match the physical configuration of your specific robotic arm. In this Python code, the logic is as follows: if the "neutral" gesture is detected, the horizontal position is mapped to the pan servo angle, while the size of the bounding box is interpreted as a change in the tilt servo angle. The "opening" and "closing" gestures translate to the movement/angle change of the claw servo. Meanwhile, the Sketch file determines which pins are used; in this case, pins 9, 10, 11. Servos angle standard setup and handles the output display on the LED matrix (a smiley face) whenever a hand gesture is detected.

All Python, Sketch and YAML files can be downloaded from Code Attachment section, then you can click Add File icon on AppLab then mod/adjust for your needs.

1 / 4

This project successfully demonstrates a basic Edge AI vision system that connects real-time object detection to physical actuation by deploying an Edge Impulse custom gestures model with reliable accuracy and low latency. The project validated the end-to-end pipeline from camera input, spatial mapping, and actuator control in a single embedded system. This project highlights the practical integration of Edge AI with robotics and serves as a foundation for more advanced applications.

Code

from arduino.app_utils import *
from arduino.app_bricks.web_ui import WebUI
from arduino.app_bricks.video_objectdetection import VideoObjectDetection

from datetime import datetime, UTC
import time

# ============================================================
# UI + DETECTION
# ============================================================

ui = WebUI()

detection_stream = VideoObjectDetection(
    confidence=0.6,
    debounce_sec=0.0
)

# ============================================================
# SERVO CONFIGURATION
# ============================================================

# PAN SERVO
PAN_MIN = 10
PAN_MAX = 170

# TILT SERVO
TILT_MIN = 90
TILT_MAX = 160

# CLAW SERVO
CLAW_OPEN = 40
CLAW_CLOSE = 100

# ============================================================
# INITIAL SERVO POSITIONS
# ============================================================

last_pan = 90
last_tilt = 120
last_claw = CLAW_OPEN

# ============================================================
# SMOOTHING
# ============================================================

SMOOTHING = 0.65

# ============================================================
# HELPER FUNCTIONS
# ============================================================

def clamp(val, min_val, max_val):
    return max(min_val, min(max_val, val))


def map_range(x, in_min, in_max, out_min, out_max):
    return (x - in_min) * (out_max - out_min) / \
           (in_max - in_min) + out_min


def smooth(current, target):
    return current + (target - current) * SMOOTHING

# ============================================================
# MAIN DETECTION CALLBACK
# ============================================================

def process_detections(detections):

    global last_pan
    global last_tilt
    global last_claw

    best_obj = None
    best_area = 0

    # ========================================================
    # FIND LARGEST DETECTED OBJECT
    # ========================================================

    for label in ["neut", "open", "close"]:

        if label not in detections:
            continue

        objs = detections[label]

        if not isinstance(objs, list):
            objs = [objs]

        for obj in objs:

            if "bounding_box_xyxy" not in obj:
                continue

            x1, y1, x2, y2 = obj["bounding_box_xyxy"]

            width = x2 - x1
            height = y2 - y1

            area = width * height

            if area > best_area:

                best_area = area

                best_obj = {
                    "label": label,
                    "bbox": (x1, y1, x2, y2),
                    "area": area
                }

    # ========================================================
    # NO DETECTION
    # ========================================================

    if best_obj is None:
        return

    label = best_obj["label"]

    x1, y1, x2, y2 = best_obj["bbox"]

    # ========================================================
    # TRACKING ONLY WHEN "neut" DETECTED
    # ========================================================

    if label == "neut":

        # ====================================================
        # PAN CONTROL
        # ====================================================

        center_x = (x1 + x2) / 2

        target_pan = map_range(
            center_x,
            80,
            560,
            PAN_MIN,
            PAN_MAX
        )

        target_pan = clamp(
            target_pan,
            PAN_MIN,
            PAN_MAX
        )

        # ====================================================
        # TILT CONTROL USING BBOX AREA
        # ====================================================

        width = x2 - x1
        height = y2 - y1

        area = width * height

        target_tilt = map_range(
            area,
            8500,
            49000,
            TILT_MIN,
            TILT_MAX
        )

        target_tilt = clamp(
            target_tilt,
            TILT_MIN,
            TILT_MAX
        )

        # ====================================================
        # SMOOTH MOVEMENT
        # ====================================================

        last_pan = smooth(
            last_pan,
            target_pan
        )

        last_tilt = smooth(
            last_tilt,
            target_tilt
        )

        # ====================================================
        # DEBUG
        # ====================================================

        print(
            f"TRACKING | "
            f"X={center_x:.1f} "
            f"AREA={area} "
            f"PAN={int(last_pan)} "
            f"TILT={int(last_tilt)}"
        )

    # ========================================================
    # CLAW CONTROL
    # ========================================================

    if label == "open":

        last_claw = smooth(
            last_claw,
            CLAW_OPEN
        )

        print("CLAW OPEN")

    elif label == "close":

        last_claw = smooth(
            last_claw,
            CLAW_CLOSE
        )

        print("CLAW CLOSE")

    # ========================================================
    # FINAL INTEGER VALUES
    # ========================================================

    pan = int(last_pan)
    tilt = int(last_tilt)
    claw = int(last_claw)

    # ========================================================
    # SEND TO ARDUINO
    # ========================================================

    Bridge.call(
        "set_arm",
        pan,
        tilt,
        claw
    )

    # ========================================================
    # SEND TO WEB UI
    # ========================================================

    entry = {
        "gesture": label,
        "timestamp": datetime.now(UTC).isoformat()
    }

    ui.send_message(
        "detection",
        message=entry
    )

    # ========================================================
    # SMALL DELAY FOR STABILITY
    # ========================================================

    time.sleep(0.005)

# ============================================================
# REGISTER CALLBACK
# ============================================================

detection_stream.on_detect_all(
    process_detections
)

# ============================================================
# START APP
# ============================================================

App.run()

// NEW PROJECT: HAND GESTURES controlled ROBOTIC ARM

#include "Arduino_RouterBridge.h"
#include "Servo.h"
#include <Arduino_LED_Matrix.h>

// ============================================================
// LED MATRIX
// ============================================================

Arduino_LED_Matrix matrix;

const uint32_t smiley_face[] = {
  0x000200a8,
  0x0a802000,
  0x04403e00,
  0x00000000
};

// ============================================================
// SERVOS
// ============================================================

Servo servoPan;
Servo servoTilt;
Servo servoClaw;

// ============================================================
// PINS
// ============================================================

const int panPin = 9;
const int tiltPin = 10;
const int clawPin = 11;

// ============================================================
// POSITIONS
// ============================================================

int panPos = 90;
int tiltPos = 120;
int clawPos = 40;

// ============================================================
// SETUP
// ============================================================

void setup() {

  Serial.begin(115200);

  matrix.begin();

  servoPan.attach(panPin);
  servoTilt.attach(tiltPin);
  servoClaw.attach(clawPin);

  servoPan.write(panPos);
  servoTilt.write(tiltPos);
  servoClaw.write(clawPos);

  // ==========================================================
  // BRIDGE START
  // ==========================================================

  if (!Bridge.begin()) {
    Serial.println("Cannot setup Bridge");
  }

  if (!Monitor.begin()) {
    Serial.println("Cannot setup Monitor");
  }

  // ==========================================================
  // PROVIDE FUNCTION
  // ==========================================================

  Bridge.provide("set_arm", set_arm);

  Serial.println("Robotic Arm Ready");
}

// ============================================================
// LOOP
// ============================================================

void loop() {

}

// ============================================================
// FUNCTION CALLED FROM PYTHON
// ============================================================

void set_arm(int pan, int tilt, int claw) {

  panPos = constrain(pan, 10, 170);
  tiltPos = constrain(tilt, 60, 160);
  clawPos = constrain(claw, 40, 100);

  servoPan.write(panPos);
  servoTilt.write(tiltPos);
  servoClaw.write(clawPos);

  matrix.loadFrame(smiley_face);

  Monitor.print("PAN: ");
  Monitor.print(panPos);

  Monitor.print(" TILT: ");
  Monitor.print(tiltPos);

  Monitor.print(" CLAW: ");
  Monitor.println(clawPos);
}

Credits

Jallson Surjo

7 projects • 26 followers

Tech integrator for schools. Also works as a maker and his activities include disassembling, fixing, and making things.

Hand Gestures Control for Robotic Arm