A Robotic Arm That Can Solve Math

1. Why I built this

I wanted a compact demo that stitches together three things at once: computer vision, robotic manipulation, and a small amount of “reasoning.” A handwritten math problem is a fun target — it’s easy for a human to write, interesting for OCR, and the output is a single number that maps cleanly onto a pick-and-place action.

I chose the MyPalletizer 260 specifically because it’s a 4-DOF palletizing arm. Its wrist only rotates around Z, so there are no wrist-flip singularities to babysit. For top-down pick-and-place, that’s actually easier to work with than a general 6-DOF arm — the IK never argues with you.

2. How it works (block diagram)

• Zone A — the handwriting area. The arm parks in an observation pose, the camera reads the equation, PaddleOCR returns character boxes with confidence scores, and Python evaluates the expression (respecting * / / precedence).

• Zone B — the digit-card tray. The arm parks in a second observation pose, PaddleOCR finds the target digit, its pixel center is converted to arm coordinates via a homography, and the pump grabs it.

3. The trick that makes it work: pixel → arm coordinates

The core problem is: given a pixel (u, v) in the camera image, where is that point in the arm’s base frame?

Because the workspace is planar (a flat pad) and the camera pose is fixed, the mapping is a 2-D homography — one 3×3 matrix H per zone:

[x_robot] [u]

[y_robot] = H × [v]

[ 1 ][1]

I calibrate H once per zone by placing a checkerboard, jogging the arm to four known corners, and typing in the arm’s XY reading at each corner. On my setup the residual is under 2 mm, which is comfortably inside the tolerance of a 30 mm digit card and a suction cup.

[PHOTO — checkerboard laid on the workspace with the arm hovering over one corner]

4. Hardware setup

1. Mount the MyPalletizer 260 on a rigid surface with open space in front.

2. Attach the suction pump to the end effector. Route the air hose so it doesn’t tug on the arm during motion.

3. Wire the pump’s control lines to the arm’s I/O:

– –Pin 5 — valve (0 = on, 1 = off)

– –Pin 2 — release pulse (drops the card cleanly)

4. Position the USB camera on a tripod about 35 cm above the workspace, looking straight down. Fixed focus is important — autofocus will silently invalidate your homography between calibration and runtime.

5. Split the workspace into Zone A on the left (handwriting) and Zone B on the right (digit cards).

5. Software install

pip install opencv-python numpy pymycobot paddlepaddle paddleocr

On Windows the arm shows up as a COM port; on Linux / Raspberry Pi it’s usually /dev/ttyUSB0 or /dev/ttyAMA0.

6. Calibration

Camera intrinsics — one time. Print a checkerboard, capture ~15 images at varied angles, and save mtx and dist to camera_params.npz using OpenCV’s cv2.calibrateCamera.

Plane mapping (Zone A and Zone B) — the important one. For each of the four outer corners of the checkerboard:

1. Jog the arm so the end effector is above the corner.

2. Read the arm’s current XY off the terminal.

3. Click the same corner in the camera window.

The script solves for H with cv2.findHomography and writes it to plane_mapping.json (Zone A) or plane_mapping_B.json (Zone B). Repeat for both zones.

7. Code walkthrough

7.1 Connect and configure

from pymycobot import MyPalletizer260
import time, json, numpy as np

# Windows: "COM6"       Linux / Pi: "/dev/ttyUSB0" or "/dev/ttyAMA0"
mp = MyPalletizer260("COM6", 115200)
mp.set_fresh_mode(0)

# Observation poses — 4 joint angles for the palletizer
POSITION_A = [  0.0, -20.0,  10.0,  0.0]   # over Zone A (handwriting)
POSITION_B = [-45.0, -20.0,  10.0,  0.0]   # over Zone B (digit cards)

# Load Zone B homography
H_B = np.array(json.load(open("plane_mapping_B.json"))["homography_matrix"])
observe_z_B = 180   # mm — camera-observation height above the workspace

7.2 Suction pump (same I/O as the MyCobot version)

def pump_on():
mp.set_basic_output(5, 0)          # open valve
    time.sleep(0.05)

def pump_off():
mp.set_basic_output(5, 1)          # close valve
    time.sleep(0.05)
mp.set_basic_output(2, 0)          # pulse release
    time.sleep(1.0)
mp.set_basic_output(2, 1)

7.3 Motion helper with a timeout

The SDK’s is_moving() occasionally reports “still moving” at the end of a trajectory. A timeout fallback keeps the state machine unstuck:

def wait_move_safe(timeout=6):
    start = time.time()
    while mp.is_moving() == 1:
        if time.time() - start > timeout:
            print(f"! wait_move_safe timeout after {timeout}s — continuing")
            break
        time.sleep(0.2)

7.4 OCR — read the handwritten equation

from paddleocr import PaddleOCR

ocr = PaddleOCR(use_angle_cls=True, lang='ch',
use_gpu=False, show_log=False)

def detect_and_recognize(frame, confidence_threshold=0.5):
    result = ocr.ocr(frame, cls=True)
    detections = []
    if not (result and result[0]):
        return detections

    for line in result[0]:
        box, (text, conf) = line[0], line[1]
        if conf < confidence_threshold or not text:
            continue

        # normalize operator glyphs — this is where OCR loves to go wrong
        text = text.strip()
        if   text in ['×', 'x', 'X', '*']: text = '*'
        elif text == '÷':                  text = '/'
        elif text in ['一', '-']:          text = '-'
        elif text in ['十', '+']:          text = '+'

        if not (text.isdigit() or text in '+-*/'):
            continue

        pts = np.array(box, dtype=np.int32)
        cx, cy = int(pts[:, 0].mean()), int(pts[:, 1].mean())
detections.append({'char': text, 'conf': conf, 'center': (cx, cy)})

detections.sort(key=lambda d: d['center'][0])   # left → right
    return detections

Then build the expression and evaluate:

expr = ''.join(d['char'] for d in detections)
answer = eval(expr)          # e.g. "2+3*4" → 14 (Python respects precedence)

(In a production system I’d replace eval with a proper tokenizer. For a whitelist of [0-9+\-*/] on a demo bench, it’s fine.)

7.5 Pixel → arm coordinate

def pixel_to_robot(u, v, H):
    p = H @ np.array([u, v, 1.0])
    return p[0] / p[2], p[1] / p[2]

7.6 Pick-and-place cycle (palletizer form)

def pick_digit(target_char):
    target_char = str(target_char)

    # 1. move to Zone B observation pose
mp.send_angles(POSITION_B, 30);  wait_move_safe(6)

    # 2. OCR search up to N frames for the target digit
    hit = None
    for _ in range(15):
        frame = camera.color_frame()
        for d in detect_and_recognize(frame):
            if d['char'] == target_char:
                hit = d; break
        if hit: break
        time.sleep(0.2)
    if hit is None:
        print(f"digit '{target_char}' not found"); return False

    # 3. pixel → arm XY
    x, y = pixel_to_robot(*hit['center'], H_B)
    x += 5     # small mechanical offset (suction cup vs camera axis)
    y += -10

    # 4. approach, descend, grab, retreat
    #    palletizer coords: [X, Y, Z, RZ]
    current = mp.get_coords()
    above = [x, y, observe_z_B, current[3]]
    down  = [x, y, 70,          current[3]]   # pick height, tuned to card thickness

mp.send_coords(above, 30, 0); wait_move_safe(6)
mp.send_coords(down,  20, 0); wait_move_safe(6)
    pump_on()
mp.send_coords(above, 20, 0); wait_move_safe(6)
    return True


def place_digit(place_xy, place_z=85):
    x, y = place_xy
    current = mp.get_coords()
    above = [x, y, observe_z_B, current[3]]
    down  = [x, y, place_z,current[3]]

mp.send_coords(above, 20, 0); wait_move_safe(6)
mp.send_coords(down,  20, 0); wait_move_safe(6)
    pump_off()
mp.send_coords(above, 20, 0); wait_move_safe(6)

7.7 Main loop

try:
    while True:
        # A — read the handwritten equation
mp.send_angles(POSITION_A, 50); wait_move_safe(6)
    time.sleep(0.5)
    dets = detect_and_recognize(camera.color_frame())
    expr = ''.join(d['char'] for d in dets)
    if not expr:
        print("no expression detected, retrying..."); continue

    answer = eval(expr)
    print(f"{expr} = {answer}")

    # B — pick every digit of the answer and place it
    for i, ch in enumerate(str(answer)):
        if not pick_digit(ch): break
place_digit(PLACE_XY[i])

mp.send_angles(POSITION_A, 50); wait_move_safe(6)

except KeyboardInterrupt:
    pump_off()

8. Demo

9. Challenges and lessons

• Autofocus is the enemy. My first camera kept re-focusing between calibration and runtime, silently invalidating H. Fixed-focus USB cameras are worth the extra $10.

• Operator glyphs are the OCR’s weak spot. − vs 一 vs -, × vs x. The glyph-normalization block in detect_and_recognize earns its keep.

• 4-DOF is a feature, not a limitation. No wrist-flip singularities, no IK stalls. For top-down pick-and-place, the palletizer is genuinely easier to work with than a 6-DOF arm.

• Always add a motion timeout. is_moving() occasionally lies at the end of a trajectory; wait_move_safe() keeps the state machine from freezing.

• Two zones beat one. Merging the handwriting and card areas into a single camera FOV sounded elegant, but calibrating one homography for the whole space made the residuals worse at the edges. Two smaller zones with their own H each are simpler and more accurate.

10. Next steps

• Support parentheses and multi-digit answers up to 3 digits (chain multiple picks — the loop above already does this for 2).

• Replace eval with a proper tokenizer.

• Fine-tune PaddleOCR on children’s handwriting.

• Add a small screen showing the parsed equation before the arm moves — nice for demos and lets you catch OCR errors before the arm commits.

Try It Yourself

Full source, calibration scripts, and config files live on GitHub.

Built with a MyPalletizer 260 and a lot of coffee. If you build one, please tag me — I want to see your demo.

Interested on the product

Click to find out more details on MyPalletizer 260.

Share your innovative Elephant Robotics projects with our community and join our User Case Initiative today: https://www.elephantrobotics.com/en/call-for-user-cases-en/