Tic ArUco Toe: Teaching a Robot Arm to Play (and Win)

The last time I lost at tic-tac-toe, I was seven. Yesterday, I lost again — to a 4-axis robot arm that costs less than a decent office chair. This is the story of how I taught my myPalletizer 260 to look at a paper board, decide the best move, and pick up its piece without any position sensors, no fiducials on the pieces themselves, and no cheating. Just a camera, some ArUco markers, and about 200 lines of Python.

What We Built

A fully autonomous tic-tac-toe opponent. The setup is simple: a myPalletizer 260 sitting next to a printed 3×3 board, a webcam looking down at the table, and a small pile of wooden game pieces on one side. The human plays as X (by hand, placing pieces on the board). The robot plays as O — it sees the board through the camera, computes the best response using the minimax algorithm, then picks up its piece and sets it in the right cell.

Two things make this project fun to build:

1. It's a complete pipeline. Vision, planning, and motion all have to work together, which teaches you a lot about where each layer can fail.

2. It's satisfying. When the arm places a piece where you expected — and blocks your winning move — that little "the machine knows what I'm doing" moment never gets old.

How It Works

The system has three layers, and each one is worth understanding on its own:

1. Vision — turn a camera frame into a 3×3 board state

2. Brain — pick the best move using minimax

3. Motion — pick up a piece and place it in the target cell

Let's walk through each.

1. Vision: Reading the Board With ArUco Markers

The hardest part of any physical game is: where are the pieces? Color detection is unreliable (shadows, red pieces on red squares), and template matching struggles with rotation. So I skipped the whole "detect the game piece" problem — I glued a tiny printed ArUco marker on top of each wooden piece. Different marker IDs for X and O, and now detection is a solved problem.

OpenCV's ArUco module gives you sub-pixel accuracy under normal lighting, doesn't care about rotation, and reports each marker's ID. That means one function call tells me both what pieces are on the board and where each one is.

aruco_dict =
aruco.getPredefinedDictionary(aruco.DICT_4X4_50)
 corners, ids, _ = aruco.detectMarkers(frame, aruco_dict, parameters=params)
 for i, mid in enumerate(ids.flatten()):
    cx, cy = corners[i][0][:, 0].mean(), corners[i][0][:, 1].mean()
    # mid == 10 -> X piece, otherwise -> O piece

But wait — how does the code know which cell each piece is in? I use a small calibration step. At startup, I place 9 markers exactly where the board cells will be, press 's', and the script snapshots their pixel positions. From then on, every detected piece is assigned to the closest "ghost slot." The physical calibration board can then be cleared and replaced with the printed grid — the pixel positions stay locked in memory.

This trick is worth stealing for any "board game" project. It sidesteps the need for a full board-corner homography and works with any grid you can print.

2. Brain: Minimax in 20 Lines

Minimax is the classic algorithm for two-player, zero-sum games. In plain English: the robot assumes the human will always play optimally, then picks the move whose worst outcome is the best.

Tic-tac-toe has only ~255, 000 possible game states, so we can search the entire tree in milliseconds. No pruning needed, no heuristics. Here's the whole thing:

def minimax(board, is_maximizing):
    if wins(board, 'O'): return 1
    if wins(board, 'X'): return -1
    if is_full(board):   return 0

    if is_maximizing:
        return max(minimax(place(board, m, 'O'), False) for m in empty(board))
    else:
        return min(minimax(place(board, m, 'X'), True)  for m in empty(board))

def best_move(board):
    return max(empty(board), key=lambda m:
minimax(place(board, m, 'O'), False))

Reading it top to bottom: at each terminal state, we return +1 if the robot won, -1 if the human won, 0 for a draw. At every internal node, we recurse — max on the robot's turn, min on the human's. The robot's chosen move is the one whose minimax score is highest. That's it. That's the whole AI.

3. Motion: Pick and Place With 4 DOF

The myPalletizer 260 is a 4-axis palletizing arm — X, Y, Z, and yaw. That's actually perfect for this project because tic-tac-toe pieces don't need clever wrist orientations; you just move above a cell and drop straight down.

The nine board cells and the pickup pile are all hardcoded coordinates I measured by hand once. I use the vacuum pump (controlled via Raspberry Pi GPIO pins 20 and 21) to grab pieces. The motion sequence for each move is:

1. Hover above the pickup pile

2. Descend, turn on pump

3. Lift the piece up to travel height

4. Travel to hover position above the target cell

5. Descend, turn off pump

6. Return to hover

TIC_TAC_TOE_COORDS = [
    [180,15, 100, 10], [180, 0, 100, 10], [180, -55, 100, 10],
    [130,25, 100, 10], [130, 0, 100, 10], [130, -55, 100, 10],
    [ 80,25, 100, 10], [ 80, 0, 100, 10], [ 80, -55, 100, 10],
]
PICK_COORD_ABOVE = [-10, -130, 100, 10]
PICK_COORD = [-10, -130, 65, 10]

def pick_and_place(cell):
    tx, ty, tz, tyaw = TIC_TAC_TOE_COORDS[cell]
    arm.send_coords(PICK_COORD_ABOVE, 40); time.sleep(3)
    arm.send_coords(PICK_COORD, 20); time.sleep(3)
    pump_on(); time.sleep(1.5)
    arm.send_coords(PICK_COORD_ABOVE, 40); time.sleep(5)
    arm.send_coords([tx, ty, tz, tyaw], 20); time.sleep(5)
    pump_off()

One thing I learned the hard way: don't rush the sleeps. When the arm hasn't fully settled and you turn off the pump early, the piece skids across the board. Better reliable and slow than fast and dramatic.

Putting It All Together

The main loop is boring in the best way: read board from camera, check win/draw, run minimax, execute pick-and-place, wait for human turn (detected as "new X on the board"), repeat.

while not game_over(board):
    if human_turn:
        board = wait_for_new_x(board)   # detect new marker of ID 10
    else:
        move = best_move(board)
        pick_and_place(move)
        board[move // 3][move % 3] = 'O'
    human_turn = not human_turn

That's the whole game.

What Tripped Me Up

• Marker size vs. camera distance. My first pieces had 8 mm markers, and the camera couldn't resolve them from 40 cm away. Bumping to 15 mm fixed detection instantly. If you're getting flaky detection, don't tune HSV — just print bigger.
• Hardcoded coordinates drift. If the board moves even a few mm, the arm places pieces on the line between cells. A future improvement would be a pixel-to-mm affine transform so the code adapts to board position — but for a fixed setup, hardcoded is fine.
• Pump timing. See above. Seriously, add another 500 ms.
• "The arm keeps sitting on the pump." On the 260, the pump hangs off the last joint. If you send a coord with a low Z but forget the yaw is currently pointing in a bad direction, the arm elbows itself. Always test each new coord slowly first.

Try It Yourself

All the code lives on GitHub. Clone it, install pymycobot and OpenCV, and you're 30 minutes away from a robot that plays tic-tac-toe. Even if you don't own a myPalletizer 260, the vision and minimax code drop straight into any 4+ DOF arm — just swap the send_coords call and the pump toggle.

If you build one, tag me — I'd love to see it. And if your robot beats you, at least you'll lose to something you built yourself

Interested on the product

Click to find out more details on myPalletizer 260 M5.

Share your innovative Elephant Robotics projects with our community and join our User Case Initiative today: https://www.elephantrobotics.com/en/call-for-user-cases-en/