Pixels to Pistons: The Robotic Dino Player

We’ve all been there.

The Wi-Fi drops, the "No Internet" screen appears, and suddenly that pixelated T-Rex is the only thing standing between you and total boredom. Usually, you’d just tap the spacebar for a few minutes until the router reboots. But as I sat there, I looked at the myPalletizer 260 M5 sitting on my desk—a $299, 4-DOF robot arm that was currently doing nothing—and I had a thought:

“Why should I have to jump over these cacti when I have a literal robot to do it for me?”

The goal was simple: build a system that could "see" the obstacles in the Chrome Dino game and command the robot arm to physically strike the spacebar.

This open-source project is available on GitHub at: https://github.com/vhp8rc7p/hackster/tree/main/dino-with-260

Phase 1: Software First, Hardware Later

Before I let a robotic arm start hammering away at my MacBook’s keyboard, I needed to make sure the "brain" of the operation actually worked. I decided to build a software-only version first using pynput to simulate key presses. If I could get the code to play the game perfectly on its own, then swapping the virtual key press for a physical robot movement would be the easy part.

The first hurdle? Speed.

In the Dino game, milliseconds matter. I initially looked into standard Python libraries like PIL (Pillow) for screen capturing, but a quick trip to StackOverflow confirmed my fears: PIL is notoriously slow for real-time screen scraping. To keep the Dino alive, I needed a higher frame rate. I opted for mss, a ultra-fast cross-platform screen shot module.

The "Search Beam" Strategy

Once I could capture the screen, I had to figure out what the bot was actually looking at. I didn't need to process the whole screen; I just needed to monitor a specific "strike zone" in front of the T-Rex.

I wrote a visualization script to help me align the detection coordinates. To make it easier, I actually went into the browser's Inspect Element tool and edited the HTML border of the game canvas. This gave me a clear visual guide to align my Python capture window.

I decided on a "Dual-Beam" detection system:

• The Cactus Beam: A low-level sensor line to detect incoming greenery.
• The Bird Beam: A slightly higher sensor line.

This was crucial because the game is a bit of a trickster. Some birds fly high enough that you can just run under them, but the low-flying ones require a jump. By using two beams, I could ignore the high-altitude threats and only trigger a jump when something entered the "danger zone."

Surviving the Night Shift

Everything was going smoothly until the T-Rex hit the 700-point mark. Suddenly, the screen flashed, the colors inverted, and my bot went "blind."

In the Chrome Dino game, the world eventually shifts into "Night Mode." My initial detection logic was looking for dark pixels against a light background. When the sky turned black and the obstacles turned white, the bot stopped jumping, and the T-Rex met a very dark end.

I needed a way to detect obstacles regardless of the time of day. I realized that while the background color changes, the contrast remains consistent. Instead of looking for a specific hex code, I adjusted the logic to compare the color of the "beam" pixels to the current color of the "sky." If the pixel at my x_trigger point didn't match the background color—whether that background was white, grey, or black—I knew an obstacle was there. This simple comparison made the bot "color-blind" in the best way possible.

Giving the Brain a Body

With the software logic perfected, it was time to bring in the heavy hitter: the myPalletizer 260 M5.

The 260 collaborative robot is a fantastic piece of hardware for this because it has a dedicated Python API. My goal was to translate a virtual pynput command into a physical movement. I didn't need complex kinematics for this; I just needed a reliable "tap."

Finding the right coordinates for a keyboard press can be a headache if you try to measure everything with a ruler. I used a much simpler "gravity-assisted" method:

1. I manually moved the arm's end effector until it was resting right on the spacebar.

2. I let the weight of the arm (gravity) naturally depress the key.

3. I recorded that Z-axis value. This became my Z_PRESSED.

4. Then, I lifted the arm just a few millimeters until the key popped back up and recorded that value as Z_RELEASED.

Now, whenever the Python script detected a cactus, it would simply send a command to the arm: Move to Z_PRESSED, then immediately back to Z_RELEASED.

The "Double-Tap" Dilemma

The first time I ran the full hardware setup, I ran into a hilarious problem. The robot arm was too fast.

The software detection loop runs dozens of times per second. Because the physical arm takes a moment to move down and up, the detection script would see the same cactus multiple times while the arm was still in the middle of its "jump" motion. The result? The arm would go into a frantic "double-tap" frenzy, trying to jump again while it was already jumping.

The fix was a classic programmer’s move: a tiny time.sleep() command. By adding a small delay immediately after a keypress was triggered, I gave the T-Rex (and the robot arm) enough time to clear the obstacle before the detection loop started looking for the next one.

mc.send_coords(Z_PRESSED, 60)
 time.sleep(0.1)
               mc.send_coords(Z_RELEASED,60)
 # Small cool-down to prevent the arm from stuttering 
 # or double-jumping on the same cactus
 time.sleep(0.3)

Over-Engineered Bliss

There is something deeply satisfying about watching a $300 robotic arm diligently tapping a spacebar so a pixelated dinosaur can jump over a cactus.

What started as a way to kill time during a Wi-Fi outage turned into a full-stack robotics project—combining high-speed screen capture, dynamic color logic, and hardware API integration. It might be the world's most expensive way to play a free browser game, but as any maker will tell you: if it's worth doing, it's worth over-doing.

Developers are welcome to participate in our User Case Initiative and showcase your innovative projects: https://www.elephantrobotics.com/en/call-for-user-cases-en/.