Body Tracking, ROM Hacking, and Arm Shocking Punch-Out!!

Mike Tyson's Punch-Out!!, or simply Punch-Out!!, was one of the NES' best-selling games. Only two NES games in 1988 sold more than 2 million copies in North America: Punch-Out!!and The Legend of Zelda. If you had an NES as a kid, then you can probably still hear that catchy theme music playing in your head. To bring this classic to a new generation — and those of us looking for a hit of nostalgia — Ian Charnas created a body-tracking, ROM-hacking, and arm-shocking version of Punch-Out!! that you can play in your browser.

This browser-based version of Punch-Out!! lets you control Little Mac using your body movement and doesn't require any hardware beyond a webcam. Google's fantastic TensorFlow-based MoveNet handles the skeletal tracking. MoveNet runs at 30+ FPS on most modern computers, but still provides 17 tracking points and more than enough accuracy for an application like this. Charnas' code turns body movements, like a punch or a dodge, into control commands for the game. So throwing a punch with your right hand will cause Little Mac to perform the same action on screen.

But there is a problem that should be obvious to all veteran Punch-Out!! players. This game requires precise timing and it takes more time to move your entire arm than it does to push a button on an NES controller. To account for that, Charnas had to slow down each opponent's actions without slowing down the rest of the game. That wasn't a trivial task and required hacking the game's assembly code in ROM. Charnas was able to pull it off and that alone would be an impressive feat, but he didn't stop there.

To give players a reason to dodge and block, Charnas created an armband that produces a shock whenever Little Mac takes a punch. Of course, that feature won't work in your browser. The game communicates wireless with an Adafruit Feather 32u4 RFM69HCW development, which triggers a relay connected to a high voltage generator from a novelty electric shock toy. The electronics attach to a smartphone armband case, with copper foil electrodes on the inside. The shock is mild, but enough to keep players on their toes during a match against King Hippo.

If you want to try this body-controlled version of Punch-Out!! yourself, head over to reallifepunchout.com to launch the game.