Gameplay That Moves You

Joysticks, gamepads, and keyboards have been staples as input devices for video games for nearly half a century. These traditional controllers have proven their reliability and versatility, offering precise control and accessibility to gamers of all skill levels. And while these controllers undoubtedly still deliver a tried-and-true gaming experience that is enjoyed by gamers around the world, new technologies and innovations have expanded the realm of possibilities for input devices. Motion-sensing controllers, such as the Wii Remote, PlayStation Move, and Kinect, have revolutionized gaming by introducing a more physical and immersive approach. These devices detect and interpret the player's movements, bringing a whole new level of interactivity to gameplay.

Motion-sensing controllers have even found applications beyond traditional gaming. They are being used in rehabilitation and physical therapy to help patients regain mobility and coordination. By incorporating game-like elements into therapy sessions, these devices provide an engaging and motivating way for patients to recover and retrain their motor skills.

While these alternative controllers are not new, they have become more accessible than ever before. The cost of the sensing equipment has plummeted rapidly, and high end computing equipment is no longer needed to handle the processing. An old laptop, or even a Raspberry Pi or Arduino, can tackle the job in many cases.

YouTuber Everything Is Hacked recently created an alternative game controller of his own to show just how easy it really can be these days. And by following his example, you could build your own custom setup as well. Using a laptop, Everything Is Hacked built a body tracking system in which predefined body poses can trigger key presses on a keyboard. This, in turn, can control virtually any game.

The initial goals for the project were to build a neural network for body pose recognition, to support multiple players, to work with any game, and to improve the player’s gaming skills. Right off the bat, Everything Is Hacked ran into some trouble. He found building a custom neural network to be more challenging than expected, so he instead borrowed some methods from his previous full-body keyboard project.

Rather than recognizing each body pose, this involved using OpenCV and MediaPipe's Pose detection software to perform real-time detection of body keypoints. With that information in hand, some fairly simple calculations can be performed to determine where the keypoints are relative to one another, and thereby infer what position the body is in.

Multiplayer gameplay also proved to be more difficult than anticipated. For this to work, each player needed to be tracked in every single frame so that the system would know which player the inputs were coming from. This was far too slow on a laptop, so as an alternative, the screen was split into separate segments for each player. This means that players need to keep to their own area for their body motions to be counted, but it did speed up processing significantly. Seems like a reasonable trade-off.

A system was devised in which a wide variety of body poses can be defined in a CSV-delimited file. And since those poses can be mapped to any key, the system does lend itself well to working with virtually any game. However, that does not mean that all games are well suited to body motion-based inputs. Flappy Bird may work well enough, but Super Mario Bros. is pretty challenging without a traditional gamepad.

So, Everything Is Hacked took the next step and developed some games that were ideally suited for the new controller. One game involves matching your body pose to a pose shown on the screen, and another involves reaching out to pet cute cats and dogs for points. A heart rate monitor was also included to increase or decrease the difficulty level based on the player's level of exertion. Might as well get a workout in while you are at it, right?

By and large, this project was successful, however, it did not help to improve gaming skills, as had been hoped. Oh, well, three out of four isn't bad. If you would like to give it a try for yourself, check out the source code on GitHub.