Human pose estimation as game controller

Introduction

For a lot of people, doing sport is to much of an effort, or they don't have time for it. But these people have probably plenty of time to play video games at home, and don't need any incentive to do so. The idea is to give gamers a way to get more physical activity while playing their favorite AAA game.

Motion controlled games exist, but they are limited to specific games (dance, fitness, childish party games) or they require an expensive VR equipment.

The idea is to use a simple camera based device that will allow a player to control his favorite AAA game with his full body pose.

When you play your favorite AAA game, you are given the choice to use either keyboard+mouse or a gamepad. Adding "body control" as a 3rd standard choice as accessible in price and ease of use as a gamepad is a good way to give people the opportunity to get more physical activity while they play.

And this way of playing will probably also renew the gameplay of the game itself.

What you need to test ?

- a Xilinx Kria KV260 + Basic Accessory Pack

- a Webcam, connected to the Kria

- a PC playing your favorite game

How it works ?

The Kria processes the webcam images with the BlazePose 3D human pose detection model.

Your detected pose is translated to a mouse motion or keyboard key press.

These informations are transmitted to your PC to control the game as if a real keyboard or mouse was used.

1) Setup your PC

Open a terminal on your PC and install "pynput" that will be used to generate keyboard and mouse actions programmatically :

pip install pynput

Then clone the "bodyctrl-xlnx-pc" repository to get the additionnal required files :

git clone https://github.com/BrunoJJE/bodyctrl-xlnx-pc.git

You are now ready to launch the PC application :

cd bodyctrl-xlnx-pc
python3 bodyctrl_pc.py

2) Setup your Kria board

Follow the Xilinx getting started guide to install Ubuntu on your Kria board :

https://www.xilinx.com/KV260-Start

Follow the "Zynq Setup" part of the following TVM tutorial to install "pyxir" and "tvm" on the Kria board :

https://tvm.apache.org/docs/how_to/deploy/vitis_ai.html

Open a terminal on the Kria, and install tensorflow :

pip install tensorflow-aarch64

Then clone the "bodyctrl-xlnx-kria" repository to get the additionnal required files :

git clone https://github.com/BrunoJJE/bodyctrl-xlnx-kria.git

You are now ready to launch the Kria application :

cd bodyctrl-xlnx-kria
python3 bodyctrl_pad.py

3) Test

To play, you will be standing in front of your game screen.

The webcam must be set on the side, pointing toward you.

The best setup is to have the webcam at about 45° relative to you (when screen in front of you is considered at 0°).

The webcam must be far enough from you so it can capture your whole body (better for the pose recognition).

You can connect the hdmi of your Kria to a monitor or TV to check the area seen by the webcam and adjust things.

When your setup is ok, your will have to validate your initial strait standing position, facing precisely your game screen. This will be used as a reference by the program to identify the way your body will later bends to the front, back or side, and to trigger the corresponding keyboards action (forward/backward/side keys).

Conclusion

Just so you won't be surprised, the performance are not that good for now.

The pose recognition run at about 6 FPS, and this is for the Blazepose Light default model running on Kria CPU only.

I succeeded to generate a delagate lib with TVM for the Blazepose Full model to be able to take advandatge of the DPU, but it runs at about 1 FPS (while the default Full model running on Kria CPU without DPU "acceleration" reaches 3 FPS).

The main problem was that I didn't succeed to run the quantization part of the TVM conversion of the model (I had to comment it out), thus the poor performance for now.

To try the TVM conversion by yourself, follow the instructions of the following page to setup your PC and compile your first model :

https://tvm.apache.org/docs/how_to/deploy/vitis_ai.html

There is also the corresponding docs from Xilinx available here :

https://github.com/Xilinx/Vitis-AI/tree/master/external/tvm