Play Pong with your Face!

Interested in getting a little exercise while you game? Well, project makes that dream a reality. It is made possible, thanks to the Useful Sensors Person Detector, which is a tiny module that contains a camera and microcontroller running a TinyML model. The module communicates over I2C, making it easy to integrate, and provides the coordinates up to 4 faces that it sees in its field of view, along with confidence. With the Useful Sensors Person Detector, you get all of the magic of ML without the headaches. It is preloaded with a well trained model that has been tuned for the onboard sensor.

This project uses the location of the detected face as input for a game of Pong. The game is pretty basic - I haven't added any fancy features yet, like you know, keeping score - but it is still a lot of fun to play.

Since the sensor module communicates over I2C, it is easy to integrate into any embedded project. I put together a small driver in Circuit Python so I easily work with Pimoroni's clever Interstate 75 board. The best part is that everything simply snaps together - the Interstate 75 plugins right into a RGB Matrix and the sensor module connects to the Interstate 75 using its builtin Stemma QT / Qwiic port. I mounted the module to the RGB Matrix with a little tape and cardboard, but I have some grand plans for a 3D printed frame and laser cut smoked acrylic.

I was lucky enough to find an existing Pong implementation in CircuitPython. Over on GitHub, FoamyGuy put together a nice Pong example, that is well documented and has comments. I extended it to work on an RGB Matrix instead of a TFT display and switch the control of the paddle from buttons to the input for the sensor module.

The module scales the coordinates it reports back to be from 0 to 255. I do a little bit of processing to convert them into the coordinates for the paddle. Since the paddle can only move left and right, I ignore the Y coordinate. A bounding box is given for each face and I only work with the highest confidence one, assuming any lower confidence ones are onlookers. With this bounding box I find its center and use that to make sure the input is balance. The camera on the sensor module has a surprisingly wide field of view... and I am sort of lazy. To make it easier to move left and right, I process the input to amplify motion the further you get from the center. In general, the latency from the module seems low enough to have pretty responsive input.

There is clearly room for improvement... and that is why I am so excited. I think TinyML is going to unlock new ways for us to interact with our devices... and games! The Useful Sensors Person Detector make this all really accessible and the limits are no longer technical, but what you can imagine. Hopefully the code for this project is a jumping off point for your own experiments... and if you add score keeping, please put in a PR!

Useful Sensors sent me a prototype to work with... and I forgot to break off the programming header 🤦.