Kinect4NES – Control your classic NES with Kinect V2

Recently, I have found myself becoming involved in the exciting world of “IoT” or internet of things.  All of this started while attending a presentation on the subject  that was put on by my fellow colleague Bret Stateham.  If you are unfamiliar with the concept of “IoT”, I like to think of it as a programmable system comprised of input sensor(s) / polling service(s) which interact with a physical device and optionally store data received by the sensors to a web service where it could be optionally processed for patterns to facilitate things like forecasting.  In short, we are connecting your things that are inherently offline to the internet.

This particular project does not quite fully fit the definition above as the end result is a non-network connected thing (a classic NES console) connected to modern sensor (the Kinect V2).  However, it could be easily modified to fit such a description.  For example, this concept could be extended to allow the public to access and control a physical NES through a web interface (think Twitch plays Pokémon) ~Coming Soon~ OR it could allow users to upload Kinect Gesture profiles online that could be pulled down through the application to allow better control in certain games.  Nonetheless, it leverages concepts that are integral to most “IoT” projects, specifically hardware interface construction, software interfacing, and application development.

I’d like to elaborate a little bit more on inspiration for this project as I would really like to tear down any barriers currently holding back abled developers from breaking into this field.  At the end of Bret’s presentation, he showed off a quick demo that showcased an Intel Galileo board running Windows for IoT that contolled a blinking LED with breakpoints set in Visual Studio.  Not exactly, jaw dropping surface value, but when looked at for what it can enable rather what it is specifically doing, you may find an opportunity to expand the possibilities of your code.  It dawned on me that that this blinking LED demo was all I needed to know to allow computer code to interact with physical objects.  I began thinking about everything like a blinking LED project, SMS notifications from my washer/dryer/dishwasher when a cycle is complete, automating the addition of chemicals to a swimming pool, or firing a rocket when a threshold of retweets is achieved on a particular hashtag.   All of these become comparatively simple problems when looked at through the lens of turning on a light when a certain condition is met!  I soon found myself pondering the idea of mixing old nostalgic technology with the bleeding edge.  What if I could control a classic NES with a Kinect 2 device?  Not through an emulator, but a physical, rectangular, Gray-Box NES from 1984.

Ingredients:

• An NES console with game to test
• An NES controller OR some wiring skills and a CD4021BE 8-bit shift register
• 12 strands of wire, recommend Kynar
• 8 1k resistors (technically any value from 1k to 50k should suffice)
• 2 3.6k resistors (again higher not necessarily bad)
• IoT board capable of running Firmata, Intel Galileo or Arudio Uno etc.
• Kinect V2 Sensor for Windows OR recently announced Kinect V2 Adapter and existing Xbox One Kinect Sensor
• Machine capable of running the Kinect V2 SDK

In my writeup, I am going to assume you have zero experience with hardware development which is fitting because I literally had no idea how to even blink an LED when I started this project.  I am also going to assume you want to know how to do achieve the final product from a blank slate, let’s start by breaking down the problem into sub-problems.

Problem:

We want to use a Kinect V2 Sensor to control games on a physical NES

Sub-Problems:

1. We need to interface with the NES controller port using computer code

2. We need to speak to that hardware interface through a software interface, preferably in C#

3. We need to create an application that takes input from the Kinect V2 Sensor and processes it through the software interface, into the hardware interface, where it can reproduce button presses on the NES console based on defined gestures.

Sub-Problem #1 – Creating a controller interface in hardware

We need to understand how an NES controller works.  I found an excellent article on the subject @ the PoorStudentHobbyist Blog.  I highly recommend giving it a read.  We learn that the NES controller operates using a 4021B 8-bit shift register wired to 8 inputs (the 4 D-Pad directions + Select, Start, A, and B buttons).  Given this knowledge, we can build an interface in a couple of ways.  One would be to use an 8-bit shift register emulator like the one described @ MezzoMill or we can leverage the physical hardware within an NES controller to create the interface or we could substitute a comparable 8-bit shift register. Coincidentally, I came across 2 NES controllers brand new in the box at a local flea market for $6 a week before starting this project.  I considered it a sign and went through with deconstructing the controllers to get the parts I needed.

I removed the screws on the back of one of the controllers, opened it up, and desoldered the 5-wire braid that connects to the controller port and I also desoldered the 4021B shift register.  With the board disassembled you can determine the pinouts to see which wires / buttons are attached to which pins on the 4021b using a mulimeter or the painstaking process of tracing with your finger.

The 4021B Shift register:

Let’s assume you have never used an Arduino and have no idea what it is or does.  All you really need to know is that it is a programmable device that has the ability to turn on/off certain digital pins through programs referred to as sketches.  Those on/off pins will essentially become our buttons, where button press is determined by sending a low signal (see PoorStudentHobbyist blog for details).  Ergo, we are basically going to create a circuit and wire-up a glorified blinking LED demo, the blinking light is going to be NES controller buttons.

I followed the pinouts and wiring guide used @ PoorStudentHobbyistBlog but I did not use the proposed switch.  At this point, try running a sample program to see that your interface works by sending a low signal to the start button every few seconds or so.  The blog post above includes a sample for exactly that.

Here is a picture of the completed interface:

Sub-Problem #2 – Speaking to our Hardware Interface from C#

We are going to leverage open-source software to interface with our board via C#.  A very common scenario when developing for IoT boards is the ability to control the pinouts via an external interface i.e. a Web API or in our case the serial port.  Lucky for us, Firmata is an open-source protocol for doing exactly that!  Firmata is so pervasive that it is actually included as a default sketch in the Arduino IDE.  Simply, upload the standard Firmata sketch to your device.  Now we need to setup communication via C#.  Again, luck for us, we can leverage Arduino4Net which lets us speak to Firmata to control our board via C#.  Bonus, Arduino4Net can be brought in easily using Nuget!  At this point, you will want to create a simple test where you can verify that Arduino4NET is properly passing signals to your board.  I have included one as part of the Kinect4NES project.

Sub-Problem #3 – Creating the Kinect Application to simulate control signals based on gestures

We are going to connect up the Kinect V2 sensor and create a gesture scheme to signal button presses through our interface!  The Kinect V2 SDK includes something called the SDK Browser 2.0.  Inside you will find a Body Basics XAML sample.  Install the sample and copy out to somewhere where you can modify.

The Kinect V2 SDK Browser:

We begin by intercepting the Reader_FrameArrived method when datareceived is true.  Keeping in mind that Kinect can track more than one body, we take one of those bodies and call CalcController(Body body).  Inside this method, we setup the logic for controlling which pin we wish to signal to based on defined gestures which are determined from the joint tracking points.  All of this starts with trial and error, but essentially you make considerations based on where the joints are in relation to each other.  We could also train a gesture using the Gesture Builder tool which is included with the SDK, but that is for a later post =)  Simply find some space, get a body, and start doing some gestures and determine how to best capture!

The Kinect4NES Application:

Solution:

Once you have all this, put all the pieces together and turn on your favorite game!  I chose the pinnacle classic Super Mario 3 which worked well enough with our scheme to actually allow you to play through the first level!  Next thing to consider is trying other games out and possibly allowing for multiple gesture profiles.  For example, I have created a stub in the hosted project to play Mario by physically jumping and running as opposed to using hands.  All in all, this was an extremely fun hack that allowed me to bridge my interest in class video games with modern gaming peripherals!

If you want to get the bits and follow along with updates or even contribute to this project, you may want to check out the GitHub Project Page for Kinect4NES.