Posture Recognition using Kinect, Azure IoT, ML and WebVR

The Idea

With the recent success of depth cameras such as Kinect, gesture and posture recognition has become easier. Using depth sensors, the 3D locations of body-joints can be reliably extracted to used with any machine learning framework, specific gestures or posture can be modelled and inferred. Real world applications in Virtual Reality can be used for Yoga, Ballet training, Golf, anything related to activity recognition and proper postures. I also see application of it in the Architectural, Engineering, Construction and Manufacturing Industry by sending depth sensor data to the cloud to identify correct configurations. It could be used for quality control, anomaly detection and part recognition

This is a proof of concept to detect pose "Y", "M", "C" , "A" and stream the result back to the browser.

This video explains a littlebit how it's hooked up together.

Introduction

My name is Ron Dagdag and I'm doing a series of VR projects. This is the last of series project exploring Virtual Reality as a Maker. I'm not going to create games, I am going to explore different ways to create new experiences and interact in VR/AR. Here are the projects

Part 1: Amazon Echo VR Controller

https://www.hackster.io/RONDAGDAG/amazon-echo-vr-controller-724c89

Part 2: Gesture Controller using Particle Photon and Sparkfun RGB and Gesture Sensor APDS-9960

https://www.hackster.io/18937/augmented-reality-virtual-reality-dinosaur-experience-be9c8c

Part 3: Virtual Reality Fireworks using Color Sensor

https://www.hackster.io/RONDAGDAG/virtual-reality-fireworks-using-color-sensor-779ea7

Part 4: ConstructAR - The Holographic Tool Belt

https://www.hackster.io/team-constructar/constructar-the-holographic-tool-belt-b44698

Part 5: Control your "Earth Rover" in Virtual Reality

https://www.hackster.io/RONDAGDAG/control-your-earth-rover-in-virtual-reality-15a9fe

Part 6: Internet of "Kinect"-ed Things and Azure Machine Learning

Posture Recognition Device in Virtual Reality

Capturing the Kinect Skeletal Data, Visualize it in Virtual Reality and send it to Azure IoT for Machine Learning.

If this project made you interested in Kinect, LattePanda board, Johnny-Five, Azure Services, please click "Respect Project" button and follow me. Feel free to contact me, if you have questions.

Kinect

Kinect is an awesome input device for our Internet of Things world. It is a motion sensing input device that has RGB camera, depth sensor, multi-array microphone. I'm currently using it as a motion capture device viewable in Virtual Reality. I bought this adaptor so I can use it on a windows 10 device, Now I can plug it into a USB port.

Latte Panda

This is a small device but powerful single-board computer with fully installed Windows 10. It functions just like any normal Windows 10 desktop, but with it's size, we can plug-and-play anywhere.

And Since Latte Panda runs full version Windows 10, I was able to install Kinect SDK. That's why I call this project - The Internet of "Kinect"-ed Things.

The great thing about this device also is that it has Arduino on board. Now you can add motor controllers, sensors together with Kinect. In a robotic project, you can use Kinect for SLAM mapping. MIND-BLOWN.

I got the 32-bit version so it would only work with original Kinect, they also sell 64-bit version of Latte Panda that might work with Kinect 2.

Here's the Kinect SDK link that I installed in Latte Panda, I'm using Kinect SDK 1.8

With the form-factor of Latte Panda, sensor capabilities of Kinect, and Power of Azure we can build an awesome Posture Recognition device.

NodeJS

I installed NodeJS to prove that Azure IoT works not just in .NET world but also in other languages too.

I tested this hack first on my desktop computer with Kinect2 sensors. I found an npm package library to access the Kinect 2 data from MS SDK. Here's the link

But since I want it to run on my Latte Panda which is limited to original Kinect, I can't use the same library. I found another project on github that I can use to stream Kinect data to NodeJS using Web Sockets. Here's the link, Thanks to Lightbuzz Software

Here's the Github Repo https://github.com/rondagdag/kinectIoT

There are 2 apps

• Kinect.Server - this app would read sensor data, stream it via web sockets. The Kinect Server app is written in C# and will send data serialized in JSON format. It's a copy of this project. I compiled and run Kinect.Server.exe in the bin folder

[DataContract]
class JSONSkeletonCollection
{
[DataMember(Name = "skeletons")]
public List<JSONSkeleton> Skeletons { get; set; }
}
[DataContract]
class JSONSkeleton
{
[DataMember(Name = "id")]
public string ID { get; set; }
[DataMember(Name = "joints")]
public List<JSONJoint> Joints { get; set; }
}
[DataContract]
class JSONJoint
{
[DataMember(Name = "name")]
public string Name { get; set; }
[DataMember(Name = "x")]
public double X { get; set; }
[DataMember(Name = "y")]
public double Y { get; set; }
[DataMember(Name = "z")]
public double Z { get; set; }
}

• skeletons-browser - A node js app would receive the data from Kinect.Server to communicate to Azure IoT. It also acts as a web server to stream skelatal positions to the browser in Virtual Reality. The main file for this is index.js

There are 2 modes in the NodeJS app. Training and Predictive Mode.

In Training Mode: It would receive Kinect Data, reformat it to for training. For Example, I'm training with Pose "Y", I would pose on the screen and send the Kinect Joints with Result "Y" to Azure IoT. I don't send all my Kinect Streaming data because that would use up all my Azure Credits. To slow it down, I only send 1 frame per second. In training mode, I manually specify the outputResult variable to corresponding identifier.

In Predictive Mode: I still pass the Kinect Data to Azure IoT, this time I will not pass the any result column. It will be identified as null. This will tell Azure Streaming Analytics that I'm doing prediction. In Predictive Mode, I change the code and set the outputResult variable to null.

The NodeJS app would then subscribe to an Azure Event Hub to receive results from Azure Streaming Analytics and Machine Learning. It would pass that data to the browser to be viewed in Virtual Reality.

I could have called Azure Machine Learning Web Service from this NodeJS app directly, but I wanted to do a round trip scenario. It will definitely take longer to get the result from Azure IoT, Streaming Analytics, Machine Learning then to Event Hubs. My goal is to learn as much as I can about Azure Streaming Analytics.

Johnny-Five

Since the Latte Panda board has Arduino installed, I can then use Johnny-Five javascript IoT platform to send data to a Grove LCD RGB Backlight.

I used the sample from this http://johnny-five.io/examples/lcd-i2c/

Here's the pin layout of Latte Panda

GND -> Pin 23

VCC -> Pin 24

SDA -> Pin 6

SCL -> Pin 8

Here are the instructions how to setup the Arduino in Latte Panda

http://www.lattepanda.com/docs/#SetuptheArduino

Here's the code I used just to test it out.

I noticed that you have to specify the port depending on what Arduino IDE detected.

var five = require(
"johnny-five"
);
var board = new five.Board({ port: "COM3" });
board.on("ready", function() {
var lcd = new five.LCD({
controller: "JHD1313M1"
});
lcd.useChar("heart");
lcd.cursor(0, 0).print("hello :heart:");
lcd.blink();
lcd.cursor(1, 0).print("Blinking? ");
});

Web VR

WebVR is an experimental Javascript API that provides access to Virtual Reality Devices like Oculus Rift, Google Cardboard in the browser. WebVR is VR in your browser, without plugins. Just use the browser and be transported.

It's currently available in Firefox Nightly Builds, experimental builds of Chrome and Samsung Internet for Gear VR. It also works with mobile version of Chrome using WebVR polyfill to provide support for Google Cardboard Devices. It's using WebGL and Javascript

A-Frame

A-Frame is an open-source WebVR framework for creating Virtual Reality experiences with HTML. Instead of doing WebGL, It simplifies creating the scene, lighting, rendering, etc. etc.

It's using markup HTML to make it easy. The markup will be converted to WebGL.

It looks something like this

<html>
<head> 
<script src="https://aframe.io/releases/0.3.0/aframe.min.js"></script> 
</head> 
<body> 
<a-scene> 
<a-sphere position="0 1.25 -5" radius="1.25" color="#EF2D5E"></a-sphere> 
<a-box position="-1 0.5 -3" rotation="0 45 0" width="1" height="1" depth="1" color="#4CC3D9"></a-box> 
<a-cylinder position="1 0.75 -3" radius="0.5" height="1.5" color="#FFC65D"></a-cylinder> 
<a-plane position="0 0 -4" rotation="-90 0 0" width="4" height="4" color="#7BC8A4"></a-plane> 
<a-sky color="#ECECEC"></a-sky> 
</a-scene> 
</body> 
</html> 

to build

try it out http://codepen.io/mozvr/pen/BjygdO

Since I can send the skeletal positions, I can stream the positions of each joint using Socket.io

It's still early days of WebVR, the refresh rate and graphics are not yet at par with game engines. To simplify the problem, I opted out using spheres to represent joints. The browser would interpret the joints and position the spheres accordingly. And since it's 3d, I can have depth. It's VR, so I can position the camera where the Head Sphere is, seeing your body in VR.

I will not focus much on the 3D or the VR part of this hack, I attached the source code anyway. Instead, I'll focus on the Azure Services Side.

AZURE

For this project, you would need:

• IoTHub

• Streaming Analytics

• Azure Storage

• Machine Learning

• Event Hubs

First, Do you have an Azure Account?

If you don't have one, sign up at https://azure.microsoft.com/

then login at https://portal.azure.com

Azure IoT

I'm using Azure IoT Hub to receive data from the Kinect IoT device. It will then pass the skeletal joints to Streaming Analytics.

Create your IoT Hub: Click on New > Internet of Things > IoT Hub

Fill in the necessary fields then click Create

I named it KinectIoT and created a resource group KinectIot_ResourceGroup

Once you have a bunch of Azure Services, I highly recommend always add every project to a resource group. It's easier to find.

Find the Resource Groups and select the IoT Hub to pin to the dashboard.

Take note of your hostname

Click the Shared access policies blade > iothubowner policy, and then copy and make note of the connection string in the iothubowner

You need the shared access keys to connect and register your device.

Here's the code I used to register a device.

https://github.com/rondagdag/kinectIoT/blob/master/skeletons-browser/CreateDeviceIdentity.js

You would need to use the connection string from Azure IoT.

I followed the instructions here

Azure Blob Storage

I added a new Azure Storage account to store Telemetry data from Kinect. This will be used by Azure Machine Learning for training the model.

Let's create an Azure Storage. Click New > Data + Storage > Azure Streaming Analytics.

Fill in the Fields then Click Create. I am currently using the Classic deployment model so I can access it on the Azure Streaming Analytics.

You need to create 2 Blob service that will be used by Streaming Analytics as output streams.

• trainingoutput - this is where we dump all training data for Machine Learning

• resultoutput - this is where we dump all result data from Streaming Analytics

Azure Event Hubs

I tried using the messaging part of Azure IoT, where you can grab the event hub details. I was able to read the event hub streams from Azure IoT but I had trouble connecting it with Streaming Analytics as output. It make sense because you have know which device to target.

Azure Event Hubs is used to stream data back to the Kinect IoT Device for now. Since there's no path yet to send back data from Streaming Analytics to output into Azure IoT, I created a new Event Hub

Select New > Internet Of Things > Event Hubs

Fill in the Name, I typically use project name + eventhub, it's easier to find later. Make it part of a resource group. Choose the pricing tier then click Create.

Azure Machine Learning

I'm using Machine Learning to classify the posture. I trained it to identify "Y", "M", "C" and "A" poses.

To create a new Machine Learning Workspace,

Click New > Intelligence > Machine Learning Workspace

Fill in the form, with same resource group.

The dashboard would look something like this, you would click on "Launch Machine Learning Studio"

You would then click Studio

Create a new blank experiment

Drag Import Data

This is where we hook up Data from Blob Storage to Machine Learning

There's a series of steps involved. I recommend watching this 10 minute video, here's the link.

After that I drag the Split Data and use 0.8 fraction.

I used Multiclass Decision Forest to train the model.

When I visualize the Score Model, it looks something like this, Notice that it added Scored Labels and probabilities to classify posture.

When I visualize the Evaluation Model

Good enough to create predictive experiment. I have to deploy it as web service [Classic] because I am using it as Function in Azure Stream Analytics.

Publishing would create this

You have to click Run then Deploy as Web Service. I deployed as "Classic" because of Streaming Analytics Functions is not yet ported to new portal.

It will create a service with API key.

Azure Stream Analytics

Stream Analytics receives messages from IoT Hub to be processed. The output would either store it to a Azure Blog storage for training Machine Learning Model later or call Azure Machine Learning Web Service then pass it to an Azure Event Hub.

Note: We have to go to the old portal for this, the new portal doesn't support "Functions" yet.

Make sure that the region is the same, I use "East US". It will select a Storage Account on the same region. You may have to generate an Azure Storage Account under Data Services > Storage > QUICK CREATE

You should see on your menu something like this. I will be using "Functions"

INPUTS

I typically named my Input Alias as this pattern - <projectName> - Input

This case I named it [KinectIoT-Input]

It will help when we do our SQL Analytics Query

I'm using JSON and UTF8

OUTPUTS

I have 3 OUTPUTS for this Kinect IoT Streaming Analytics

1) [KINECTIOT-OUTPUT] - this is to stream my training data to a blog storage to be used by Azure Machine Learning later. I created a container called "trainingoutput"

2) [KINECTIOT-RESULT] - This is to debug my output data to another blob storage after calling the Azure Machine Learning Web Service. I created a container called "resultoutput"

3) [KINECTIOT-RESULT-EVENTHUB] - This is to send the output data to an event hub to be streamed back to my Kinect Device. I couldn't connect my output from Azure Streaming Analytics to Azure IoT Hub. The only way is to add an Event Hub.

FUNCTIONS

I added a Function that will connect to my Azure Machine Learning. I specify the alias as detectPose. If you publish your Azure ML in Classic Mode, you'll see the web service below.

It will build a function signature like this, this will help me figure out how to call the function in my Query.

QUERY

The NodeJS app sends values to Azure IoT that will receive the data in JSON Format. Streaming Analytics would "flatten" the JSON data with Rows and Columns. That's why my query looks like this.

I only stream the first 11 joints from Kinect which is upper body with corresponding X,Y,Z values.

From [KinectIoT-input], flatten the JSON to Columns, then pass it to KinectIoT-output] only if the Result Column is not null. What this means is that I'm passing a training data. I will use this later to build my Machine Learning Model.

SELECT e.deviceId, e.result, 
GetRecordPropertyValue(GetArrayElement(e.joints, 0), 'depthX') AS X0, 
GetRecordPropertyValue(GetArrayElement(e.joints, 0), 'depthY') AS Y0, 
GetRecordPropertyValue(GetArrayElement(e.joints, 0), 'cameraZ') AS Z0, 
GetRecordPropertyValue(GetArrayElement(e.joints, 1), 'depthX') AS X1, 
GetRecordPropertyValue(GetArrayElement(e.joints, 1), 'depthY') AS Y1, 
GetRecordPropertyValue(GetArrayElement(e.joints, 1), 'cameraZ') AS Z1, 
GetRecordPropertyValue(GetArrayElement(e.joints, 2), 'depthX') AS X2, 
GetRecordPropertyValue(GetArrayElement(e.joints, 2), 'depthY') AS Y2, 
GetRecordPropertyValue(GetArrayElement(e.joints, 2), 'cameraZ') AS Z2, 
GetRecordPropertyValue(GetArrayElement(e.joints, 3), 'depthX') AS X3, 
GetRecordPropertyValue(GetArrayElement(e.joints, 3), 'depthY') AS Y3, 
GetRecordPropertyValue(GetArrayElement(e.joints, 3), 'cameraZ') AS Z3, 
.............
..............
.............
GetRecordPropertyValue(GetArrayElement(e.joints, 9), 'depthX') AS X9, 
GetRecordPropertyValue(GetArrayElement(e.joints, 9), 'depthY') AS Y9, 
GetRecordPropertyValue(GetArrayElement(e.joints, 9), 'cameraZ') AS Z9, 
GetRecordPropertyValue(GetArrayElement(e.joints, 10), 'depthX') AS X10, 
GetRecordPropertyValue(GetArrayElement(e.joints, 10), 'depthY') AS Y10, 
GetRecordPropertyValue(GetArrayElement(e.joints, 10), 'cameraZ') AS Z10, 
GetRecordPropertyValue(GetArrayElement(e.joints, 11), 'depthX') AS X11, 
GetRecordPropertyValue(GetArrayElement(e.joints, 11), 'depthY') AS Y11, 
GetRecordPropertyValue(GetArrayElement(e.joints, 11), 'cameraZ') AS Z11 
INTO 
[KinectIot-output] 
FROM [KinectIot-input] AS e 
WHERE e.result is not null 

If the Result column is null, that means I want to call Azure Machine Learning Web Service. I'm not in Training Mode but I'm in Predictive Mode.

Here's a sample of what it would look like in a CSV format

The result column is my training data, the rest are my joints data from Kinect.

In order to call the ML Web Service, I would create a subquery calling "detectPose" as described below, I get the record property first, get array element.

WITH subquery AS (
SELECT System.Timestamp t, detectPose(e.deviceId, e.result, 
GetRecordPropertyValue(GetArrayElement(e.joints, 0), 'depthX'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 0), 'depthY'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 0), 'cameraZ'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 1), 'depthX'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 1), 'depthY'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 1), 'cameraZ'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 2), 'depthX'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 2), 'depthY'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 2), 'cameraZ'), 
...........
...........
GetRecordPropertyValue(GetArrayElement(e.joints, 10), 'depthX'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 10), 'depthY'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 10), 'cameraZ'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 11), 'depthX'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 11), 'depthY'), 
GetRecordPropertyValue(GetArrayElement(e.joints, 11), 'cameraZ'))     
as dataResult from [KinectIot-input] AS e 
where e.result is null  
) 

After it runs, I started reading the results and pass it to [kinectiot-result-eventhub] and also [kinectiot-result]

Select dataResult.[Scored Labels],  
dataResult.[Scored Probabilities for Class "y"], 
dataResult.[Scored Probabilities for Class "m"], 
dataResult.[Scored Probabilities for Class "c"], 
dataResult.[Scored Probabilities for Class "a"] 
Into [kinectiot-result-eventhub] 
From subquery 
From subquery 
Select System.Timestamp t , 
dataResult.[Scored Labels], 
CAST(dataResult.x0 AS float) as x0, 
CAST(dataResult.x1 AS float) as x1, 
CAST(dataResult.x2 AS float) as x2, 
.....
CAST(dataResult.x7 AS float) as x7, 
CAST(dataResult.x8 AS float) as x8, 
CAST(dataResult.x9 AS float) as x9, 
CAST(dataResult.x10 AS float) as x10, 
CAST(dataResult.x11 AS float) as x11, 
CAST(dataResult.y0 AS float) as y0, 
CAST(dataResult.y1 AS float) as y1, 
CAST(dataResult.y2 AS float) as y2, 
......
CAST(dataResult.y7 AS float) as y7, 
CAST(dataResult.y8 AS float) as y8, 
CAST(dataResult.y9 AS float) as y9, 
CAST(dataResult.y10 AS float) as y10, 
CAST(dataResult.y11 AS float) as y11 
Into [kinectiot-result] 
From subquery 

Here's the sample output to [KinectIoT-Result], The "scored labels" tells me what Azure Machine Learning service predicted. The rest of the columns are joint data.

{ "t": "2016-08-27T12:15:00.0790000Z", 
"scored labels": "a", 
"x0": 0.539950251579285,  "x1": 0.53042995929718, 
"x2": 0.52039498090744,  "x3": 0.512354016304016, 
"x4": 0.439354211091995, "x5": 0.432462215423584, 
"x6": 0.432531297206879, "x7": 0.432533800601959, 
"x8": 0.603047430515289, "x9": 0.558691143989563, 
"x10": 0.43837097287178, "x11": 0.392004281282425, 
"y0": 0.766573369503021, "y1": 0.621958374977112, 
"y2": 0.479696065187454, "y3": 0.392257124185562, 
"y4": 0.55032080411911,  "y5": 0.520365953445435, 
"y6": 0.449876934289932, "y7": 0.42228952050209, 
"y8": 0.561868786811829, "y9": 0.656122446060181, 
"y10": 0.735241293907166, "y11": 0.772029280662537 
} 

This is what my Azure Storage Account Looks like. I have 2 containers on my Blobs. ResultOutput and TrainingOutput. Everytime Azure Streaming Services run, it will add the results to the Blob Storage.

This is where I believe the strength of Azure services in IoT space. Being able to collect telemetry data, pass to streaming analytics, split the data into outputs.

• Gathering Training Data to feed your machine learning model

• With captured telemetry data, use it to call a predictive analytics function and transform the output streaming data.

When I realized this, now I see what Internet of Things is all about. It's not just a building a device, It's adding the Smarts to it. The Predictive Analytics part, it's the real-time data processing and the flexibility of the cloud. Something clicked, MIND-BLOWN.

If this project made you interested in Kinect, LattePanda board, Johnny-Five, Azure Services, please click Respect Thumbs Up button and follow me. Feel free to contact me, there's a lot to digest on this one.