Chromotherapy with Raspberry and Arduino

How can technology help us to improve our lives ?

That is the question I asked myself when I thought about the new project to be realized .

Every day I spend many hours working on computer, I would like that the technology "take care" of me, trying to understand my moods and to find a ways to relax and make me happy.

How can my pc interpret my expression and let me be able to make me feel good?

The answer was immediately clear ... using Microsoft Cognitive Services and my favorite gadgets (Raspberry and Arduino).

Let's talk about my project, let's go!

The Idea

In chromotherapy , each color is associated with particular mental and spiritual characteristics of individuals , and supporters believe that the same would have particular effects on the functioning of the body .

There are no clinical or scientific evidence of the merits of those unproven assertions .

Each color is then associated with specific properties , often based on simple psychological analogies.

I created a system able to see my expression, understand if i am happy and decide what is the best color of the room in order to change my mood.

I Think IT'S VERY COOL!!!

Hardware setup

My hardware is shown below

Software

Arduino

The first step is to create communication between my devices (Raspberry and the Arduino). To accomplish this, I decided to use SoftwareSerial Library. I was inspired by the projects hosted by David Jones Project and Jiong Shi.

Next Step is to change the default name of my Bluetooth device. Default name of the module is HC-05 and it's not cool!

To change bluetooth module's name, we should study and use AT Commands. Refer this link for more information about AT Commands and how to use them using Arduino and Bluetooth Module.

Communication between Raspberry and Anrduino, as I said, is made up of a Blueetooth connection. I decided to implement a method in the UWP application that sends a string that contains 3 character (equivalent to 3 byte) to Arduino, and Arduino receives and decode input data.

Just for your reference, a section of the Arduino decodation code is as shown below (complete source code is available in code section below):

if (BTSerial.available())
{ 
	// Read string from Raspberry 
	delay(10); 
	ch1 = BTSerial.read(); 
	delay(10); 
	ch2 = BTSerial.read(); 
	delay(10); 
	ch3 = BTSerial.read(); 
	delay(10); 
	if (ch1 == '1') 
		digitalWrite(GREEN, HIGH); 
	else 
		digitalWrite(GREEN, LOW); 
	if (ch2 == '1') 
		digitalWrite(BLUE, HIGH); 
	else 
		digitalWrite(BLUE, LOW); 
	if (ch3 == '1') 
		digitalWrite(RED, HIGH); 
	else 
		digitalWrite(RED, LOW); 
	... 
	... 
	... 
} 

Now is time to create a new solution and implement our code!

Universal Windows Platform App

I decided to explore the Emotion API from Microsoft (please see the link for more details about the SDK).

If you are creating UWP project from scratch, you must install the "Oxford" Package, open the "NuGet Package Manager Console" in Visual studio and type Install-Package Microsoft.ProjectOxford.Emotion

Make sure you have generated your Emotion Service Key over Azure. To do so, go to the this and click on "Buy on Azure"

Then, you will be redirected to your azure account and now you have to compile some fields, as you see below I used a "Pricing Tier Free" that allow you 30000 image transactions per month. (If you don't have a free azure account, please visit this link)

Now explore your Cognitive service account in Azure and go to Keys section.

There are 2 keys, choose one and "copy and past" the value in your MainPage.xaml.cs in the following string:

string subscriptionKey = "xxxxxxxxxxxxxxxxxxx";

Essential for UWP

To access Bluetooth and other various features like webcam, pictures, etc, we need to mention them in "Package.appmanifest" file.

To do so, follow steps mentioned:

Code Snippet:

<Capabilities>
  <Capability Name="internetClient" /> 
  <uap:Capability Name="picturesLibrary" /> 
  <uap:Capability Name="removableStorage" /> 
  <DeviceCapability Name="bluetooth.rfcomm"> 
    <Device Id="any"> 
      <Function Type="name:serialPort"/> 
    </Device> 
  </DeviceCapability> 
  <DeviceCapability Name="webcam" /> 
  <DeviceCapability Name="microphone" /> 
</Capabilities>

Basic Idea (Image Capture, UWP App)

We need to convert the captured image into MemoryStream. To do so:

var reader= new Datareader(photoStream.GetInputStreamAt(0);
var bytes = new byte[photoStream.Size];
await reader.LoadAsync((uint)photoStream.Size);
reader.ReadBytes(bytes); 
var stream = new MemoryStream(bytes);

So now, using a single call, you can retrieve emotion result from Azure:

Emotion[] emotionResult;           
emotionResult = await emotionServiceClient.RecognizeAsync(stream);

Hardware

UWP Application

In the First picture you can see the application start screen. On the left side it show the picture captured from webcam. In the middle the Emotions status and on the right side you can configure the Bluetooth connection with the Arduino.

Each time that an Image has been aquired, the system evaluate what is the best color for light to turn on and send via Bluetooth to the Arduino. (As you can see from the code behind, I created a Timer that captures pic every 10 seconds.)

At the moment the UWP application runs in my PC because it is easy to make a screenshot, but it is an UWP App, so it will be not a problem to deploy app on Raspberry.

Currently, the system manages 3 colors stated below, but it will be very easy to create the logic, you would prefer:

if (emotionResult[0].Scores.Happiness >= 0.85) 
{ 
	Send("111"); 
} 
else if (emotionResult[0].Scores.Happiness >= 0.50 && emotionResult[0].Scores.Happiness < 0.85) 
{ 
	Send("001"); 
} 
else if (emotionResult[0].Scores.Happiness > 0.25 && emotionResult[0].Scores.Happiness < 0.50) 
{ 
	Send("100"); 
} 
else 
{ 
	Send("110"); 
} 

I assumed color combination as:

000 = Cyan
001 = Light green  // Mild Happy
010 = Violet
011 = Red
100 = Light blue   // Neutral
101 = Green
110 = Blue
111 = OFF          // Happy

As you can see, if you are happy you don't need to have a light :)

Let's see in action

Future Developments

• The Emotion Api are able to determine a status score for each person in the image frame. So you will be able to determinate an average value of the happiness :) and decide to turn on the best color of your room

• Create a nice wrapper for the Hardware

• In this project I evaluate only the "Happiness", but you can try to develop an algorithm that considers all status available in the API... it's very easy!!