Pac-Man on a Phone - Jedi Hand Controls with a Photon

This tutorial shows you how to use hand movements (and a cleverly palmed Photon) as a game controller,  running a Pac-Man app on a Android or iOS device over a local WiFi. Remember those garbage mashers on the first Death Star? An underlying purpose is to also introduce you to hybrid app development, and give you a starting point to make your own projects based on WiFi and the Photon. We've stacked our microcontroller with an IMU shield that will be used to control Pac-Man himself in the game.

In order to make the most of this tutorial you'll need a Particle Photon and a smartphone (iOS/Android). I will also assume that you are familiar with Particle Build. If you’re not, Particle provide an excellent introduction on their web page to bring you up to speed.

The basic Pac-Man app (controlled using arrow keys on a PC-based browser) has been rewritten entirely in HTML5 by Platzh1rsch,  developer of the pacman-canvas application. This project tutorial is based on this legacy work, as found in this repository.

Introduction

This update from earlier editions of the Photon includes a 120 MHz (STM32F205) microprocessor and a 6x RAM from previously being 128 KB. The development board is also equipped with both DAC and CAN, features fairly unusual to this kind of development board.

In short, the team behind Particle has developed an impressive and thought-out offering, where the development environment is mainly cloud based (Particle Build) although more experienced users may choose to opt-in for the “offline” Particle Dev software.

SparkFun were kind enough to provide a Photon and a couple of their shields. Among the shields I got my hands on were the Battery Shield and the IMU Shield. Ever since I begun developing examples using Evothings, I have thought that it would be interesting to try to add connectivity and external controls to an already existing, and normally stand-alone mobile application. And being written in HTML5/JavaScript, it's good to go on both Android and iOS using Evothings Studio software.

Source Code

You can browse the source code for this tutorial in Evothings GitHub repository.

Also you will find Platzh1rsch pacman-canvas in its original version in his GitHub repository.

What You Need

In order to follow this tutorial I assume that you are familiar with Particle Build. You need the following hardware:

• A Particle Photon (Spark Core should work although I have not tested it)

• A SparkFun Photon IMU Shield

• Optional – 1 x SparkFun Photon Battery Shield

• An iOS or Android smartphone

• A local network with a DHCP server

• A standard breadboard

Step 1 – Hardware

The first, and most simple step in this tutorial, is hardware preparation. Simply put the shields together and you are ready to start coding. I prefer to put the stack on a breadboard to minimize the risk of unintentionally shortcut any of the exposed pins. You can view my stack in the picture below.

Step 2 – Embedded Software

Preparation

As stated earlier I assume that you are familiar with Build, so we'll dive directly into the code. The complete source code described below can be found on the Github repository.

The objective is to build a small server on the Photon. When a client connects to the device, it will begin sending data in JSON format containing acceleration values to a connected client. We also need to configure the shield so that it communicates using the I2C bus. In this tutorial I will assume that the Photon and the smartphone are connected to the same local network. Yet with some additional networking skills,  our example can be expanded to be usable over the public internet if one is interested in that.

// This #include statement was automatically 
// added by the Particle IDE
#include "SparkFunLSM9DS1/SparkFunLSM9DS1.h"
// Configure SparkFun Photon IMU Shield
#define LSM9DS1_M 0x1E 
#define LSM9DS1_AG  0x6B
// Configure example#define DELAY_BETWEEN_TRANSFERS 50 // (milliseconds)
#define SERVER_PORT 23
#define RGB_BRIGHTNESS 128
#define RGB_R 0
#define RGB_G 255
#define RGB_B 0 TCPServer 
server = TCPServer(SERVER_PORT);
TCPClient client;LSM9DS1 imu;
void setup() 
{ 
 Serial.begin(9600);   // Start listening for clients   
 server.begin();	   // Initialize SparkFun Photon IMU Shield   
 imu.settings.device.commInterface = IMU_MODE_I2C;   
 imu.settings.device.mAddress = LSM9DS1_M;   
 imu.settings.device.agAddress = LSM9DS1_AG;   
 imu.begin();
}

In the code we have included the SparkFunLSM9DS1 library that allows us to communicate with the SparkFun IMU Shield, along with some #defines which can be used to tweak the application to fit specific needs. In this case we assume that it is left untouched. The next step is to define the server and client which we use to communicate with clients over the WiFi-connection. And lastly we define the imu  which is our interface to the IMU shield.

Inside setup() we enable the serial communication that we will use to print the received networks settings. Then we start listening for TCP connections by executing the server.begin() method. The IMU shield is initiated after some settings have been updated. Now we have configured and initiated the serial connection, TCP server and the IMU shield.

The following part covers the loop() function that is executed continuously on the development board.

void loop()
{   
 if (client.connected()) 
 {      
   // Discard data not read by client       
   client.flush();      // Take control of the LED      
   RGB.control(true);      
   RGB.color(RGB_R, RGB_G, RGB_B);      // Read IMU data      
   imu.readAccel(); 		      // Create JSON-object       
   char buffer [40];       
   size_t length = sprintf(buffer,       
    "{\"ax\": %.3f, \"ay\": %.3f, \"az\": %.3f}\n",      
   imu.calcAccel(imu.ax),      
   imu.calcAccel(imu.ay),      
   imu.calcAccel(imu.az));         // Flash LED           
   RGB.brightness(RGB.brightness() == RGB_BRIGHTNESS ? 0 : RGB_BRIGHTNESS);      
   // Transfer JSON-object      
   server.write((uint8_t *)buffer, length);   
 } 
 {
   // Turn on LED and release control of LED      
   RGB.brightness(RGB_BRIGHTNESS);      
   RGB.control(false);      // Check if client connected      
   client = server.available();   
 }   
 // Send network information if serial data 
 // is received   
 if(Serial.available()) 
 {      
   Serial.println(WiFi.localIP());      
   Serial.println(WiFi.subnetMask());      
   Serial.println(WiFi.SSID());      
   while(Serial.available()) 
   {        
     Serial.read();      
   }
 }   
 delay(DELAY_BETWEEN_TRANSFERS);
}

The application main if-else statement checks whether or not there is a client connected to the development board. If there is a client present, the application starts to discard data not yet read by the client by executing the  client.flush() method. Then the software takes control of the RGB LED and defines a color that will be used to signal if there is a client connected to the development board. The imu.readAccel() method updates the internal data structures containing the latest acceleration data fetched from the shield. This method is called before calling the imu.calcAccel() method in order to get the latest sampled acceleration. The next step is to define a buffer that we fill with a JSON-object containing the latest acceleration data. Each time before the buffer is transmitted to the client by calling the server.write() method the RGB LED on the development board is flashed.

If no client connected is to the Photon, the application ensures that the LED is turned on and the control is released. Then the application reviews if there is a client connected.

If the application receives data from the serial connection, it transmits the network information from the current network connection. This includes the local IP address, subnet, gateway and the name (SSID) of the network. As a last action the implementation reads and discards all data received from the serial connection.

Before we're done with step 2, I'll need to compile and flash the software onto your development board and open a serial connection to the board, send any data to the board and note the network information. Note the IP of the board, we will use that later in the mobile application.

Step 3 – Mobile Application

The complete source code described below can be found on the Github repository.

Fetch the pacman-canvas commit 3c24c7a995b1e329c5849ac24421eedfc1d70474 from GitHub. Unzip the application and open the files index.htm and pacman-canvas.js in an editor of your choice. These are the only two files that we have to edit in order to achieve our goals. Let’s begin with index.htm .

The first thing that we will do is to remove some code snippets that we don’t need. Remove the code from the following sections:

• Google Analytics

• Google Adsense

• Highscore

The reason that we remove the Highscore section is that it is implemented using the PHP scripting language, which is a technology not supported for executing an application in a mobile client context. Locate the<div> with the id menu-button and delete the following code:

<li class="button" id="highscore">Highscore</li>

The next step is to add the code snippet below just before the </head> tag.

<script>	
// Redirect console.log to Evothings Workbench.	
if (window.hyper && window.hyper.log) 
{ 		
  console.log = hyper.log	
}
</script>
<script src="cordova.js"></script>

This code redirects every console.log()  call to Evothings Workbench special function hyper.log() in order to debug applications back to the Tools section of the Workbench. The rest of the snipped includes a required Cordova-specific library.

The final item on the list  is to add two buttons to the main menu. One that we will use to connect the Photon and one that will take us back to the start screen. Locate the <div> with the id menu-control and add the following code:

<li class="button" 
id="photon-control">Connect Photon</li>
<li class="button" 
id="evothings-back">Back</li>

There is nothing left to edit in the index.htm at this point. Press Save and so move on to editing thepacman-canvas.js file. As a first step, we are going to create an object that will represent the Photon. At the top of the file, right after the declaration of the global variables add the following code snippet, don’t forget to add the IP of the Photon:

// Photon handlerfunction 
Photon()
{  
  // IPAddress and port of the Photon  
  var IPAddress = 'YOUR IP ADDRESS HERE'  
  var port = 23 
  var socketId  
  var previousDirection = {}  
  this.connect = function() 
  {    
     // Return immediately if there is 
     // a connection present    
     if(socketId)
     {      
        return    
     }    
     chrome.sockets.tcp.create(function(createInfo) 
     {      
        socketId = createInfo.socketId;      
        chrome.sockets.tcp.connect(socketId,
        IPAddress,port,connectedCallback)      
     })    
  }  
  this.disconnect = function () 
  {    
     chrome.sockets.tcp.close(socketId, function() 
     {            
     socketId = 0;     
     // Reset graphics and callbacks associated         
     // with the button in the main menu      
     $('#' + previousDirection.name).css('background-color', '')      
     $('#photon-control').text('Connect Photon')      
     $(document).off('click','.button#photon-control').on('click','.button#photon-control',
     function(event) 
     {                  
        photon.connect()                
     })              
  })  
}  

The method connect()  tries to connect to the development board using the defined IPAddress and port . This method is basically a wrapper for the chrome.sockets.tcp.create()  that ensures that the application only is connected to one development board.

function connectedCallback(result)
{    
  if (result === 0) 
  {      
     // Update graphics and callbacks associated 
     // with the button in the main menu       
     $('#photon-control').text('Disconnect');      
     $(document).off('click','.button#photon-control')                   
    .on('click','.button#photon-control',                   
     function(event) 
     {                  
        photon.disconnect();                 
     });    
     // Set callback to handle received data      
     chrome.sockets.tcp.onReceive.addListener(receiveData);    
  }    
  else 
  {      
     alert('Failed to connect to Photon!' +
     ' Try again!', function() {})    
  }  
} 

The callbackconnectedCallback()  is executed asynchronously when there is a result of the connection attempt. The callback evaluates if connection succeeded or not. If there is an established connection (result === 0) the main menu is updated to handle a disconnect and a callback to handle (receiveData() ) is configured. If the connection failed for some reason, an alert dialog is showed on the screen and the socketId  reset to zero, in order to enable new connection attempts.

// Function to handle received data.   
function receiveData(info) 
{    
  // Convert buffer to string containing 
  // the sent JSON-object    
  var jsonString = String.fromCharCode.apply(null, new Uint8Array(info.data))    
  // Try to convert the string to an actual 
  // JavaScript object     
  try 
  {      
     var jsonObject = JSON.parse(jsonString)
  }    
  catch(e) 
  {      
     return
  }    
  var ax = jsonObject['ax'] 
  var ay = jsonObject['ay'] 
  var az = jsonObject['az']
  // Adjust pacman direction depending on 
  // received acceleration     
  if(Math.abs(ax) > Math.abs(ay)) 
  {      
     if(ax < 0) 
     {        
        adjustPacmanDirection(down)      
     }      
     else 
     {        
        adjustPacmanDirection(up)      
     }    
  }    
  else if (Math.abs(ay) > Math.abs(ax)) 
  {      
     if(ay < 0) 
     {        
        adjustPacmanDirection(left)      
     }      
     else 
     {        
        adjustPacmanDirection(right)      
     }    
  }   
}  

The receiveData()  function is executed each time there is new data received. The first step is to convert the received data to a JavaScript object from which data can be extracted. The second step is to adjust the direction of the pacman depending on the values of the acceleration.

// Move Pac-Man around
function adjustPacmanDirection(direction)
{    
  pacman.directionWatcher.set(direction)    
  $('#' + direction.name).css('background-color', '#008000')
  if(!direction.equals(previousDirection)) 
  {      
     $('#' + previousDirection.name).css('background-color', '')      
     previousDirection = direction
  }  
}

The function adjustPacmanDirection() changes the direction of the pacman figure in the game as well as updates the arrows to show which direction the figure is heading at the moment.

Declare a variable named photon among the global variables by adding the following code on the line below the declaration of the variable mapConfig .

var photon

The next step is to define the variable photon. This is done by adding the following code just below the definition of the game variable (game = new Game() ).

photon = new Photon() 

The next and final step is to add initial callbacks to the buttons we added to index.html earlier in this tutorial. The best place to declare this would be in the $(document).ready() method, since the DOM has to be fully loaded before we can add the callbacks. Locate this method and add the following code where the other buttons callbacks are defined.

$(document).on('click','.button#photon-control',
  function(event) 
  {  
    photon.connect()});
    $(document).on('click',
    {  
    history.back()
    })
  })

That's it, you should now be able to press Run in the Evothings Workbench and set Pac-Man to work on your Android or iPhone.

Conclusions

It was fun to work with the Particle Photon.  I will definitely use the development board as a base in future projects. Using Evothings Studio, I managed to develop the application in one go.

Time for you to start exploring the exciting world of mobile IoT applications.. Download Evothings Studio and start developing your own IoT apps!