Lobot 2.0: Wearable Headgear

This project was inspired by the character Lobot in The Empire Strikes Back, a character who wears an "AJ6 Biocomp" headgear which allows him to communicate with the computers of Cloud City.  Originally, I planned to update a costume prop I built to the Particle platform and add the functionality in this project to that prop. However, I decided it would be more interesting to see if I could build a wearable device for modern times that could be worn "everyday" and see what functionality I could add or use.  So, "Lobot 2.0" was born.

The Lobot AJ6 headgear device is a wearable computer that interacts with a Pebble Time smartwatch and a web page, allowing you to track the location of the device, interact with features on the device and receive GPS coordinates from a specific web page or the Pebble Time, and navigate to that destination.  There is no voice communication in this project.  I implemented audible cues to help the wearer navigate and alert them when they have arrived at their destination.

The last 30 seconds of that video is a brief demo of the headgear in navigation mode.  If you listen carefully, at the start of the navigation demo, the headgear beeps to navigate and go forward in rapid succession.  This could be improved in the code as I work on this to ensure these different audibles do not collide (navigation mode beep, seaking bearing beep).

Project Summary

There are 3 parts to this project:

• Audio headphones serve as the platform to host (currently) a flashlight (bright LED), speaker, temperature sensor, accelerometer / compass, GPS, panic lights and buttons.  The headgear can navigate from one GPS coordinate to another.
• Web page - simple HTML file that interacts with the headgear and displays real time data about the headgear, including GPS location and compass heading.
• Pebble smartphone integration - Pebble Time and a custom app can be used two ways.  First, to control the flashlight and panic lights of the headgear, and second, to send the current GPS coordinates from the watch to the headgear.  The headgear will enter navigation mode and guide the wearer to the watch wearer's location.

Software

This project is compiled using Particle firmware 0.4.6 due to the MLX9016 temperature sensor library - it would not run on newer firmware.  Please take note of that.  Also, the web page works flawlessly on Firefox 44.0 on Mac OSX 10.10.5.  I tested also Safari on iPad iOS9 and IE8 on Windows 7 and there are some minor deficiencies in web page layout and function across browsers which appear to relate to differences in how CSS is handled.  You should still be able to interact with the features but the compass arrow may not work as expected (uses CSS transforms).  The layout may not be perfect, either, causing some elements to appear behind others on a smaller monitor (less than 1280x1024 resolution).

Included in the github repository are app.js (Pebble Smartwatch app), Spark_Lobot.cpp (Photon app), several adafruit libraries for the Photon build (from several places) and two web pages - index.htm is the main page.  Index2.htm can be used to send gps coordinates to the photon, in place of using the Pebble integration.  The web pages are stand alone and not secure. I chose to do this to simplify the install - no dependencies on node.js or other web frameworks, although client side JavaScript and 2 JavaScript libraries are used.

Headgear Features

GPS, Compass and Accelerometer

A GPS module is attached to the back of the headgear, and it is used in two ways. First, the headgear is programmed to publish its current GPS coordinates every 3 to 4 seconds, which the web page reads (using Spark JS) and formats for display, including using Google Maps API. Second, the current GPS position is used in calculations while in navigation mode. To improve responsiveness of the GPS and to prevent losing my cloud connection, I adapted the Adafruit_GPS.h file to allow me to collect only position data (not time and velocity) by adding this one line to the header file.

#define PMTK_SET_NMEA_OUTPUT_GGA "$PMTK314,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0*29" // added - position data only

The compass is used to calculate the direction the wearer is facing and is reported on the web page as their "heading".  Compass data also is used in Navigation Mode (see below).  The accelerometer reports the tilt on x and y axis, so we will know if Lobot is sleeping on the job ;). The measurement on z axis is the approximate gravitational force.

Navigation Mode

When the smartwatch sends a command with its current GPS coordinates to the headgear, the headgear enters navigation mode (by means of a software flag). 

//* Navigation
int navigating = 0; // 1: we are navigating and should sound audible alerts
int distance = 0; // distance in meters between gps coords
int arrived = 0; // if distance < 10 meters, = 1
int seekBearing = 0; // if 1, then use sound to orient user until their bearing is toward destination
double landoLAT = 0; // destination to pebble watch
double landoLON = 0; // destination to pebble watch
double bearing = 0; // the direction we should travel while navigating

While in navigation mode, the headgear calculates the distance between the wearer and the smartwatch and also calculates a "bearing" (not to be confused with the wearer's current compass "heading").  The bearing indicates which direction the headgear wearer should walk to reach the smartwatch destination.  The wearer will hear an audible beep in the speaker during navigation. 

If the wearer presses the Push Data button (behind left ear), the headgear enters "Seek Bearing" mode.  In this mode, the headgear recalculates a bearing to the destination using the compass data, and uses audible cues for left and right to inform the wearer to look left or right until they are looking in the correct direction.  I left a range of about 10 degrees within which the user is consider to be looking in the correct direction.

At that time, the headgear issues a different  audible tone indicating the wearer is facing the correct way, and then the headgear returns to Navigation Mode.  When the wearer has arrived within 10 meters of the destination, yet another melody is played and the wearer can turn off navigation mode by pressing the Push Data button.

Flashlight

The flashlight can be turned on and off three ways: by button press on the headgear, by a button on the web page and by a button press on the smartwatch.

AutoLight Mode (Photo-Resistor)

There is also an "AutoLight mode" which can be enabled through the web page.  The wearer must first press the Push Data button so that the headgear can calculate the current light conditions and a threshold value.  Then, while in AutoLight mode, if the on-board photo resistor detects a drop in light levels below the threshold value (for example, room lights off), the flashlight will automatically turn on until the light level is higher than the threshold value. Pressing the Push Data button will recalculate light levels (among other things).

Temperature Sensor

The temperature sensor is mounted above the left ear.  It can collect both the ambient temperature, as well as the infrared (IR) temperature of an object near the headset.  The IR temperature is only published when the user presses the Push Data pin on the headgear, located behind the left ear.  As a design decision, I made the assumption we only care about the IR temperature value when the wearer wants to publish that data.

Panic Lights

There are 2 red LEDs on the headgear which are meant to draw attention to the wearer - perhaps as a visual beacon to locate the wearer or to indicate an emergency in progress.  The panic lights are activated by the push of a button on the headgear, the web page or the smartwatch.

Finally, there is a button to restart the device under the left earphone.  This restart can also be initiated from the web page.

Web Page Features

I know it is not a pretty design.  The web page is self-contained (meaning it does not require node.js or a server to run).  It is not built on any specific frameworks, either.  And it is not secure, so do not publish this page to any web servers.  It is meant to run from your desktop and requires a web browser, JavaScript enabled, and an Internet connection.  It has dependencies on the Spark JS 1.0 JavaScript library and Google Maps JavaScript API, both of which are referenced in the web page.

To make the web page work, edit the index.htm file in a text editor program and enter your Particle device ID and authentication token as strings, then save the file.   For this reason (saving your ID and token in the file), do not publish this file to a public server / online with your token or ID embedded.

var accessToken = "your token"; //particle access token
var deviceID = "your ID"; // particle device id

The web page JavaScript calculates a text heading (N, NW, S, SW etc) and this can be further refined to be more accurate than my values.

As soon as your photon connects to the cloud, the web page will begin receiving data and display the Photon's startup progress in the bottom text box.

Web Page Explained

At the top of the page you will see your Device ID and Authentication token.

On the left side are "command" buttons to issue commands to the headgear such as to enable or disable the flashlight and panic lights or play the navigation sounds and melodies (buttons with numbers). 

Additional buttons appear across the middle of the page representing all the variables exposed on the photon - these variable names are hard-coded in this web page, they are not discovered. Clicking these buttons will retrieve a JSON formatted string output of the variable's value from the Particle servers to display in the bottom text box. 

At the bottom of the page is a "text box" which will show the JSON data that was received from the headgear after each button press.  Also displayed here will be any "reports" which are published regularly, (approximately every 10 seconds) and "pebble" interactions (on button press).

In the middle of the screen are three textual boxes which display the parsed JSON data in a more legible format.  To see all the data fields populated, be sure to click the Push Data button on the headgear and the AJ6 Stat variable button in the middle of the web page.

Finally, on the right side are two navigation boxes - the top one displays the current location of the headgear on a Google Map using the headgear GPS coordinates.  The bottom box is a compass displaying the current "heading" of the headgear in near real time (updated every 7 seconds).

Smartwatch Integration

If you do not use the pebble watch integration, it is still possible to send decimal formatted GPS coordinates to the headgear through a web page, also included in the files of this project.

I used Pebble Time smartwatch and an iPhone for this part of the integration which allows me to control features (currently, turn on and off the flashlight and panic lights ) and send GPS coordinates to the headgear which activates navigation mode on the headgear.  To run the smartwatch app (app.js) as I developed it (thanks to many tutorials and sample code at Pebble.js), you will need a Pebble Time watch and a cell phone or other mobile device.  To use the Navigation feature, it must be possible for the installed pebble app to retrieve location data from the device.  You will also need an account at cloudpebble.net to develop the watch app.

Last, you will need an account at Pebble Developer Portal to receive a Timeline UUID, if you wish to use the timeline feature I used, which publishes "pins" to the Pebble watch timeline.  This feature is not necessary to this project, it is just a work in progress.  I used this feature because I wanted also the possibility to update the pins, giving the watch wearer a sort of "ACK" that the headgear wearer is on their way.  Although this is not implemented in the headgear as of Jan 31, 2016, you can achieve this "ACK" through a web page URL using CURL.  Once your app is published to the developer portal (Pebble App store), your application can remain private and you can use the Timeline UUID in the watch app.  Again, the use of timeline pins is not necessary and you can comment them out of the pebble app.

Pins in Pebble App.js

// Create the pin
// comment out from here to **** END PIN **** if not using pins
id = "pin-" + Math.round((Math.random() * 100000));

In the Pebble app, adapt the device ID and access token which will be used by the smartwatch app to communicate with the Particle cloud / headgear:

var deviceID = 'your device ID'; // particle deviceid
var accessKey = 'your access token'; // particle access token

See the video above a demo of the smartwatch sending GPS coordinates to the headgear.  Here is a sample of the Pebble Smartwatch log file recording the interactions of the user sending GPS coordinates to the headgear.

[PHONE] pebble-app.js:?: JS: starting app: 793CD583-XXXXXXXXX LandoJS
[PHONE] pebble-app.js:?: JS: LandoJS: (+) [menu 1] : [menu 1]
[PHONE] pebble-app.js:?: JS: LandoJS: Regular launch not by a timeline event.
[PHONE] pebble-app.js:?: JS: LandoJS: XXX.80303619377119,XXX.09296012173114
[PHONE] pebble-app.js:?: JS: LandoJS: LandoJS creating PIN pin-82926 for time 2016-01-31T23:49:23.275Z
[PHONE] pebble-app.js:?: JS: LandoJS: Registering pin in watch: {"id":"pin-82926","time":"2016-01-31T23:49:23.275Z","layout":{"type":"genericPin","title":"Pickup-pin-82926","tinyIcon":"system://images/NOTIFICATION_GENERIC"}}
[PHONE] pebble-app.js:?: JS: LandoJS: LandoJS lat long
[PHONE] pebble-app.js:?: JS: LandoJS: lat=XXX.80303619377119 lon=XXX.09296012173114
[PHONE] pebble-app.js:?: JS: LandoJS: My timeline token is SBXXXXXXXXXXXXXXX
[PHONE] pebble-app.js:?: JS: LandoJS: timeline: request sent.
[PHONE] pebble-app.js:?: JS: LandoJS: timeline: response received: OK
[PHONE] pebble-app.js:?: JS: LandoJS: LandoJS insertUserPin Result: OK
[PHONE] pebble-app.js:?: JS: LandoJS: LandoJS ajax success

Particle Software

Please build using firmware 0.4.6.  The code is commented to explain the purpose of functions and variables and to credit sources.

The libraries included in this project work together. I have made a couple minor alterations in some libraries, such as the Adafruit_GPS.h file to limit the GPS output (mentioned above). I kept all the source code comments intact so you can still find the original code / authors online, if needed.

To interact with the headgear, in this project, 9 variables and 2 functions are exposed:

SENDGPS - used by the smartwatch and index2.htm to pass GPS coordinates and a timeline pin identifier in a structured string to the headgear for processing.

CMD - is a function that receives a string value and based on that value will perform some task like turn on and off lights, restart the headgear, play sounds, collect data points and publish and update variables.

The biggest challenge in developing the software was controlling the timing of events and limiting the use of delays to prevent loss of cloud connection.

timer = millis(); // controls publish every 10s
timer2 = millis(); // controls GPS publish
timerPanic = millis(); // frequency of the panic LED blink
timerAccel = millis(); // checks accelerometer
timerMag = millis(); // checks compass
timerNavSeek = millis(); // used while navigating to help user gain a bearing

Headgear Build

Below is a summary of the steps I took to modify the headphones for use in this project.  I built a breadboard prototype first, before attempting to install anything to the headgear.

First, find some headphones you do not mind destroying.  I like the look and feel of behind-the-head headphones and these ones can be found online for about $20.  You may wish to use headphones that are larger and offer better places to hide components.  Now, disassemble the earphones carefully and remove the speakers.  I use only one speaker in this project because the earphones are too small to house the photon and a speaker together.  I used a .25w, 8ohm speaker in this build.  In the right earphone, I placed the Photon and needed to carve a lot of plastic away to seat the Photon snug and secure.  Since I was using a photon with header pins, I needed to cut these down to short stubs.

Next, make openings in the plastic outer shell so you can reset the device (I use a modified paper clip to reach the mode and reset buttons), add the uFL antenna (I do not use it) and see your status and board LEDs.  I expose the D7 LED because I programmed it to blink when the Photon is executing certain code and it pays homage to Lobot's movie prop.

Third, attach your components.  I added a photo resistor "AutoLight", flashlight (super bright 3.3v white LED, housed in a PaperMate pen body glued to the earphone), 2 bright red LEDs, the accelerometer and GPS.  I also added 4 buttons to activate features.  Obviously, if you use a smartwatch integration, you will probably have access to GPS and accelerometer data and will not need these extra modules.

Next, install wiring and solder everything in place.  I kept the breadboard alongside my build and used alligator clips to connect the headgear wiring back to the breadboard until the last possible minute to ensure everything worked as it was moved from breadboard to headgear. I still had one issue, though, a poor solder joint on the ground wire to GPS header which shorted out the compass and accelerometer.  If you look in the video at the start of this project at the demo of navigation mode, the wires are exposed due to the last minute fix which took a day to isolate.

Below, is the finished prototype.  I did not have a battery holder to mount batteries on the headgear; that could be improved upon.  In the meantime, power is delivered through the one cable which runs off the right earphone.  Also, for this prototype, the accelerometer and GPS are too obvious and exposed, for daily use. In the future, I will make sure to run all wires along the headphone armature to further hide the wires.  Hopefully, the idea is clear.  Thank you for reading!