“Lend me your ears!”: Web Bluetooth and Voice Recognition

Introduction and Motivation

This project will show you how to control relays with an Arduino/Genuino101 using its’ on-board Bluetooth Low Energy (BLE) capability through the Web Bluetooth API. I will also show how to add voice control with the "annyang" voice recognition library with no special hardware required! Along the way we'll also get acquainted with the Physical Web.

I began this project to investigate the Web Bluetooth API as a means to control the Arduino/Genuino101 using its on-board BLE capability. What is Web Bluetooth? The best answer is from the developer’s themselves:

“The Web Bluetooth API lets websites discover and communicate with devices over the Bluetooth 4 wireless standard using the Generic Attribute Profile (GATT)". --https://googlechrome.github.io/samples/web-bluetooth/device-info.html

So instead of having to write a native app, we can write HTML/JavaScript/CSS serve it up as a website and allow anyone with a browser to access our device.

Web Bluetooth is one component of a broader effort to create the Physical Web. And what exactly is the the Physical Web?

“The Physical Web is an effort to extend the superpower of the web - the URL - to everyday physical objects. Our premise is that you should be able to walk up to any “smart” physical object (e.g. a vending machine, a poster, a toy, a bus stop, a rental car) and interact with it without first downloading an app. The user experience of smart objects should be much like links in a web browser, just tap and use.” (https://github.com/google/physical-web)

In our case, our smart device won’t be advertising it’s URL. It will be advertising it’s BLE service. We will use a website hosted on Github Pages that allows our browser to connect with our Arduino/Genuino101 over BLE to turn a desk lamp on or off.

So here is one of the beautiful features of the Physical Web. If you build the circuit and upload the sketch to your Arduino, you only have to access the URL of my Github page and be within BLE range of the device. You don't have to write any of the website code at all. You don't even need a Github account. Although this is not a pure Physical Web project, it is a reasonable representation of it of the Physical Web and some of its potential.

Because the Web Bluetooth API works within the browser as a BLE central device, we can take advantage of any browser based functionality to enhance this project. We are opening up a vast range of HTML/Javascript/CSS options to use to interact with our projects.

Addendum:

In the spirit of "Proudly Invented Elsewhere", after putting this project together and doing my write-up, I found this awesome and eerily similar project:

https://medium.com/@urish/start-building-with-web-bluetooth-and-progressive-web-apps-6534835959a6#.7rv8trf4j

It's part of series of excellent posts by Uri Shaked on the Physical Web and Web Bluetooth that are well worth reading!

Some Important Considerations

Before you get started on this project it is very important to consider what you will need.

No special hardware is required to implement voice recognition and any Arduino/genuino101 relay circuit you wish to use is fine. However, I found the Arduino UNO click shield and RELAY click board from MikroElectronika make an elegant and easy to use combination.

These components are not required to demonstrate BLE control and voice recognition as you could simply have the Arduino/genuino101 blink an LED in response to a voice command, but who needs yet another blinking LED example?

Web Bluetooth is still in development and not universally supported on all browsers and the platforms they run on: https://github.com/WebBluetoothC/web-bluetooth/blob/master/implementation-status.md

Currently you can use Web Bluetooth on Chrome behind an enabling flag on ChromeBook, Mac or a Linux system but you cannot use Web Bluetooth in the Chrome browser running on a windows machine. What better reason could you have to finally make your Windows box a dual boot Linux/Windows system?

Additionally you will need an active WiFi connection for the annyang voice recognition library to work. The voice recognition does not occur on the browser, but on the cloud. So no WiFi, no voice control but this is a small price to pay for free voice recognition.

If you would like to develop a website for your own project, you will also need a way to host your HTML/JavaScript/CSS files as a website with HTTPS. There are several options to do this, which are described at: https://developers.google.com/web/updates/2015/07/interact-with-ble-devices-on-the-web?hl=en.

This project will use Github Pages to host the project.

If all this sounds a bit intimidating and you are saying to yourself, “Thanks, but I’ll just stick with nRF master control. . .”, don’t give up before you’ve even started! What you learn will be well worth the effort and I can pretty much guarantee you won’t be able to get nRF Master Control to open the pod bay doors with a voice command!

Before we dive into the details, here is the big picture of the project:

Our browser, Chrome running on a ChromeBook, will access our web site served over HTTPS on github pages, and use the Web Bluetooth API to control our Arduino/Genuino101. The annyang voice recognition API will access the chromebook microphone and send voice signals to the cloud over WiFi and receive the interpreted text back.

The Sketch

The software component of this project is divided into two major components which are the sketch running on our Arduino/Genuino101 and the website, which we will host on Github Pages. This is not a line by line description of either. If you want to understand how it was written, you will have to read the code, comments, and read the examples out there, do some tutorials, etc. I will only highlight a few key points below.

For those who have written sketches using the CurieBLE library or who have tried the sample programs at https://www.arduino.cc/en/Guide/Arduino101, or the projects on Hackster: https://www.hackster.io/arduino/products/arduino-101-genuino-101#projects, this sketch should be simple to understand.

In the first part of our sketch, we create a relay object. In a straightforward application where we simply want to turn relays on and off, this is definitely not required. However, this sketch was initially written for a slightly more complicated project and I chose to reuse it without changing it for this project. It should be noted that this approach will allow flexibility to expand this project in the future if we wish to. For example, It will make it easy to maintain track of the specific relay’s state if we want to expand well beyond 2 relays. What if a relay has a specific on and off interval or perhaps we want to implement additional state variables?

Additionally this approach allows us to implement polling and delay() free code and use a state machine approach as explained in this fine set of adafruit tutorials: https://learn.adafruit.com/multi-tasking-the-arduino-part-1.

As we think about our relay service, we can see that the communication is one way. The central tells the peripheral to turn a relay on or off. We will implement our relay service as a very simple state machine. If the relay is currently in the off state and is written to, then it will be turned on. If the relay is currently in the on state and is written to, it will be placed in the off state.

The code to represent this as a BLE service is below:

/* establish BLE service & characteristics */
BLEPeripheral blePeripheral;
BLEService relayService("917649A0-D98E-11E5-9EEC-0002A5D5C51B");
BLECharacteristic relayCharacteristic("917649A1-D98E-11E5-9EEC-0002A5D5C51B", BLEWrite, 5);
BLEDescriptor relayDescriptor("2902","relay");

This could be represented as a single byte characteristic, but as you read the code you will see that the characteristic is actually 5 bytes long. The reason for this is that this sketch was originally written as a slightly more complex project. The 1st byte of the 5 is used to represent the specific relay. The other 4 bytes are used to represent an on/off interval in milliseconds as a long type integer. Long type integers are represented by 4 bytes. Hence the extra four bytes and the union. This code is not relevant to the project here as it is currently written. I simply reused the code to try out Web Bluetooth. So you can ignore this or use it as you wish.

Looking at the first byte of the characteristic, with just this 1 byte we could potentially use this to identify up to 256 unique relays! This is a certainly a house full of gadgets to control!

Interfacing with the RELAY click board is quite easy. The documentation is a bit sparse: https://github.com/ClickerSupport/Examples/tree/master/Relay, but this is because using the RELAY click board is simply an exercise is using the the digitalWrite() command:

/**   
* pins for each relay specified here.   
* depend on the Arduino click shield specifications, with the RELAY click board    
* plugged into slot 1    
*/ 
#define RELAY1 6  
#define RELAY2 10
. . .
void turnOn()
{
digitalWrite(pinAssignment,HIGH);
}
void turnOff()
{
digitalWrite(pinAssignment,LOW);
}

To determine the pins for controlling the relays consult the Arduino UNO click shield and RELAY click board documentation cited above.

The loop() function continually checks for state changes to the relay objects and toggles them based on this. Again keep in mind that this code was written to not only toggle relays but also provide an option to the user to set a toggling interval, like the intermittent wiper blades of your car. This more complex project will hopefully be the subject of a future tutorial!

The Web Site

Although I love writing Arduino sketches, the real excitement comes with writing our website. As I mentioned above, you don't need to perform this step in order to make the project work, you can simply build the circuit and access it with my web page. However, I hope this section gives you a better idea of what you might need to do in order to build your own project website.

The big three of web design are HTML/JavaScript/CSS. Our website will consist of an index.html file to provide our user interface, style.css to style the site and some JavaScript to control the action and some really great and free to use libraries for voice recognition and styling. I will focus here on the JavaScript code as this is where we implement the Web Bluetooth and annyang APIs. Again close reading of this code and more importantly the samples and tutorials on the web: https://github.com/googlecodelabs?query=bluetooth will help you understand the code and help you modify these examples to suit your own project.

Here is the file structure of our website on github:

In developing this program, I borrowed code and overall structure heavily from the examples I found on the web: Proudly Invented Elsewhere!

index.html - this is our website's front end, or user interface. Pretty simple, add a switch and the obligatory picture of the HAL9000.

relayClick.js - this JavaScript file represents our relay object. In the code snippet below, you can see how the BLE service uuids are represented in the constructor. This will give you an idea of how you might change this reflect your own project’s needs:

class RelayClick {
/**
* customize your project here to reflect the uuid of your service and characteristics.
*/
constructor() {
  this.deviceName = 'relays';
  this.serviceUUID = '917649a0-d98e-11e5-9eec-0002a5d5c51b';
  this.characteristic1UUID = '917649a1-d98e-11e5-9eec-0002a5d5c51b';
  this.device = null;
  this.server = null;
  // The cache allows us to hold on to characeristics for access in response to user commands 
  this._characteristics = new Map();
}

This constructor also creates a cache of characteristics that can be used throughout your program. This cache is a map() of key-value pairs of uuids and characteristics:

this._characteristics = new Map();

Many of the Web Bluetooth example programs you will see on the web either immediately grab the values from characteristics in a series or chain of JavaScript promises: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise or hand off updates to a event handler function for characteristics with the notify property. In our case, we want to be able to write to our characteristic whenever a voice command is recognized. The characteristic cache lets us do this.

connect(){ 
...  
.then(service=>{
    return Promise.all([
    this._cacheCharacteristic(service, this.characteristic1UUID),
    // this._cacheCharacteristic(service, 'uuidCharacteristic2Here'),
    ])
...

As you can see above, we only have one characteristic to cache in the current version of the code, but the comment line below that demonstrates how to handle more than one.

The last line of code here creates our relayClick object, which is available to use within the app.js file:

window.relayClick = new RelayClick();

app.js - this is the file that connects the relayClick.js object with the actions from our index.html file. It is also where we specify and implement the annyang libraries for voice recognition. Our recognized voice commands with their associated function calls are established here:

var commands = {  
'relay 1': relayOne, 
'relay1': relayOne, 
'one': relayOne, 
'1': relayOne, 
'lamp': relayOne,                                
. . .
'two': relayTwo,                
'2': relayTwo  
};

When you are working with the annyang library you will want to call the annyang.debug() function in your code and keep the developer tools console window open. This will help you see the words that are being returned from your voice commands. It is pretty interesting to watch the return feed as you speak into your browser’s microphone:

The bulk of the code is written within the context of an annyang “object”. You can learn more from reading the examples and information at https://github.com/TalAter/annyang. The code I have written here barely scratches the surface of the power and functionality of this amazing library.

style.css - I relied very heavily on this tutorial to help me style the front end:

http://tympanus.net/codrops/2012/09/13/button-switches-with-checkboxes-and-css3-fanciness/

The rest of the website structure is pretty self explanatory. As this is a work in progress, you will see some messiness in the code. My apologies.

Project Recipe

• The circuit is put together by stacking the three major components, the Arduino/Genuino101, the ArduinoUNO click shield and the RELAY click.

• ***** WARNING ***** I do not recommend trying to control high voltage AC or DC devices or exposing the wires of high voltage AC or DC devices. You have been warned. Do not do it. The desk lamp cord I used outputs 10.5 V and 500 mA from a wall wart. You have been warned.

• With the wall wart UNPLUGGED, separate the two wires of the cord.

• Cut one wire in half and strip the plastic coating from about 1 cm.

• Tin the bare leads.

• Plug tinned leads into RELAY1 or RELAY2 and screw down.

• ***** WARNING ***** I do not recommend trying to control high voltage AC or DC devices or exposing the wires of high voltage AC or DC devices. You have been warned. Do not do it. The desk lamp cord I used outputs 10.5 v and 500 mA from a wall wart. You have been warned.

Once the circuit is completed you try and access the device by enabling the Web Bluetooth flag on Chrome and restarting your Chromebook:

Then go to the following URL: https://lachendekatze.github.io/relay-click/

After you give permission to use your microphone, you should see something like the following:

The developer window Console tab shows no errors and all of our voice word commands have been loaded. Now click on the gray BLE button in the top left and you should see the following:

The active button click and the request to connect to the relays device are security features of Web Bluetooth.

You can start speaking now and see both the recognized word feed on the right and hopefully your attached device turning on and off. If you don't have a device attached you can just test the relays. They have an indicator LED when active and also make a pleasing clicking sound. There may be a slight lag between speaking and action on the relay as the voice signals have to be sent to the cloud, recognized and then sent back to you.

If you would like to create your own website go to github.io and create an account, if you have not done so before this, and go through their tutorials.

• Create a new repository for this project.

• I recommend downloading the zip file from my repository and extracting it as a starting point.

• Use your file browser to open up the relay-click folder and select all the folders and files except for the README.md file.

• Drag and drop these files into the upload files window and commit these as master.

• Read through the guide on Github Pages(https://pages.github.com/).

• Set up your example page as part of the tutorial as a test.

• Now go to your project and select the settings icon for your repository.

• Select master as web page:

• Open up Chrome on your chromebook. You will have to enable the Web Bluetooth flag. This is a security measure for the platform in its current state.

• Go to your github pages link. I highly recommend opening the developer tools window to the console for debugging purposes.

• Your browser should now look like screens above unless you have redone the index.html and style.css files.

Conclusion

I hope you enjoyed reading about this project and are encouraged to explore the Web Bluetooth API and annyang library for your own projects!

Essential Reading:

https://medium.com/@urish/start-building-with-web-bluetooth-and-progressive-web-apps-6534835959a6#.7rv8trf4j

https://webbluetoothcg.github.io/web-bluetooth/

https://googlechrome.github.io/samples/web-bluetooth/

https://developers.google.com/web/updates/2015/07/interact-with-ble-devices-on-the-web?hl=en

https://github.com/WebBluetoothCG/web-bluetooth

https://www.talater.com/annyang/