Alexa, Where Are My Keys?

See what's new at Remyx.ai

Alexa is especially well-suited to information retrieval tasks and monitoring assets using wireless home networks. It's natural to consider putting valuables on the grid for fast retrieval. We hack cheap bluetooth low energy beacons for the network range and battery longevity and build a smart application so that Alexa knows where we left the keys.

Demo

Project Code

How to do it...

Hacking Bluetooth beacons

A set of 3 beacons can be purchased for less than $15 and are supported with Android/iOS applications but we'll opt out for privacy. Besides, finding our keys should not turn into finding our phone.

This adafruit tutorial on reverse engineering smart lights helped us control the beacons. Start by turning on the beacon scan for the device address by running:

sudo hcitool lescan

Find & copy the address labeled with the name 'iTag,' then run:

sudo gatttool -I

Connect to the device interactively by running:

connect AA:BB:CC:DD:EE:FF

Try running 'help' to view options or 'primary' to view services:

Running 'char-desc' followed by the service handle as above, we find UUIDs which we look up by referencing the gatt characteristic specifications and service specifications. For more on these services, check out this.

Inspecting traffic with Wireshark, we find that 0100111000000001 triggers the alarm and logically, 0000111000000001 turns it off. Now we have the simple python function:

import pexpect
def sound_alarm(BD_ADDR):
   child = pexpect.spawn('gatttool -I')
   child.sendline('connect {}'.format(BD_ADDR))
   child.expect('Connection successful', timeout=30)
   child.sendline('char-write-cmd 0x000b 0100111000000001')

Next, we focus on creating the Alexa skill to trigger the beacon when we are looking for the keys.

Creating an Alexa Skill and app

We create a skill that will be linked to a local server. Then we configure our server to take any action we'd like, in this case, provide an approximation for where the keys might be located and make the Bluetooth beacon beep.

Flask provides a simple and easy to use python library to serve an application. Using flask-ask, we can configure the server to communicate with our Alexa skill we'll build later. Well serve the application with Ngrok, which will give us an https link we'll need for our Alexa skill. First we built the application with the simplest functionality: to make our BLE beacon beep when triggered.

#!/usr/bin/env python
from flask import Flask
from flask_ask import Ask, statement
import pexpect
app=Flask(__name__)
ask = Ask(app, '/')
BD_ADDR = 'AA:BB:CC:DD:EE:FF'          #Your bluetooth beacon id here
@ask.intent('findkeys')
def retrievr():
   sound_alarm()
   speech_text = "Your keys are around here somewhere."
   return statement(speech_text)
def sound_alarm():
   child = pexpect.spawn('gatttool -I')
   child.sendline('connect {}'.format(BD_ADDR))
   child.expect('Connection successful', timeout=60)
   child.sendline('char-write-cmd 0x000b 0100111000000001')
if __name__ == "__main__":
   app.run(host='127.0.0.1', port='5000')

We used the function sound_alarm() we wrote earlier to make the BLE beep. For the function that will be used for the intent, we add the ask decorator with our intent "findkeys". When we make the Alexa skill on the Amazon developer dashboard we will use this name for our intent. Write this script to a file named app.py and run

python app.py

This will serve your application on http://localhost:5000. Run an ngrok server and copy the https link generated. You'll need it when you configure the Alexa skill. For more details, check out this post.

We have successfully set up a simple application, now we'll write the Alexa skill. Navigate to the Amazon developer dashboard and log in. Click on Alexa and get started with Alexa Skill kit

Create a new skill by clicking Add a new skill. You'll be greeted with a screen like this:

Follow the instructions given by the gui.

• Under the Interation Model tab you'll want to fill out the Intent Schema box with the following:

{
 "intents": [
   {
     "intent": "findkeys"
   },
   {
     "intent": "AMAZON.HelpIntent"
   },
   {
     "intent": "AMAZON.StopIntent"
   },
   {
     "intent": "AMAZON.CancelIntent"
   }
 ]
}

• In the Sample Utterances box, you want to write some sample commands a person might use to invoke the skill. We wrote these:

findkeys find my keys
findkeys where my keys
findkeys I lost my keys

• In the Configuration tab, make sure to choose the service end point to HTTPS. Copy your https link and paste it in the Default box underneath. Account linking can be left to No.
• In the SSL Certificate pick the middle option, "My development endpoint is a sub-domain of a domain that has a wildcard certificate from a certificate authority".
• The Test tab will allow you to test the new skill by typing in one of your sample commands.

Finish filling out the last two tabs until all the check marks are green. Then launch your skill with the Beta Testing feature. This allows you to host your skill on any echo device before publishing it. Follow the instructions on the email link to install the skill on your echo device.

Making Our Skill Smarter

We put the idle computers spread throughout the house to work in querying the bluetooth beacon to report the RSSI signal strength.

Taking readings from multiple machines, we can utilize signal strength to as a proxy for distance. We need to figure out how to use this to compute the most likely part of the house to find the beacon.

We turn to machine learning. A crontab job every 2 minutes, builds up a csv file of RSSI signal strength tuples from various available machines. P

To label each input tuple, we place the beacon in different spots like: 'Bedroom', 'Bathroom', 'Kitchen', 'Living Area' updating the label along the way.

After building up several dozen readings from each location, we load the csv file with pandas and use tree-based models like xgboost's XGBClassifier. In a nutshell, we train with:

clf = XGBClassifier(max_depth=4, learning_rate=0.05, n_estimators=300, objective='multi:softmax', num_class=4)
clf.fit(train, label)
s = pickle.dump(clf, open('bluetooth_model.dat', 'wb'), protocol=2)

Below, we print the hold out predictions and actual values (encoded as an integer), and the confusion matrix, and the run time of a simple script to load the dataset and train a gradient boosting machine.

The xgboost implementation of gradient boosting will handle the missing data from timed out readings, training in a couple seconds.

We use python pickle to persist the trained model. At inference time, we load the pickled model into our alexa retrievr application and run against the most recent readings.

mdl_ = pickle.load(open('bluetooth_model.dat', 'rb'))
preds = mdl_.predict(val)

Alexa is especially well-suited to information retrieval tasks and monitoring assets using wireless home networks. It's natural to consider putting valuables on the grid for fast retrieval. We hack cheap bluetooth low energy beacons for the network range and battery longevity and build a smart application so that Alexa knows where we left the keys.DemoProject CodeHow to do it...Hacking Bluetooth beacons A set of 3 beacons can be purchased for less than $15 and are supported with Android/iOS applications but we'll opt out for privacy. Besides, finding our keys should not turn into finding our phone.This adafruit tutorial on reverse engineering smart lights helped us control the beacons. Start by turning on the beacon scan for the device address by running:

sudo hcitool lescan

Find & copy the address labeled with the name 'iTag,' then run:

sudo gatttool -I

Connect to the device interactively by running:

connect AA:BB:CC:DD:EE:FF

Try running 'help' to view options or 'primary' to view services.

Running 'char-desc' followed by the service handle as above, we find UUIDs which we look up by referencing the gatt characteristic specifications and service specifications. For more on these services, check out this.

Inspecting traffic with Wireshark, we find that 0100111000000001 triggers the alarm and logically, 0000111000000001 turns it off. Now we have the simple python function:

import pexpect
def sound_alarm(BD_ADDR):   
    child = pexpect.spawn('gatttool -I')   
    child.sendline('connect {}'.format(BD_ADDR))   
    child.expect('Connection successful', timeout=30)   
    child.sendline('char-write-cmd 0x000b 0100111000000001')

Next, we focus on creating the Alexa skill to trigger the beacon when we are looking for the keys.

Putting it all Together

Having a model to approximate the last location of the keys, we can add it to the application to improve the statement returned by Alexa. We've modified the script to read:

import os
from flask import Flask
from flask_ask import Ask, statement
import pexpect
import pickle
import pandas as pd
import numpy as np
from collections import defaultdict, Counter
from reverse_read import reverse_readline
app=Flask(__name__)
ask = Ask(app, '/')
@ask.intent('findkeys')
def retrievr():
   os.system("/path/to/repo/sound_alarm.py &")
   speech_text = guess_locate()
   return statement(speech_text)
def guess_locate():
   read_dict = {}
   line_gen = reverse_readline('YOUR_DATA_FILE.txt')
   res_lst = []
   while len(res_lst) != 20:
       ln = next(line_gen)
       if ln.startswith('Host'):
           _, ip, _, reading = ln.split()
           read_dict[ip] = reading
           res_lst.append(read_dict)
           if ip == 'ip.of.one.computer':
               read_dict = {}
       else:
           pass
   val = pd.DataFrame(res_lst).replace({'N/A': np.nan}).values
   mdl_ = pickle.load(open('location_model_file.dat', 'rb'))
   preds = mdl_.predict(val)
   guess = Counter(preds)
   guess = guess.most_common(1)[0][0]
   reply_str = 'Try looking in the '
   if guess == 1:
       reply_str += 'bedroom'
   elif guess == 2:
       reply_str += 'bathroom'
   elif guess == 3:
       reply_str += 'kitchen'
   elif guess == 4:
       reply_str += 'living room'
   return reply_str
if __name__ == "__main__":
   app.run(host='127.0.0.1', port='5000')

We created a new function called guess_locate() which takes a file with the latest recorded rssi signal strengths. It'll then run the samples against our pickled xgboost model and return the most likely location string. This location will be returned when Alexa is prompted.

Since establishing a connection to a beacon can take a few seconds, we run a separate process calling that function in sound_alarm.py.

Check out our demo!