Al3xa LEGO Plott3r

The Idea

My idea for the Alexa/LEGO MINDSTORMS challenge was to hook Alexa up with my 2D pen plotter (Plott3r) that I already had designed, so that various thing could be requested to be drawn. The plan would be to add an additional EV3 running ev3dev which would act as a message broker between Alexa and the existing EV3g code. An additional benefit of implementing the message broker mechanism is that this could be applied to other EV3g programmed robots with minimal effort.

The Challenges

Apart from learning Alexa Utterances & Intents, NodeJS, and Python, the main challenges of this project were the communication aspects of Alexa <-> ev3dev, and ev3dev <-> EV3g. These can be broken down in to:

• Sending and receiving EV3g format "Mailbox" messages within Python on the ev3dev brick.
• Sending messages back to Alexa to be said.
• Obtaining the Shopping or To-Do lists to be sent to the Plott3r.
• Getting GeoJSON map data over to be drawn.
• Obtaining a Sudoku, and possible solution, to be plotted.

EV3g Mailbox Messages

In order for the ev3dev brick to communicate with the stock LEGO EV3 operating system, the EV3g format Mailbox message would need to be implemented in Python. This has a relatively simple format of:

LL DDDD N n...n\0 MM m...m
LL = Message Length (2 bytes)
DDDD = Direct message type identifier (4 bytes)
N = Message name length, includes NULL byte (1 byte)
n...n\0 = Message name (2+ bytes)
MM = Message payload length (2 bytes)
m...m = Message payload

I have previously written a library for App Inventor 2 to perform this task, and have blogged about my experiences:

https://r.jander.me.uk/index.php/2016/08/16/my-first-bt-message/https://r.jander.me.uk/index.php/2016/08/25/update-on-my-bluetooth-ev3-mailbox-testing/https://r.jander.me.uk/index.php/2018/03/04/receiving-bt-mailboxes-from-ev3-by-an-ai2-app/https://r.jander.me.uk/index.php/2018/03/09/updated-bt-comms-between-ev3-and-ai2/https://r.jander.me.uk/index.php/2018/03/10/further-ai2-ev3-bluetooth-coding/

However as I was new to Python, I decided that it may make more sense to look for libraries that would perform this encoding/decoding task for me. The few I found were either incorrectly implemented or were incredibly bulky for my needs. Instead I wrote a new Python class to handle this task - which in itself was a good learning exercise! The new class also attempts to perform automatic decoding of the contents, which can only be determined from the size and NULL bytes in the payload. I have also blogged about my experiences of doing this:

https://r.jander.me.uk/index.php/2019/11/25/ev3-mailboxes-in-python/

I have made this class available from:

https://gitlab.com/Jander/ev3-mailbox-python

Inter-Brick Communication, Status, and Reporting

The standard Bluetooth library (import bluetooth) was used for the communication. Earlier code had been written to handle the Bluetooth I/O within the main code. This worked in general, but mean all communications had to be synchronous - if the EV3g brick sent a Mailbox message with a name that wasn't being expected, it'd break the flow of the code. To solve this I wrote an additional Mailbox handler class that deals with all the sending and receiving of messages. That handler is then queried for any outstanding messages with the name requested. Every time a message needs to be sent to the EV3g brick the state of the connection is checked, making a new connection if needed. Any errors in sending or receiving a message are reported back to the Alexa user, along with dropping the Bluetooth connection. This also allows for the EV3g brick to be autonomous from the other EV3 as connections will be remade on demand. This extra class is included in the git repo above.

One aspect that would need to be handled is that interaction with Alexa can happen at any time, but the Plott3r can only be doing one thing at a time. This means that the status of the Plott3r must be checked before it's instructed to perform an activity. If it is busy at the time the user asks it to do something, the user must be informed of this.

The status can be explicitly asked for, but is also checked on each command request. The _status method handles all the reporting back to the user, so the main code only needs to test for the IDLE status:

def _status(self):
    """
    Get the status of the EV3g brick
    """
    status = Plott3rStatus.UNKNOWN
    try:
        self._send_mailbox("Status", platform.node(), Mailbox.Type.TEXT)
        mailbox = self._recv_mailbox("Busy", 5)
        if mailbox != None:
            status = Plott3rStatus.BUSY if mailbox.value else Plott3rStatus.IDLE
    except Exception as e:
        print(e)
    if status == Plott3rStatus.UNKNOWN:
        self._send_event(EventName.Speak, {'speech': "Plotter state is unknown"})
    if status == Plott3rStatus.BUSY:
        self._send_event(EventName.Speak, {'speech': "Plotter is busy"})
    return status

The reporting of speech from Python is based upon mission-05 in the examples for this challenge. At the time of writing I still have the occasional issue where Alexa may repeat the speech a second time - I haven't yet identified what's causing that.

Shopping and To-Do Lists

The major goal of this project was to have the Plott3r write out the user's shopping or to-do lists. The baseline code for accessing those lists was based upon the Alexa Cookbook code:

https://github.com/alexa/alexa-cookbook/blob/master/feature-demos/skill-demo-list-access/lambda/custom/index.js

I have utilised sections of that code so that it can now select a list based upon its suffix, and will return a string containing the contents. The suffix is determined from the optional ID that can be associated with the Slots aspect of utterances, which is discussed in the next section.

Utterances, Slots, and Consistent Identifiers

When building the Intents for the model, samples of the expected speech are required, along with identifying the contents of the Slots. The example missions had multiple Slot definitions for the same thing, e.g. "backward" and "backwards". This required that the Python code also understood those multiple commands.

In the Slot definitions it is possible to add synonyms and a SlotID value. This is used for all the utterances in this model. This allows for "shopping" list to also be requested as "grocery", "groceries", "market", and "supermarket" list. The same principle is used for the drawings, so that the same drawing can be referred to in more than one way.

Getting this slot ID was non-trivial. There is a method to get the Slot value, Alexa.getSlotValue(request, 'SlotName', but there isn't an associated function to get the ID. This has to be performed by accessing attributes of the Slot itself, e.g.:

// Get the picture type request, and its ID
const picture     = Alexa.getSlotValue(request, 'PictureType');
const pictureSlot = Alexa.getSlot(request, 'PictureType');
const pictureID   = pictureSlot.resolutions.resolutionsPerAuthority[0].values[0].value.id;

GeoJSON Map Data

One thing I decided late on in building and coding this, is that I wanted it to be able to draw out maps. Hunting around I found a free-to-use GeoJSON dataset, from:

https://github.com/AshKyd/geojson-regions

A variation on the cookbook's httpGet method was implemented:

function GetGeoJSON(query) {
return new Promise(((resolve, reject) => {
    var options = {
        host: 'raw.githubusercontent.com',
        path: '/AshKyd/geojson-regions/master/countries/110m/' + encodeURIComponent(query),
        method: 'GET',
    };
...

My first plan was to have the Alexa Skill NodeJS code obtain the data and then send that to the ev3dev brick. The repo has an index.json file that could be parsed to find the correct country data file, which allowed me to use the AMAZON.Country slot type, which would help with some of the validation. Initial coding would retrieve the full map data and try to send it to the ev3dev gadget. This, however, never worked - the message never arrives. I could only assume that message payload size was far too big.

The second version would tell the ev3dev brick to plot a map, but only supply the country name and the filename for the data. This message certainly works, but now requires the ev3dev brick to be on the WiFi - given that the Echo needs to be online, this isn't too much of a problem. The country name is matched on UPPERCASE:

const indexData = await GetGeoJSON('index.json');

...

let mapFile;
let mapName;
Object.keys(indexData.all).forEach(function(key) {
    if (country.toUpperCase() === indexData.all[key].countryName.toUpperCase()) {
    console.log(`Country KEY = ${key}, CountryName = ${indexData.all[key].countryName}, FileName = ${indexData.all[key].filename}`);
    mapFile = indexData.all[key].filename;
    mapName = indexData.all[key].countryName;
    }
});

This may lead to issues with some maps where the country name has accents, e.g. Côte d'Ivoire, but I'm not sure how AMAZON.Country represents those - I'll look for a better string match soon.

The ev3dev brick now downloads the information for the country and corrects the data for Long/Lat -> X/Y projection. The raw Longitude values cannot be used directly as X coordinates as that will result in horizontal stretching of the map further away from the equator. I thought I'd got that sorted, as I was always testing with the UK, or other European countries, e.g.:

I was, however, proved wrong in my coding when I tried to draw Australia. Once I realised that I needed to remap the "meridian" to the centre line of the map I wanted, all went well:

It was during the development of this part of the project that I realised there were issues with the synchronous Bluetooth communications. It takes some time to transmit the data, during which time every data point is responded to with an "ACK" message. During this phase it'd be possible to ask Alexa about the status of the Plotter - this would result in a "Busy" message response. The unexpected arrival of that message would break the program flow. By implementing a message handler that essentially pigeon-holes messages as they arrive, and then informs any waiting listeners solved this problem.

Drawing out a Sudoku

The final capability I wanted to add was that of Sudoku puzzles. Generating these on the ev3dev brick in Python may have been a little too slow for the hardware, so I looked around for any online sources of Sudokus. Eventually I found www.websudokuo.com that had a clean HTML page that could be parsed, including an ID and the solution hidden in there.

The ID can be used to request the answer to the Sudoku - instead of rendering the puzzle grid, the embedded answer would be rendered instead. For this you ask "Draw me the answer to Sudoku number _ID_on_the_page."

Putting It All Together

Putting all these pieces together I now have an EV3 running Python that can accept events from Alexa with a consistent ID, and also send speech back to Alexa to be said. The Python code can send and receive messages via Bluetooth to/from an EV3 running the stock EV3g language.

The result of this is the ability to ask Alexa the following (or variations on):

• Draw me a spiral
• Plot me a Hilbert Curve
• Plot a mini figure
• Draw me a female mini fig
• Write out my shopping list
• Print my to-do list
• Is the plotter busy?
• Take a memo - This will then ask you what your memo is, and whether it heard correctly.
• Draw me a map of _country_, eg Draw me a map of Germany.
• Draw me an (easy | medium | hard) Sudoku

Code and Build Instructions

The code for this project is available from: https://gitlab.com/Jander/alexa-plott3r. This includes:

• Alexa NodeJS, and Utterance data
• Python code to run on the ev3dev EV3
• A copy of the EV3Mailbox and EV3Messages classes code
• The EV3g that runs on the stock EV3

In the EV3g directory is the EV3g code and some other rtf files. Those files will need to be downloaded to the EV3 brick after the code has been. They contain the data files for the mini-figures, the letter data, and a calibration file for managing the backlash in the Plott3r. Without those files, the Plott3r will error when it starts up.

Build instructions for the model can be obtained from:

• http://jander.me.uk/LEGO/resources/Plott3r.pdf
• http://jander.me.uk/LEGO/resources/Plott3r-PenHolder-v2.pdf

The latter link is an improved version of the pen holder which is used in the videos shown. A second EV3 support will be needed for the left EV3 - simply build a second instance of the holder in the main instructions and mount it on the left side of the Plott3r.

Example Videos and Pictures

The following video shows a map being drawn. There is a ~50 second delay (from 0:20 to 1:10) in the video where the ev3dev brick has downloaded the data and is encoding and transmitting the data via Bluetooth to the EV3g brick. This is unfortunately slow and cannot be made faster.