Lego Battle Alexa Voice Controlled Playground

I have Alexa at home and my two kids love Lego building blocks. I never thought I could do something with both that helps me to improve my development skills and that can make my kids to see my work from such a funny perspective. They are always telling me that they want to play with the lego robot and I am not going to say it was easy to develop all the capabilities that this solution offers, so I had to tell them that it takes time to build great things. Now I see them playing with it and I know that this project somehow has had an impact on them and not only on me.

There are two main pieces in this project. One is the Alexa skill, and the other one is the Lego Mindstorms robot. Let's have a look into both of them here below.

ALEXA skill

The skill has been developed in the Alexa Development console where the intents and the slots have been defined.

To create the skill log into Alexa Development Console and click in create skill.

Give it a name, choose custom model, and choose to provision your own back-end resources as we will be using our own Lambda function instead of the Alexa hosted skill functionality.

Once in your created skill paste the json skill model definition that you will find in the GitHub repository provided and save the model. There are three json models for three different languages.

The skill's service endpoint is an AWS Lambda function. In the endpoint section you will need to introduce the ARN of the Lambda function (see the lambda function creation section below) and you will need to use the Skill ID that you can find here.

The Custom Interface Controller interface and the Alexa Presentation Language (APL) interface has been enabled to communicate with the Alexa Gadget (Lego Mindstorm) and to manage the Amazon Echo display in case the device has one.

Language and voices

This is a custom Alexa skill that is available in two languages, English (US and UK) and Spanish (ES).

The skill model definition for the language English (US) and English (UK) are the same. The skill in Spanish (ES) has the same intents and slot types, but the sample utterances are translated, as the slot value synonyms maintaining the same slot value. This way when receiving the directive from Alexa device we can use the same Lambda function handlers for all the languages.

I have only introduces Spanish but any other language can be added using the same approach.

There are two json files in the speech folder of the Lambda function that contains the Alexa speech expressions in both English (en-US.json) and Spanish (es-ES.json), that Alexa will reply back to the user requests, or to the Lego Mindstorm events. One of these two will be selected taking into account the "locale" property of the Alexa device request that indicates the language selected in the Echo device.

There are two main voices in the echo device, the Alexa default voice in each of the languages that will be the skill assistant, and a male voice that will represent the Lego robot voice (Justin for US, Enrique for ES and Brian for UK).

Interactionmodel: Invocation,intents and slots

The invocation model is "lego battle" in English and "batalla de lego" in Spanish.

There are 12 intents defined and 14 slots.

goSomewhere intent allow the user to move the robot in the Lego Battle field. There are 6 places that the robot can go to, that are home, heliport, parking, train station, port and the color line.

There are 4 weapons available for the robot, that are the gun, the hammer, the blade and the picker. The changeTool intent allow the user to tell Alexa that there has been a change of weapons, and useTool intent is used to command the robot to use the weapon on the specified target, or in the current place and direction, in case there is no target defined.

findColor intent provides the user with the ability to ask the robot to find an specific color in the color line.

setSpeed intent allow the user to select between slow, normal or fast speed.

setTarget is an intent to define the target of the robot attack. There are 5 targets defined: the plane, the boat, the tractor, the tree and the mobile target.

Finally the last custom intent is remoteControl, that can be triggered to open the remote control in the echo device screen, in case it has one. With that remote control the user can move the robot forward, backward, left and right 90º touching the different arrows in the screen, rendered using APL.

Also 5 default built-in intents are available. The usual suspects.

Sample utterances have been defined for each of the intents, trying to cover many of the possible ways users may make those request to manage the Lego robot in the Lego Battle field.

A set of 14 slot types have been defined to collect the relevant slots needed for the skill to work properly, such as placeToGo, finalTool, target, speed and colorFind, but some others to cover the possible different user expressions.

Not only one value is valid for each of the slot types, having defined several synonyms to cover more scenarios.

You can find the json definition of the skill in the /lego/models/ folder in the GitHub repository provided. Loading them for the specific skill language interaction models, saving and building them will make everything work perfectly fine, once the mentioned interfaces have been enabled and the Lambda function endpoint is connected.

Lambda function endpoint

To create the Lambda function you need to log into the AWS console Lambda service and create a new function.

Select Author from scratch, give it a name and click in Create function.

In the next screen you need to click in add trigger. In this screen you can copy the function ARN that is needed in the Alexa skill development console to define the Lambda endpoint.

When adding the Alexa Skill Kit trigger, enable the skill ID verification so that only request coming from this skill will be accepted. This skill ID can be found in the Endpoint section of the Alexa Skill Development console. Introduce it and click add.

After this you can deploy the nodeJs code that you can find in the GitHub repository provided (/lego/lambda/), and then everything will be working in what regards to the Alexa skill. Don't forget to install the node modules needed. "npm install" will install all modules listed as dependencies in package.json.

The only piece missing will be the Alexa Gadget product creation in the Alexa Voice Service and the Lego Mindstorms building and configuration. Before explaining that let's dive a bit deeper into the nodeJs code pieces provided.

Clone the Github repository locally and open it with Visual Studio Code or your preferred development application.

The lego folder contains:

• documents folder with 2 APL documents to be rendered in Alexa device screen. Sequence, that is the generic one for this skill, renders squence of images and hidden texts, so that Alexa is speaking speeches while showing images, by means of Sequential, Scroll and SpeakItem parallel commands directive. Arrows document will present arrows images embedded within TouchWrappers that will send events to move the robot in different directions.
• images folder with a json file that contains the links to the images stored in an S3 bucket, or the root of the folder that contain the images. The S3 buckets are made public so that they can be accessed from lambda without having to manage permissions just in case this is needed for the contest but it is recommended to set up those permissions only allowing this lambda function to access the bucket images. The S3 bucket images are also provided in the Github repository /lego/s3images/ folder, with the same folder structure that I used within the bucket.
• node_modules folder containing the modules used and needed for nodeJs execution. This will be created when npm install command is executed.
• speech folder where there are two json files containing the sentences that will be said by alexa as response to user echo device directives and custom Alexa gadget events. One will contain the English version (en-US) used in both the Us and UK skill languages, and the other the Spanish one (es-ES). The skill uses some Speech Synthesis Markup Language capabilities like emotions (<amazon:emotion name="excited" intensity="high">), audio (<audio src="" />), available speechcons / interjections (<say-as interpret-as="interjection">), effects (<amazon:effect name="whispered">) and voices (<voice name="Justin">).
• common.js file expose 6 handlers for the Alexa skill Built-In intents (HelpIntent, CancelIntent, StopIntent), the Error handler, and a Request Interceptor to log the request.
• index.js file is, with resources.js, the main nodeJs file containing the skill backend logic. It is the code to handle the different intents triggered by the user through the Echo device, and to manage the events sent from the Lego robot. LaunchRequestHandler will take care of the first response when the skill is opened. ChangeTool, UseTool, GoSomewhere, FindColor, SetSpeed and SetTarget intents work more or less the same way. When one of these intents is received the main slot is collected and the function calculateSequenceDatasource from resources.js is called in order to get the datasource, commands, speech output and directives to be used in Alexa ResponseBuilder to render the sequence APL document in the alexa display (if it has one). This is giving the user a response and sending a directive to the Lego Mindstorms so that the robot executes the action requested by user. RemoteControl intent is a bit different calling the function getRemoteControl to get the datasource and speech to be used when rendering the arrows APL document. This one contains TouchWrapper components that will trigger UserEvents that are handled in TouchEventHandler and analyzed in getClicked function from resources.js, in order to get the directive to be send to the robot brick and the speech to be said by Alexa. Finally EventsReceivedRequestHandler will take care of the custom events received from the Lego Mindstorm, so that Alexa reacts saying something and showing images in the display.
• resources.js file contains functions that support the intent handlers included in index.js mentioned above, and some ev3 related functions like buildStartEventHandler that builds the directive to start the Event handler, getConnectedEndpoints to get the connected endpoints so that Alexa can send the directives to the connected gadget, and build function to build custom directives payload to the gadget.

Alexa Gadget

As described in Mission 1 page (https://www.hackster.io/alexagadgets/lego-mindstorms-voice-challenge-mission-1-f31925) you can register the Alexa Gadget in the Alexa Developer Console.

Following these instructions a product was created in Alexa Voice Service, getting the Amazon ID and Alexa Gadget Secret that are needed to be included in the file lego.ini that you may find in the GitHub repository (add your own Amazon ID and Gadget secret in this file).

Also described in this page it is the how to establish the Bluetooth connection with the Echo device. Follow those instructions there.

LEGO Mindstorms

Lego Mindstorm Tracker building

I was looking at all the possible robots that can be built with the Lego Mindstorms kit and after some time deciding between SPIK3R and TRAK3R I finnaly built the tracker following this instructions:

https://www.lego.com/cdn/cs/set/assets/blt5703aa5eb10dfc68/31313_TRACK3R_2016.pdf

But I was not alone doing that, as my kid helped me to build it. He loved it and enjoyed a lot building his first robot ever. What can a kid like more than doing this with his father?... Here you can see the video building it.

We have build all weapons available. Besides the blade in the video, we built the gun, the picker and the hammer.

It was a huge fun!

Besides that I have added two sensors: the color sensor and the touch sensor so that the Track3r is now capable of tracking colors in the color line and also can be reset in home position when it loses its position in the battle field.

Here some pictures of how the color sensor is connected to the brick.

Some other images to explain how to connect the touch sensor in the lego robot.

EV3Brick Setup

I have followed the instructions in LEGO MINDSTORMS Voice Challenge: Setup page (https://www.hackster.io/alexagadgets/lego-mindstorms-voice-challenge-setup-17300f) in order to set up the EV3 development environment and prepare the EV3 Brick to work with an Echo device and Alexa.

Download the ev3dev software, flash the SD Card and connect to the EV3 Brick (in my case with the cable). Installing the ev3dev-browser extension in Visual Studio Code you can connect with the device, send the workspace files, and execute the python program with the SSH terminal.

The instructions are well detailed in the link provided above.

There are some known issues with Bluetooth in Lego Mindstorms so in case you cannot see the Bluetooth in the EV3 Brick try restarting it unplugging all the sensors connected and if it still not working, try to re-flash your SD Card.

Python code

The code that is transferred to the Ev3 Brick and executed to receive control directives via Bluetooth from the Alexa Echo device has this structure:

There are two folders:

• images folder contains.bmp images that are downloaded from https://sites.google.com/site/ev3devpython/learn_ev3_python/screen/bmp-image-collection. These images are displayed in the Ev3 Brick lcd when the robot receives a Custom.Mindstorm.Gadget control directive.
• sounds folder contains 4 sound files taken from a.wav sound library found in here: https://sites.google.com/site/ev3devpython/learn_ev3_python/sound. These sounds are used to make the robot express that has received some info like a target or speed. And also horning when finding the indicated color, or playing a laser sound when the robot uses the weapon.
• lego.ini contains the Amazon ID and the Alexa Gadget Secret, used to communicate the Echo device with the Alexa Gadget that represents the Ev3 brick. This is information that you can obtain as it is indicated in the Alexa Gadget section above or in mission 1.
• lego.py is the python code that contains all the logic to be executed in the robot, receiving control directives from Echo device that are translated into actions like using a motor, showing a image in the display, reading the value of the color sensor, sending events back to Alexa, etc.

Capabilities, key code pieces and video demonstrations

Here below I will try to explain all the capabilities of the Lego Battle voice experience with schemas, key code pieces and videos demonstrating the different functionalities. The idea is not to present all the code for all the capabilities, but to explain some of them, so that the main idea about how all the experience was developed gets clearer. The code in the pictures is reduced to the most relevant code lines, so it is just to give an idea and it is recommended to see the whole code in the Github repository provided.

Skill Launch

When the user pronounces the skill invocation name, "open lego battle", Alexa receives the LaunchRequest directive and then, by means of the Echo device screen and the voice, Alexa introduces the Lego Battle story. In the second part of the sequence, with an image of the robot, a new voice representing the robot speaks and asks users for commands to be executed.

During the LaunchRequest response management in Lambda nodeJs function, Alexa sends the Ev3 brick a CustomInterfaceController.StartEventHandler directive with an specific token to enable the events in the brick to be sent to and to be acepted by Alexa Echo device with that token.

Another functionality that is present when opening the skill is the APL document rendering. I am using one APL document called sequence.json that can be found in documents folder. Main properties are shown here below.

The document is a template with containers that have an horizontal sequence component with image components that are binded to the document via the listImages array in the datasource. Another container contains invisible text components that will be used in the SpeakItem command.

Everything that is needed to render the APL document is calculated in the calculateSequenceDatasource function in resources. First we initialize the datasource and commands. Depending on the intent, the speechOutput that contains the array of strings with the sentences that Alexa has to say, the array of images that are going to be supporting each of the sentences, and the directive to send to the Ev3 gadget, are calculated. And then all these arrays are navigated and put in the right place within the datasource and commands json.

With all these data the Alexa.Presentation.APL.RenderDocument and Alexa.Presentation.APL.ExecuteCommands directives can be used in the responseBuilder. In case the APL is not supported by the Echo device, then only the sentences are said, and the directive to the gadget is sent.

It looks difficult but is simple. Here my kids open and using the skill, setting fast speed and moving to the Heliport.

Spanish enabled language

As explained before skill was created to support English (US and UK) and Spanish (ES).

The skill is also available in Spanish. As you can see in the videos, my kid and myself can speak Spanish with Alexa lego battle skill (in this case the invocation name is "batalla de lego"), and the skill works the same way it does in English.

The two json files (en_US.json and es_ES.json) contain the speech transcription in both languages. When they need to be used in resources functions the locale attribute in the received request is analyzed and taking into account the device language it will use one json file or the other.

Lego Brick starting routine

When executing lego.py program in Lego Ev3 Brick, all the motors, sensors, Ev3 capabilities and python variables are initialized. After that there is a short routine of images and one speak command that show the robot is alive.

show_image function was created to facilitate the image show in the Ev3 Brick lcd screen, getting the images from the image folder.

Find Color command

When the user asks the robot to find an specific color, Alexa is receiving the findColor intent and getting the color slot. Alexa then shows the color to be found in the screen and sends a CustomInterfaceController.SendDirective directive to the Ev3 Brick, passing the color and speed in the payload. Here some python source code that relates to findColor directive.

After having imported the ColorSensor class, and having created an instance of that, color related variables are initialized, and a thread is created to manage the color search routine.

When receiving a custom directive of type 'findColor' the color is persisted in a variable and the findColor function is called. In this function robot moves to color line and set findColorIOn variable to True, making the code to execute a while routine to search for the specified color within the _find_color function associated to the mentioned thread. Robot starts to move following the color line. If the color is detected then the motors are stopped, and a FindColor event is sent to Alexa with color detected in the payload. If the color is not found then the same event is sent but in this case with color "none" as payload.

Here an example of the robot trying to find color yellow that my kid is getting out of the color line.

Let's have a look into the Lambda nodeJs code that manages the findColor Intent.

Once the findColor intent is received the response is calculated, and the directive type findColor is built to be sent to the gadget. Whether color is found or not, FindColor event is received, and again the function to calculate the sequence datasources is called with value FindColorEvent or NoFindColorEvent.

Set speed and robot movement

In this section we will talk about the setSpeed and the goSomewhere intents, that are understood when the user ask Alexa to move to a place defined in the battle field (heliport, parking, train station or port), and when the user set the speed of the robot.

Some videos to demonstrate it with the robot moving from one place to the other at different speeds.

Let's see some python code related with the robot movement.

The robot position and the different places in the field are defined with an X and Y axis value. The orientation is a value between 0 and 2*pi radians. Robot starts in position (1, 1) and orientation 0, that is the home place position too.

When the gadget receives a goSomewhere control directive then it goes to the indicated target place (toPlace). If it is currently (fromPlace) is home position or the target is the home position then there is one stop in between the initial and the final place that is the position homeEntrance. This is for the robot to leave home straight forward and to enter home backwards.

The distance to move from initial to target point is the hypotenuse of the the X axis journey and the Y axis journey. The direction is calculated using arc tangent and then the turn to be made is the difference of the direction with the current orientation.

Once this is calculated the movement is performed starting the turn to face the target point, moving forward the calculated distance, and turning again to get in the target place position orientation.

Once in that new place the setPosition function is called to persist the new position for next movements.

The turn and the move functions are taking into account the radius of the wheels and the distance between wheels in the robot. With some mathematical equations the degrees that the motors have to move are calculated.

setPosition function persist the final position of the robot in the field using the internal variables defined in the gadget class.

In what regards to the nodeJs code that takes care of the goSomewhere intent there is nothing new to say.

The calculateSequenceDatasource is called to get the data needed to render the APL document in Alexa display showing the picture of the target place and sending the goSomewhere control directive to the gadget.

As you can see in the code picture, in case the remote control has been used, or the position is unknown because of a successful find color intent, then a different value is passed to the sequence calculation function, the same if the position that is requested is the current one the robot is in.

Setting the target, weapons change and usage

Here we will explain how weapons work. These tools are connected to the Ev3 medium motor. There are four weapons available, the gun, that is the default one, but also the hammer, the blade and the picker. Depending on the weapon the action to execute is different. The motor reacts different if the robot has to fire the gun, or if is has to smash the hammer, or use the blade.

When the user sets a target the robot uses the weapon on that target, but if there is no target set, then it uses the weapon in the place it is at the time it receives the command.

Here you can see a video where the boat and the tractor are set as target and then the robot fires the gun on them. Like in most of the directives managed in the Ev3 brick some images are presented in the LCD screen and a sound is emitted.

Now smashing the hammer on the tractor.

Video changing the tool from the hammer to the gun.

Let's have a look on the particular python and nodeJs code for these intents.

The position of the different targets are defined when the MindstormsGadget class is initiated. As we did with the goSomewhere directive we calculate the distance using the hypotenuse and the direction using calculateDirection function that uses the arc tangent, mathematical function that is the inverse of the tangent function.

As we are using different weapons, we have defined an offset, that is the distance the weapon reaches from the robot position, so when the robot faces the target it stops the offset distance before the target position. The position remains known as we know where we stopped in each target and weapon scenarios.

When the weapon is the gun it is fired without moving from the position. There is only a change in orientation.

When the target defined is the mobile one, then the robot has to first find out where the target is. In this case the target is the infrared sensor remote control beacon. The beacon signal that the robot reacts to is the one in channel 4.

The function facingTarget calculates the direction of the beacon. As the Infrared sensor is in the back of the robot then we sum pi radians to the heading returned value, converted into radians.

The distance takes into account that there are approximately 70cm infrared range, so the value given by the infrared sensor distance function is multiplied by 7.

I will only comment something more about useTool nodeJs code, as setTarget and changeTool intents have nothing special comparing to others already explained.

When a useTool intent is received and the user gives the weapon name and the action to execute, the code checks that these two are connected, so that you are not trying to fire the hammer or smash the gun. If the tool is not the right one for the action requested then the robot will inform the user, suggesting to change the weapon if that is needed.

Alexa Display remote control

Triggering the remoteControl intent, Alexa will show a remote control in the Echo device screen that contains arrows that can make the robot move forward, backwards, left and right. Here is a video of my kids using it.

In this case the APL json document that is rendered is the arrows.json. It is a container with a background image and another container with a TouchWrapper components around arrow images that send events with the direction of the arrow pressed as argument.

When the user push one of the arrows an event generated in the TouchWrapprer is sent and analyzed when received in order to send a custom directive to the Ev3 gadget that triggers the movement action.

RemoteControl and Touchevent handlers call getRemoteControl and getClicked resources functions to get the information needed to give the right response back to the user, and the directives to send to the gadget.

If the remote control is used the position is lost and then the user has to put the robot back into home position and push the red button in the robot. This is demonstrated in this video, where you can see that Alexa tells the user that the position is unknown and what to do. When the user clicks the button the robot position is set to "home, sweet home".

In the following python code lines you can see how the TouchSensor class is instantiated, and how the thread that manages the button being pushed is created and how it works.

If the remote control has been used the remoteControl variable has value True and then, if the button is pressed, the position is set to home, and the event HomeButton is sent to Alexa.

Infrared remote control

The Infrared remote control has been configured to manage the robot movement, controlling the left motor and the right one, with channel 1 left and right buttons. In channel 2 the third motor that activate the weapons is controlled with the top right and top left buttons. Here a demonstration video picking one lego car with the picker using the remote control.

Here you can see some python code related with the Infrared remote control.

An instance of class InfraredSensor is created and used to define what needs to be triggered when each of the buttons in channel 1, and channel 2 are pressed.

There is a thread created to check if one of the infrared remote control buttons has been pressed, so that it triggers the process of that function connected with the button. Each of the buttons in channel 1 call remote_move function, selecting the left or the right motor and the speed. Within that function, if the button is pressed (state = True) the motor will run forever and it will stop when the button is released (state = False).

Remote control is really simple to manage once you get used to it. Here my 5 years old and my 3 years old kids using the remote and having fun with lego battle.

Lego Battle skill in Fire TV

I tried to connect without success the FireTV with the Alexa gadget. How cool would it be that the skills that uses APL are executed in FireTV so that you can see the images and videos in such a big screen.

Here a simulation of how you see the skill running in the TV using FireTV, but as the gadget is not Bluetooth connected to FireTV, no response is provided from the Lego robot. I changed the code a bit to record this video because if there is not gadget connected, the skill informs user about that, and when asking questions the skill fails when sending directives to an undefined gadget endpoint.

I hope you enjoyed the project described here and just connect with me in case of questions.