Workflow Monitoring - DialogFlow and AWS Walkthrough

Here is a tutorial to show how to simply connect google assistant to Amazon AWS through DialogFlow.

The goal is to is make a program which is able to manage and response a user voice request.

But first, what is DialogFlow?

Dialogflow is a Google-owned developer of human–computer interaction technologies based on natural language conversations.

The following image shows the Flow of conversation within DialogFlow

The component we are going to use here are:

User: We, Machines!

Text / Voice : The user interacts with an app like smart watch / google home.

Agents : Help convert user request into actionable data.

Intent: Support or the service that the user wants from the agent. Intent is configured by the developers. Intent determines the action by the code.

Fulfillment: This is a web-hook. This part of the conversation lets you pass on the request from your app to an external source and get response and pass it back to the user. Setting up a web-hook allows you to pass information from a matched intent into a web service and get a result from it.

Note

No programming experience is required as this series is mostly going to be using GUI(Graphical User Interface) of Google’s dialogflow

DialogFlow - Walkthrough

Setting up DialogFlow account

• Go to : https://dialogflow.com/
• Create an account with a gmail account, and “agree” to the terms & conditions.

We are gonna build an agent that helps us with workflow monitor e.g, check if we have tires in the inventory  

Build the Agent

Agents translate user requests to actionable data i.e. intents. It’s essentially a module within dialog flow which incorporates Natural Language Processing to understand what the user meant and to figure out what “action” has to be carried out

Explore the settings page…the ML settings gives us the type of machine learning we want to use for this agent.

In this project we will use Use the Hybrid (Rule-based and ML) mode.

We will define the threshold value for the confidence score at 0.3, In simple terms, if the match is less than 30% percent then the inbuilt “fallback intent” (sorry, I don’t understand etc) will be triggered.

Create The Intent

Think of using Intent as a mapping what a user says and what your software should execute. Intents are configured by developers and used to determine the action taken by the code.

Phrases you can expect from the user that will trigger the intent.

Action & Parameters

The inputs you might need from the user to take an action on the user request. For eg: In tires intent, we might need the Size, etc to finish the action. We have to set entities first for it to be mapped here but go with the flow for now.

Response

The responses of the agent in this project will be sent to the Google Assistant integration.

Entities

Entities are used to extract parameter values from user queries. So when the user says “Book me a flight ” we usually ask for the city, date etc.these are called entities.

here our Entities will be called tire which will be the size of the tire 16, 18 inch etc

Fulfillment

Our Amazon AWS web service will receive a POST request from Dialogflow in the form of the response to a user query matched by intents with webhook enabled.

In the next part we will explain how to configure AWS in order to receive dialogflow POST requests and be able to respond.

AWS - Walkthrough

Create an AWS user and get in the AWS console.

DynamoDB

First let's open DynamoDB console and let's create a db table called "orders" where we will store all the orders requests for certain tire size. We will insert the primary key and the sort key.

Create an AWS IoT Rule

Next, we should open the IoT console and link an MQTT topic to a certain rule which will trigger some actions. So we click on the "Act" tab, and we create a rule.

Set rule query statement

We define the rule query statement which will identify to which mqtt topic we are linked. We want to listen on outOfStock messages and insert in the DB a new order.

Rule options

By creating the rule we can choose which kind of actions the rule can triggers, there are a lot of options.

Configure “insert in DynamoDB” action

We will choose the action to insert the payload of the message received in a DynamoDB table which we already created.

NOTE: In order to automatically insert an item in a dynamoDB table through AWS IOT rule, both primary and sort key need to be defined on the destination table.

Iot Role

For every type of action AWS is asking to assign a particular role to that action in order to give the permission to one service of AWS to access another service. In this case we create a new role which will be allowed to access DynamoDB from AWS IoT.

Create a Lambda function

Now that we created an MQTT topic and an action we need to actually publish to that topic. We will do so by using an AWS Lambda Function. Our function will be called by google assistant, our google action will send us a JSON request asking for a certain tire size.

So next we need to open the Lambda Console and we create a lambda function from scratch, choosing as a language Python.

We need to fill the basic information..

Allow Lambda to access AWS IoT

In order for the lambda function to access DynamoDB or AWS IoT it should has a particular role. We need to go to the IAM Console (Identity Management Console). We will see that a role for our function was already created in the list. We click on that role and we will attach an inline policy by pasting this JSON which will give to our function the permission to publish on an IOT topic.

By refreshing our lambda page we can see that on the "output" side of the function we now have AWS Iot as a possible "output".

Allow Lambda to access DynamoDB

Still on the IAM Console on our function role we will then add the micro-services policy which will give us the DynamoDB access permission.

And again by refreshing the lambda page we see that DynamoDB was enabled

Write the Lambda function

We now start by writing our function. We parse the JSON request looking for the tire size requested, we access the db in the "stock" table (which we created earlier with some items, at the end of the post you can see a snapshot) to see if a tire with that size exist. If it exist we will respond to google with the model name and a picture of it.

Otherwise we will answer to Google Assistant by saying that we couldn't find anything and we will publish an outOfStock event by requesting a certain size and a certain quantity.

Here's the full code:

import json
import boto3
from boto3.dynamodb.conditions import Key, Attr
from botocore.exceptions import ClientError


def lambda_handler(event, context):
   #Parse your JSON message and get out all your attributes
   tire_size = event["queryResult"]["parameters"]["tires"]
   tire_size = tire_size[:2]
   print("Tire size requested:", tire_size)
   #Accessing DB to check for existence
   dynamodb = boto3.client('dynamodb')
   db_response = dynamodb.get_item(TableName='stock', Key={'inches':{'N': str(tire_size)}})
   try:
       #Found an item corresponding to the size requested
       item = db_response['Item']
       print("Found model:", item["model"]["S"])
       model = item["model"]["S"]
       #Preparing the response for Google Dialogflow
       response = { "payload": 
           {
           "google": {
                   "expectUserResponse": "true",
                   "richResponse": {
                       "items": [
                       {
                           "simpleResponse": {
                           "textToSpeech": "Here's what I got Gianluca"
                           }
                       },
                       {
                           "basicCard": {
                               "title": str(tire_size) +"inches tires - "+ model,
                               "image": {
                               "url": "https://di-uploads-pod4.dealerinspire.com/mercedesbenzbrampton/uploads/2017/05/mercedes-benz-tires.jpg",
                               "accessibilityText": "Cool tires"
                               },
                           "imageDisplayOptions": "WHITE"
                           }
                       }
                       ]
                   }
               }
           }
       }
   except KeyError as e:
       #Couldn'find a correct model for the size
       print("Couldn't find anything")
       #Preparing the response for Google Dialogflow
       quantity = 50
       response = { "payload": 
           {
           "google": {
                   "expectUserResponse": "true",
                   "richResponse": {
                       "items": [
                       {
                           "simpleResponse": {
                           "textToSpeech": "The item you requested is out of stock, I placed an order for you"
                           }
                       },
                       {
                           "basicCard": {
                               "title": str(quantity)+ " "+ str(tire_size) +"inches tires were ordered",
                               "image": {
                               "url": "https://di-uploads-pod4.dealerinspire.com/mercedesbenzbrampton/uploads/2017/05/mercedes-benz-tires.jpg",
                               "accessibilityText": "Cool tires"
                               },
                           "imageDisplayOptions": "WHITE"
                           }
                       }
                       ]
                   }
               }
           }
       }
       #notifying the order to aws-iot
       client = boto3.client('iot-data', region_name='us-east-2')
       client.publish(
           topic='iot/outOfStock',
           qos=0,
           payload=json.dumps({"inches":int(tire_size), "quantity":quantity})
       )
   return response 

REST API creation

In our last step we need to give Google Dialogflow a link to which send its requests. Google is able to only send POST requests so we open the API Gateway console of AWS and we create our API, choosing the protocol and a name.

API Resource creation

Then we click on "Actions" and "Create new resource" which we will call "tire"

Create API method

Then again on "Action" we click on create Method and we will select "POST" method, we will then select the integration type which in our case is a Lambda Function. We need to insert our Lambda function name and when we click OK it will ask the confirmation to give the API the access to that function.

Deploy API

In order to make it accessible from Google we need to deploy our API. So click again on Action "Deploy API" and create a new stage name. Now by clicking on the "Stages" section of your API (on the left menu), you can find the stage name you created and by clicking on it you will get the URL to which external services can send their POST requests.

API triggers Lambda Function

By doing these steps we can refresh again our lambda function page and we will see that automatically the API Gateway was added as a trigger for the lambda function.

Subscribe to MQTT & Test

Now we can finally test all of our environment. We can go on the AWS-IOT console in section "Test", subscribe to the topic "outOfStock" where we will publish with the lambda function. We send a google voice command which will ask for a tire size which is not in the DB and here we will see that a message was published asking for an order.

Results

As a result in our DynamoDB "order" table we can see the that the item was inserted.

The Stock Table

This is our "stock" table