Simple Home UI for the Temporary

Introduction

When we started Casa Jasmina project, we had clear in mind the question “What is the meaning of home?”, question made more meaningful by the huge changes the digital age was doing at the private, secret place where our life be.

In this scenario, the rise (or coming back?) of conversational UIs prepared the field for interesting voice interaction, setting our eyes free from the screen. Among voice platforms Snips offers a powerful alternative to the main Google/Amazon mainstream offer - we like this because it’s more transparent and works offline, in a fog computing environment.

Casa Jasmina is / became a temporary Home, an exhibition space lived by its temporary inhabitants, hence we worked on developing solutions for this specific target group. The temporary home is indeed the new hotel, and we worked on this with the short stay or even the needs for elder people in mind. Special tools for Special Needs.

In a very naive way we tried to create a simple layer flatting the possible requests a guest may have towards a home environment. This implies technology driven solutions as well, but this first obliged us to normalize where things were stored, kept organized. We wanted to create a Universal Multilanguage Home Labelling System. We began by defining an English / Chinese / Italian vocabulary - please contribute by adding you language. We later prepared signs that were put on furnitures using magnets (you can do your own) and we want to streamline the language selector allowing a host to choose among the most relevant languages in his city / home (Berlin hosts may find italian inappropriate, and would add French and Spanish, etc…)

Node-Red, Open Hardware and Snips: Infinite opportunities

The open source Ecosystem offers so many challenging and powerful opportunities for the Home environment. Home Assistant,OpenHab and many others software have widening and growing communities. As integrators of different solutions, bridging industry and DIY standards we chose to implement our home automations in Node-Red.

Prerequisites

In order to accomplish this project you will need:

• Snips.ai device and account
• Node-RED instance

Node-RED is mandatory since the app is using it as a REST API for the database configuration and automation. You can install both on the same device, but if you are using Node-RED for other flows too, I suggest you to keep them separated.

Basic Installation

In the repository, you will find node-red-flow.json. This file is a basic template for a first setup of the database. Inside you will find the database http request flow and some injects to easily change the database content to your own needs (e.g. changing the user name or the answer to a category).

You can check if the http call is working by copying the url of your Node-RED instance and adding /simpleuidb in the search bar of your browser, it should give you a json object with everything you set up.

Now that the Node-RED side is complete and working, we can switch to the Snips.ai side.

Open your web browser and connect to http://console.snips.ai with your credentials, here you will find all the assistants you’ve created so far. Our app is available only for english language at the time of writing, so you will need an english assistant to use it.

Open your assistant and you’ll see your installed apps; to install a new one, open the app store and look for Simple UI or follow this link. Install the app and then deploy your newly trained assistant by clicking on deploy.

Paste the command to your computer terminal and let the SAM utility do its job. After a while, the installation procedure will ask you to enter your database url: copy and paste the url you used to check if the database was working (http://<yournoderedaddress>:1880/simpleuidb).Now the system is configured and Snips will answer to “Hey Snips, help me!”.

Advanced Use and Configuration

The App supports a deep variety of user integration and customizing. By forking it on the console, you get access to all the intents we have configured and you can modify them to your own needs.

Simple UI supports shortcuts. You don’t have to say “Hey Snips, help me” everytime, instead use the keyword of the category to select what you want (e.g. “Hey Snips, transportation” gives you transportation info). You can add any kind of shortcut by adding examples to intents.

You can add an action to a specific category by defining an url field in the category.

"transportation":{
   "categories":null 
   "transportation":"Check the dashboard to find the tourist guide, we                             are close to taxis and underground station",
   "url":"http://192.168.0.115:1880/callback" //add this field to add actions
}

This way, when Snips detects the intent, it will make a http get request to the address you specified.

Adding a new category is straightforward. In the database, add your new category to the categories array and also add a category object explicitating if there are subcategories and answers.

{
...
"categories":["food","sleeping","bathroom","cleaning","temperature","emergency","transportation","NEW CATEGORY"],
...
"NEW CATEGORY":{"categories":null,
                "NEW CATEGORY":"SNIPS ANSWER"
                }
...
}

Then in the intent console, add examples in the WhereIs intent matching the category name you’ve added. This way Snips will recognize the category and the app will route it correctly.

Adding a category with subcategories needs creating a new intent and editing the action script. In the Simple UI app, add the intent for your category, where you put subcategories as examples. The parameter field has to be called “category”.

In the script use the template callback (food_callback) and change the parameters according to your configuration.

   def NEWCATEGORY_callback(self, hermes, intent_message):
       global databaseurl
       #get categories from database
       good_category = requests.get(
           databaseurl).json().get("NEW CATEGORY").get("categories")
       category = None
       if intent_message.slots: #check is user request match subcategory
           category = intent_message.slots.category.first().value
           if category.encode("utf-8") not in good_category:
               category = None
       if category is None:
           Answer = "Sorry I didn't understand. Say " + \
               ", ".join(good_category) + " to get help."
           hermes.publish_continue_session(
               intent_message.session_id, Answer, ["casajasmina:INTENTNAME"])
       else:
           #get the answer from the database
           Answer = str(requests.get(
               databaseurl).json().get("NEW CATEGORY").get(category))
           hermes.publish_end_session(intent_message.session_id, Answer)

Then add it to the master_intent_callback elif condition matching your intent name and to the whereIs_callback. In this way the assistant will be able to route your requests to the correct flow.

#generic callback for every category in the database, handles single and multi category topics
   def whereIs_callback(self, hermes, intent_message):
.....
               #multi category handling
               Answer = "You asked for " + category + \
                   ", I can give you tips on " + ", ".join(subcategory)
               #add new intent here...
               hermes.publish_continue_session(intent_message.session_id, Answer, ["casajasmina:Food", "casajasmina:emergency","casajasmina:INTENTNAME]) 
.....
def master_intent_callback(self, hermes, intent_message):
       coming_intent = intent_message.intent.intent_name
       if coming_intent == 'casajasmina:WhereIs':
           self.whereIs_callback(hermes, intent_message)
       elif coming_intent == 'casajasmina:ConnectMe':
           self.askHelp_callback(hermes, intent_message)
       elif coming_intent == 'casajasmina:Food':
           self.food_callback(hermes, intent_message)
       elif coming_intent == 'casajasmina:emergency':
           self.emergency_callback(hermes, intent_message)
       elif coming_intent == 'casajasmina:INTENTNAME':
           self.NEWCATEGORY_callback(hermes, intent_message)

Conclusions

This project is under an ongoing development. We are adding new features and making the assistant feel less robotic and cumbersome, polishing its structure and interactions. The next milestone is to release multilanguage support (initially Italian) and streamline the integration between the labelling system and the Snips behaviour.