Imagine you can literally ask your smart fridge to do grocery shopping for you
We wanted our application to support the basic online shopping process. It should retrieve information about the products from an API (for our demo, we used Amazon.com) to display it on the screen, enabling the user to browse the store and search for specific products using voice commands. The app should allow adding products to the cart or removing them when needed, just as it is with the conventional online shop. But, in our case, no touch interactions required. The assistant would recognize human speech and respond back, displaying lists of the requested products, checking on the number of the items needed and adding them to the cart. The user then would be able to make a purchase and complete the checkout right away.
What we’ve learned developing for Tizen and where are the pitfalls for you to watch
This is the list of the tools that we chose as a technology stack:
· Tizen platform-specific .NET API, exposing Tizen features
· Xamarin.Forms as a UI framework
· Microsoft.Bot framework that powers our bot for recognizing text with the help LUIS model and returning product related data, such as category, description, price, quantity etc.
· Microsoft LUIS (Language Understanding Intelligent Service) to power language recognition and allow our bot to respond with a relevant action.
As a first step, we needed to set up an installation environment, which included Xamarin, Visual Studio Tools for Tizen preview, Tizen emulator and Tizen MI-Based Debugger. Concerning the emulator, it has a specific hardware requirements that you will have to meet. Make sure you have at least a 2nd generation Intel HD Graphics processor or a discrete video card. In our case, we used GeForce 8400GS.
Another pitfall happened when we tried to run our app in the debug mode and failed to do this. It turns out that this mode will be fully implemented only with Tizen 4.0 beta. So, we had to create a custom logger to trace any code bugs.
And one more quite important issue we faced with Tizen is that the project couldn’t detect NuGet packages in the runtime mode. So, we had to add them to the Tizen lib folder manually. Only after all these manipulations were completed could we start developing our app.
By default, Tizen uses native SpeechToText library to transform speech into text (for mobile apps). In our case, to provide language recognition, we used a custom LUIS model. Generally, the process looks this way: the assistant interprets the user’s text and transfers it to LUIS, so the model could analyze it and trigger the bot to perform a certain action. Using the Microsoft.Bot framework, we enabled our bot to interact with LUIS, recognizing texts and returning various sets of data, such as the list of the products in a specific category, a certain product from this list or a quantity of the products that were requested by the user.
To be able to emulate an online shopping experience, we needed to create a base of goods and build a user interface replicating the functionality of an online shop. First of all, we made up some fake data for the products that we wanted to have in-stock and developed a simple store UI.
As our next step, we had to implement the services for transferring speech into text and vice versa. With our first demo, we used Google Speech API and Microsoft Cognitive Services. But later on, in January 2017, the second Tizen preview was released providing additional API packages. So, we could use the native Tizen SpeechToText and TextToSpeech libraries instead of online the services mentioned above, which improved the performance of our application drastically.
Then we needed to enable our bot to parse request messages and return products listed by a certain category, a count of products or any specific item selected by the user.
It’s worth mentioning the Prism Framework that we also used while building our app. We opted for Prism since it allows building maintainable and testable XAML applications for Xamarin Forms. The framework provides an implementation for a set of design patterns that help creating well-structured XAML applications, including MVVM, dependency injection, commands, EventAggregator and others. We also used the Unity container to support the dependency injection.
Wrapping up the experiment – voice-controlled shopping assistant
Interested in how the things looked by the end of our experiment? Here is a video to show you how our bot actually works.
If you want more details, read the full story on https://goo.gl/t0hFrg. And of course, your comments and questions are welcome.
Comments