The Google Assistant is powerful and can help users in many ways. The most common ways to interact with the Google Assistant are through speech and text; however, not all users are able to interact with the Google Assistant using their voice or typing. Voice and text are also not the best ways to communicate with the Google Assistant in all scenarios.
To help in those situations, we created a device that lets users trigger commands to the Google Assistant pressing a button or using RFID/NFC tags that can be attached to physical objects. To create the device, we leveraged a Google AIY Voice Kit and had some fun hacking the solution.
Learn more about the idea behind the DIVA project in our publication.
At the moment this approach is letting Giovanni, brother of one of us, affected by Down Syndrome, almost blind and not able to speak, gain autonomy on starting entertainment content on his TV and speakers.
We hope it can also be used to help other people with disabilities to gain autonomy on their daily tasks.Solution
The idea is to create a device that:
- Waits for the button to be pressed or an RFID tag to be tapped
- Retrieves the command associated with the button or with the specific RFID tag
- Send the command to the Google Assistant
- Reproduces the answer provided by the Assistant
Have a working Google AIY Voice kit connected to the Internet. Follow the Google AIY Voice kit instructions till the end, when you run the Demo.
Please double check your Google AIY Voice kit is able to run the following script:
This is the script that will create the following configuration files that we will use in the script:
If you can’t run the assistant_grpc_demo.py script or you can’t see the 2 files listed above, please check you completed all steps in the instructions.2. Hardware Components
The project requires few hardware elements:
- Google AIY Voice Kit
- USB RFID reader
- RFID tags
and it is easy to assemble. Just connect the RFID reader reader to the Google AIY Voice Kit via the Raspberry Pi 3’s USB port. When it detects an RFID tag, the RFID reader will send the TAG ID to the kit as a string of text, just like a keyboard.
Next, we’ll create a script that:
- Initializes and starts the Assistant
- Accepts text input as a command for the Assistant (the USB RFID reader sends TAG IDs to the Pi as if they were typed by a USB keyboard)
- Checks if the text is associated with a command configured in the script
- Sends the command to the Google Assistant SDK instance
- Reproduces the answer provided by the Google Assistant SDK instance on the speaker of the device.
Start by opening a new script called 'diva.py’ using your preferred text editor, for example nano:
pi@raspberrypi:~ $ nano diva.py
Copy and paste the script attached 'diva.py' into your file. Then configure the COMMANDS object as a mapping between the ID of the RFID TAG(s) you are planning to use and the corresponding text command you want to send to the Assistant.
To test the script you may try commands like :
- What is the weather like?
- Tell me a joke, please
- Play YouTube video on my TV
From the terminal, we can make the script executable:
pi@raspberrypi:~ $ chmod +x diva.py
And execute the script:
pi@raspberrypi:~ $ ./diva.py device_model_id: DEVICE_MODEL_ID device_id: DEVICE_ID Say "OK, Google" then speak, write a command, press the button or tap a card. Press Ctrl+C to quit...
Feel free to create RFID tags to play your favorite music, control your light, or check your room temperature:
Once you have your tags configured, you can attach them to plushes to start your favorite cartoons, or to your CDs cases to create your own music jukebox.Conclusion
This project was a fun experience and we created something useful for at least one person with disability. We hope you will enjoy this project and that it may help other people with disabilities.
We’d like to continue to iterate on this, and we see several options to improve this project.Some ideas are:
- Use Actions on Google to configure mapping between TAG IDs and command
- Relay on Firebase DB to store configuration information
We would appreciate any feedback or suggestions on what to prioritize, please leave a comment below.