Voice Assistant System for Senior

INTRODUCTION

Motivated about finding ways to enhance the quality of life and mobility with an innovative and complete product design, I have developed a Voice Controlled System which is integrated by the following three technologies:

A. HUMIDIFIER: System to turn on/off a humidifier and developed with Snips and the Matrix Labs board for healthcare of seniors.

B. FETCH A CANE: System to fetch a cane and developed with Android and the ESP-WROOM-32 board for Alzheimer's issues.

C. SMART HOME: System for remote controlled home devices and developed with Android and the SONY Spresense board.

A schematic diagram of this is shown below.

A. HUMIDIFIER

(Timing: 20 hrs)

Humidifiers have many health benefits. They can prevent your skin from drying out and help with respiratory problems caused by dry air. Many models are difficult to clean and often require carrying a heavy tank full of water from the sink to the humidifier. This is something that an elderly person could not do. Health benefits: humidifiers are very useful in the winter. The moisture they produce can lessen cold symptoms, let you breathe easier, lower your risk of infection, and prevent your skin and lips from cracking due to dryness.

Humidifiers can cause burns if people get too close. That’s why most consumer advocacy groups recommend cool-mist models. Ultrasonic humidifiers use the same principle as ultrasonic wave nebulizers, and has an electronic oscillator generate a high frequency ultrasonic wave (1.7 MHz approx), which causes the mechanical vibration of a piezoelectric element. This vibrating element is in contact with a liquid reservoir and its high frequency vibration is sufficient to produce a vapor mist. Another advantage is that the ultrasonic vibration is almost silent.

Hardware Prerequisites

Before getting started, you can check with more details what hardware I used:

• Raspberry Pi 3 B+
• MATRIX Creator and datasheet here
• Micro-USB power adapter
• Micro SD Card (16 GB & class 10 speed)
• Micro-USB Cable
• USB Keyboard & Mouse & HDMI Monitor
• Internet connection
• Grove Water Atomization

Software Prerequisites

You can check what required software on my personal computer I used:

• SD Card Formatter: To easily format our Micro SD Card
• balenaEtcher: To easily flash our compressed Raspbian Stretch image
• MATRIX kit image here which includes Stretch + MATRIX packages (2.18 GB)
• Node.js: Dependancy for Sam CLI Tool (Windows 64 bits)
• Git Bash: requirement asked by Node.js
• Snips' Sam CLI Tool: Creates & manages Snip assistants on your Raspberry Pi

Let's Get Started

• First at all, You need a registered snips.ai account.
• Be sure to setup your Raspberry Pi with your MATRIX Creator device.
• Enable SSH on your Raspberry Pi. This allows the SAM CLI tool to connect to your Raspberry Pi board
• Install MATRIX Lite JS. This allows Snips to work on you Raspberry Pi board
• Configure Snips.ai

Once everything is well installed on your Raspberry Pi board, You can check the versions of: npm, node and nodejs.

Connect Sam To Your Raspberry Pi

On the command prompt of you Raspberry Pi board type ifconfig to know the IP address of this device.

$ ifconfig

My IP address is 192.168.0.8. From your computer's terminal, you can connect to the Raspberry Pi board. Also you'll need your username and password

sam connect 192.168.0.8

Sign in through the Sam CLI Tool

Creating a Snips Assistant and App

Sign into your Snips.ai account, and create and assistant called humidifier and App named Humidifier. .

Adding an Intent

Intents are what Snips uses to handle user requests. For this Assistant, we'll start by creating an intent for turning On/Off the MATRIX Creator's App.

Create a new intent HumidifierState for your Humidifier app. Once you've defined your intent, you need to extract whether the user wants their humidifier on or off. This is where slots come in. Create a new slot called power and a custom slot type with the same name.

Make sure your power slot type has on & off as values. Finally, create example sentences containing the words on and off. Highlight these words in your example sentences to assign them to your recently created power slot.

Deploy Your Assistant

Be sure to save! You can now deploy your assistant! On the bottom right of the page will be a Deploy Assistant button. Use the Sam CLI Tool to deploy the assistant to your Raspberry Pi. Below is an example of the command to use.

Software

This step will go show you how to setup a MATRIX Lite project with Snips. You can download next three files on the "Code" section and then copy and paste this files into the /home/pi/lite_js folder of your Raspberry Pi board.

assistant.js will be used to listen and respond to events from your Snips assistant. This file combines all the code from everloop.js & relay.js to control the everloop and relay switch through voice.

/////////////
//VARIABLES//
/////////////
var mqtt = require('mqtt');
var client  = mqtt.connect('mqtt://localhost', { port: 1883 });
var relay = require('./relay.js');
var everloop = require('./everloop.js');
var snipsUserName = 'guillengap';
var wakeword = 'hermes/hotword/default/detected';
var sessionEnd = 'hermes/dialogueManager/sessionEnded';
var lightState = 'hermes/intent/'+snipsUserName+':HumidifierState';
//////////////
//ON CONNECT//
//////////////
client.on('connect', function() {
console.log('Connected to Snips MQTT server\n');
client.subscribe('hermes/hotword/default/detected');
client.subscribe('hermes/dialogueManager/sessionEnded');
client.subscribe(lightState);
});
//////////////
//ON MESSAGE//
//////////////
client.on('message', function(topic,message) {
var message = JSON.parse(message);
switch(topic) {
// * On Wakeword
case wakeword:
everloop.startWaiting();
console.log('Wakeword Detected');
break;
// * Robotic Arm Change
case lightState:
// Turn robotic arm Here/There
try{
if (message.slots[0].rawValue === 'on'){
relay.humidifierOn();
everloop.stopWaiting();
console.log('Humidifier is On');
}
else if(message.slots[0].rawValue === 'off'){
relay.humidifierOff();
everloop.stopWaiting();
console.log('Humidifier is Off');
}
}
// Expect error if `here` or `there` is not heard
catch(e){
console.log('Did not receive an On/Off/Read state')
}
break;
// * On Conversation End
case sessionEnd:
everloop.stopWaiting();
console.log('Session Ended\n');
break;
}
});

everloop.js incorporates the code required to control the LEDs on the MATRIX Creator.

/////////////
//VARIABLES//
/////////////
const matrix = require('@matrix-io/matrix-lite');
var matrix_device_leds = 0;// Holds amount of LEDs on MATRIX device
var methods = {};// Declaration of method controls at the end
var waitingToggle = false;
var counter = 0;
setInterval(function(){
// Turns off all LEDs
if (waitingToggle == false) {
// Set individual LED value
matrix.led.set({});
}
// Creates pulsing LED effect
else if (waitingToggle == true) {
// Set individual LED value
matrix.led.set({
b: (Math.round((Math.sin(counter) + 1) * 100) + 10),// Math used to make pulsing effect
});
};
counter = counter + 0.2;
// Store the Everloop image in MATRIX configuration
},50);
///////////////////
//WAITING METHODS//
///////////////////
methods.startWaiting = function() {
waitingToggle = true;
};
methods.stopWaiting = function() {
waitingToggle = false;
};
module.exports = methods;// Export methods in order to make them avaialble to other files

relay.js includes the code required to turn the humidifier on and off. This relay will control power to the humidifier.

/////////////
//VARIABLES//
/////////////
const matrix = require('@matrix-io/matrix-lite');
var methods = {};// Declaration of method controls at the end
var relayPin = 0;// The GPIO pin connected to your relay
matrix.gpio.setFunction(0, "DIGITAL");
matrix.gpio.setMode(0,"output");
////////////////////////////////////OFF METHOD
methods.humidifierOff = function() {
matrix.gpio.setDigital(0,"OFF");
console.log("Humidfifer Have Been Turned Off");
};
////////////////////////////////////ON METHOD
methods.humidifierOn = function() {
matrix.gpio.setDigital(0,"ON");
console.log("Humidifier Have Been Turned On");
};
module.exports = methods;// Export methods in order to make them avaialble to other files

Hardware

Below you can see the electrical diagram with the connections of this project, it's recommended to do it patiently so as not to make mistakes.

To measure the relative humidity I made a humidifier meter with my Mega 2560 board, a DHT-22 humidity sensor and a 20x4 LCD display. Then I show you the electrical diagram.

The MATRIX Creator board also has a humidity sensor but it seemed to me that the DHT-22 humidity sensor is more accurate. Below I show you the code: humidity_meter.ino

//Libraries
#include <DHT.h>;
#include <Wire.h> // Library for I2C communication
#include <LiquidCrystal_I2C.h> // Library for LCD
//Constants
#define DHTPIN 7     // what pin we're connected to
#define DHTTYPE DHT22   // DHT 22  (AM2302)
DHT dht(DHTPIN, DHTTYPE); //// Initialize DHT sensor for normal 16mhz Arduino
//Variables
int chk;
float hum;  //Stores humidity value
float temp; //Stores temperature value
// Connect to LCD via I2C, default address 0x27 (A0-A2 not jumpered)
LiquidCrystal_I2C lcd = LiquidCrystal_I2C(0x27, 20, 4); // Change to (0x27,16,2) for 16x2 LCD.
void setup()
{
Serial.begin(9600);
dht.begin();
// Initiate the LCD:
lcd.init();
lcd.backlight();
}
void loop()
{
//delay(1000);
//Read data and store it to variables hum and temp
hum = dht.readHumidity();
temp= dht.readTemperature();
//Print temp and humidity values to serial monitor
Serial.print("Humidity: ");
Serial.print(hum);
Serial.print(" %, Temp: ");
Serial.print(temp);
Serial.println(" Celsius");
// Print 'HUMIDIFIER' on the first line of the LCD:
lcd.setCursor(5, 0); // Set the cursor on the fourth column and first row.
lcd.print("HUMIDIFIER"); // Print the string "HUMIDIFIER"
lcd.setCursor(0, 1); //Set the cursor on the first column and the second row.
lcd.print("********************");
lcd.setCursor(0, 2); //Set the cursor on the first column and the second row.
lcd.print("Humidity:    ");
lcd.print(hum);
lcd.print(" %");
lcd.setCursor(0, 3); //Set the cursor on the first column and the third row.
lcd.print("Temperature: ");
lcd.print(temp);
lcd.print(" C");
delay(500); //Delay 1 sec.
}

Printing and Assembling the Humidifier's box

For the "Grove Water Atomization" device I have developed a 5x8x2.5 cm box, which has the following functions of protect this device, and give a correct location so that the piezoelectric element is easily placed on the recipient with the water. You can download STL and GCODE files on the "Custom parts and enclosures" section.

Test

Before we start to do tests we need to do next actions: On your Raspberry Pi type: node assistant.js and on your PC type sam status and sam warch

Now we cant test our humidifier

Conclusion

This humidifier is a good prototype for the seniors healthcare, and we can add and activate more medical devices such as an electrocardiogram. Thi device doesn't weigh much, and the elderly people doesn't have to move to turn it on and off.They just have to make use of their voice and once programmed, you no longer need an internet connection to use it.

B. FETCH A CANE

(Timing: 13 hrs)

Alzheimer's disease, is a chronic neurodegenerative disease that usually starts slowly and gradually worsens over time. The most common early symptom is difficulty in remembering recent events. As the disease advances, symptoms can include problems with language, disorientation , mood swings, loss of motivation, not managing self-care, and behavioural issues.

Everyone knows that when senior people want to look for important things, they forget where they left them. In our system we want to solve this problem in a simple way and for this reason we have based on using a Smartphone and an ESP-WROOM-32 board. This board has the advantage of being a small device, and that includes communication via Bluetooth low energy. More information on how to install the library here.

Hardware

In the figure below I show you the schematic diagram of my solution:

How does it work?

1. The android application has two application buttons: "Walking Stick" and "Speech Recognizer"

2. When we press the "Walking Stick" button then we send the "A" character, and when we press the same button for the second time, then it sends the "B" character.

3. The information travels via Bluetooth and reaches ESP32, when it receives the "A" character then the LED turns on and the buzzer is activated. When character "B" arrives, then the LED turns off and the buzzer is deactivated.

4. When we press the "Speech Recognizer" button, then we have the opportunity to use voice commands to send the characters "A" and "B" as in steps 2 and 3.

5. The voice command used to activate the cane finder is: "Walking stick on"

6. The voice command used to deactivate the cane finder is: "Walking stick off"

Software

Then I show you the most important points of the android application that I developed with MIT AppInventor 2:

The official App Inventor site is: https://appinventor.mit.edu/ and this is the design of my application called: "cane"

The code of the "Walking Stick" button is:

The code of the "Speech Recognizer" button is:

To program the ESP-WROOM-32 board we have used Arduino IDE. The code is shown below and the file "cane.ino" you can get on the download section.

class MyCallbacks: public BLECharacteristicCallbacks {
void onWrite(BLECharacteristic *pCharacteristic) {
std::string rxValue = pCharacteristic->getValue();
if (rxValue.length() > 0) {
Serial.println("*********");
Serial.print("Received Value: ");
for (int i = 0; i < rxValue.length(); i++) {
Serial.print(rxValue[i]);
}
Serial.println();
// Do stuff based on the command received from the app
if (rxValue.find("A") != -1) {
Serial.print("Turning ON!");
digitalWrite(LED, HIGH);
digitalWrite(ledPin, HIGH);
}
else if (rxValue.find("B") != -1) {
Serial.print("Turning OFF!");
digitalWrite(LED, LOW);
digitalWrite(ledPin, LOW);
}
Serial.println();
Serial.println("*********");
}
}
};

Test

The box used in this device is the same as in section A, and you can download it at "Custom parts en enclosures" section. Below, we can see the assembling of this device:

In the video below, I show you the tests I did with this system, both with the activation button and the voice command.

Conclusion:

This design is practical, economical and functional. This device only can be used in the house or department where the cane is located and the design could be made smaller with a small 5V battery. Now every time a senior with Alzheimer's issues wants to locate his cane he only has to use this application and his voice to find it.

C. SMART HOME

(Timing: 15 hrs)

The term "smart home" is used to describe a house that contains a communication network that connects different appliances and allows them to be remotely controlled, monitored and accessed. This is a help system to open and close doors, windows and other devices through the use of voice. In this project we´re going to use the SONY Spresense board. The connection is via wireless via a Bluetooth HC-05 device. Its applications can be to remotely control some devices for some senior people who have the inability to move easily to activate these devices.

Hardware

This project has the goal of developing a system in which we can support people with disabilities and who are affected by mobility, as is the case of people in wheelchairs or elderly people, who have difficulty opening a door, light a lamp or activate an alarm. Below we can see the schematic diagram and you can get the electrical diagram in the Download section.

How does it work?

1. First we have to install the "SONY Spresense" library here.

2. When the board is feeding with 5V and is turning on, then we activate the Bluetooth of our Smartphone, and immediately turn on the "Voice Assistant" application. In this application, we press the "Select Bluetooth" button. We look for our device and finally press the "Connect" button. The device will inform us when our "android" application is linked to Bluetooth.

3. When we press the "Light" button, we can turn the lamp on or off. Our application sends the "a" character when we turn on the lamp. When the lamp is off, we are sending the character "b".

4. The "a" character arrives on our "SONY Spresense" board, and through the "D03" port, it reaches the L293B driver, and the 5 Watt lamp is turned on. The "b" character will turn off our lamp.

5. When we press the door button, we can open or close the door. Our application sends the character "c" when we want to open the door. When the door is going to be closed, we are sending the character "d".

6. The "c" character arrives on our "SONY Spresense" board, and through the "D04" and "D05" ports, arrives at the L293B driver, a 5-volt motor is activated, which turns in one direction and open the door. The character "d" close our door.

7. When we press the "alarm" button, we can activate a security alarm. Our application sends the "e" character when we want to activate the alarm. When the alarm is going to be deactivated, we are sending the character "f".

8. The "e" character arrives on our "SONY Spresense" board, and through the "D06" port, arrives at the L293B driver, the PIR motion sensor is activated, which emits a buzz whenever it detects an infrared color . The "f" character will deactivate this sensor.

9. We use the transistor BC547, to provide enough power to the buzzer, since the pulse that comes out of the "SONY Spresense" board has low current. Also at the output of the PIR sensor we have very little power.

10. To use the "Voice Assistant", we only have to press the "Speech Recognizer" button. But before we have to activate the wireless connection of our Smartphone, in my case, I made use of the WiFi connection of my wireless modem, but you can use the telephone network that has an internet connection. The operation is the same as we have described so far, and the voice commands are the following:

11. For example, when we say the command: "Light", the "Google Speech Recognizer" system records our voice, encodes it and sends it to the remote server. Process the information and return the text of our audio. When this text is equal to one of our voice commands that we have programmed, then proceed to send the character that corresponds to the order that we gave. In this example, the "a" character will be sent.

Software

I show you the most important points of the android application that I developed with MIT AppInventor 2:

For example, the code of the "Light" button is:

And continuing with the same example, the "Speech Recognizer" code is:

To program the SONY Spresense board we have used Arduino IDE. The code is shown below and the file "smart_home.ino" you can get on the download section.

int LIGHT=3, DOOR_OPEN=4, DOOR_CLOSE=5, ALARM=6;
void setup()
{
Serial.begin(9600);
Serial2.begin(9600);
pinMode(LIGHT, OUTPUT); //LIGHT
pinMode(DOOR_OPEN, OUTPUT); //DOOR
pinMode(DOOR_CLOSE, OUTPUT); //DOOR
pinMode(ALARM, OUTPUT); //ALARM
digitalWrite(LIGHT,LOW);
digitalWrite(DOOR_OPEN,LOW);
digitalWrite(DOOR_CLOSE,LOW);
digitalWrite(ALARM,LOW);
}
void loop()
{
while (Serial2.available())
{
char dato= Serial2.read();
switch(dato)
{
//Case a - Turn on Light
case 'a':
{
digitalWrite(LIGHT,HIGH);
Serial.println("Light is on");
delay(10);
break;
}
//Case b - Turn off Light
case 'b':
{
digitalWrite(LIGHT,LOW);
Serial.println("Light is off");
delay(10);
break;
}
//Case c - Open the door
case 'c':
{
digitalWrite(DOOR_OPEN,HIGH);
digitalWrite(DOOR_CLOSE,LOW);
Serial.println("Open the door");
delay(750);
digitalWrite(DOOR_OPEN,LOW);
break;
}
//Case d - Close the door
case 'd':
{
digitalWrite(DOOR_OPEN,LOW);
digitalWrite(DOOR_CLOSE,HIGH);
Serial.println("Close the door");
delay(750);
digitalWrite(DOOR_CLOSE,LOW);
break;
}
//Case e - Alarm on
case 'e':
{
digitalWrite(ALARM,HIGH);
Serial.println("Alarm is on");
delay(10);
break;
}
//Case f - Alarm off
case 'f':
{
digitalWrite(ALARM,LOW);
Serial.println("Alarm  is off");
delay(10);
break;
}
default:
for (int thisPin = 2; thisPin < 7; thisPin++) {
digitalWrite(thisPin, LOW);
}
delay(50);
}
}
}

Test

In the video below, I show you the tests I did with this system, both with the activation button and the voice command.

Conclusion

This system proved to be very efficient and practical. Now we can help senior people to facilitate their movility. For example, if a person with a disability is in a wheelchair and is on a second floor and has to open the door of his residence, then he only has to press a button to open or close his door.