Hardware components | ||||||
 |
| × | 1 | |||
 |
| × | 1 | |||
| × | 1 | ||||
| × | 2 | ||||
| × | 1 | ||||
Software apps and online services | ||||||
 |
| |||||
This project aims to replace the front desk cashier of food trucks and fast food joints with voice-activated vending machine, that can understand the order, checks for allergies, take payment, and deliver orders through vending- machines (soft drinks, water bottle) and through window at counter, it takes help of facial recognition to deliver order to correct person. It also remembers your allergies and your last order.
Steps:1. Speech Recognition
2. Face Recognition
3. NFC card reader
4. Everloop
5. GUI for front desk and kitchen
6. Controlling Motors
1. Speech Recognition:First, we need to make an account in snips.ai .
Creating an app.
- If you are logging into the console for the first time it will ask for the name of assistant and the language. Give the name of your choice and select English.
Creating assistant
- After creating the assistant our first job is to create an app, you can also add apps made by others.
- Go to add app and then create a new app, initially our app will not have any training data hence it will be shown as weak.
Creating new intents:
- In the next step, we are required to create new intents. Go to create new intent. Give it a name and short description.
Creating Intents
- Intents refer to the intention of the user. In our project, we have to recognize the intents like add items, remove items, add allergies, etc.
- Each intent will contain some slots that can be number, time, names, etc, these will appear in the sentences of that intent and will be used in the intent callback codes that we will discuss further.
- Start adding slots by clicking on add new slots, We can use slot types provided by default or add custom slot.
Adding custom slots
- Give a name to your slot type and define the values, for example, dishes name. Make sure to define all the dishes that you want to recognize.
- Add different slots and train your model by adding enough sentences and highlighting the respective slots (most of the time it will be done automatically).
adding training examples
- Similarly, we need to add intents like:-
- addItems : when a person wants to add new items to order.
- removeItems: when a person wants to remove items.
- response: when a person response back a questin in words like no, yes or continue.
- allergies: when a person tells his allergies.
- specialRequest: when a person wants to add special requests like less spicy, sweet, etc.
- suggestion: When a person asks for suggestions while making an order, like a bestseller, today's special, etc. (Currently not implemented).
- Once we have trained our model with enough examples, we are ready to go to the next step i.e. downloading it in the raspberry pi.
Installing SAM CLI tool for offline speech recognition:
Followthis tutorial for installing SAM CLI tool to your raspberry pi, for working with matrix creator.
Installingassistantto raspberry pi :
- After installing SAM successfully, we need to download our assistant to raspberry pi for offline speech recognition (all sam commands will be run from PC terminal, not ssh)
- Connect to raspberry pi by
sam connect <raspi ip address>- login to your snips account.
sam login- Enter your snips.ai account's e-mail and password.
sam install assistant- It will fetch your assistant, if you have more than one it will ask you to select one.
- After installing the assistant you can check the result by saying "Hey snips" and any sentence. You can check the snips output by this command.
sam watch- You can check if every service is running fine by
sam statusAt the time of writing this article the snips audio server and tts were having some issue so I downgraded them.
sudo apt install snips-audio-server=0.63.3
sudo apt install snips-tts=0.63.3Coding Intents:
- It is better to start with a code template, you can download one from here.
- Now we have to code functions for our intent callbacks. For example, when someone will asks to add items to their order, snips will return message with the intent name
addItemsand the slots that we have defined while training our intents and each slots will contain detected words that the user has said. - We will extract this information about items and their quantity from the slots and if they are present in our menu and will give a confirmatory reply to the user.
def addItems(self, intent_message):
for name,values in intent_message.slots.items():
if name == "item":
items = list(map(lambda x: str(x.value), values.all()))
if name == "amount":
amount = list(map(lambda x: int(x.value), values.all()))
try:
if len(items) == len(amount):
add = {}
add = dict(zip(items,amount))
dialogue = ""
for dish,amount in add.items():
if dish in self.order.keys():
self.order[dish] = self.order[dish] + amount
else:
if dish in kiosk.menu.keys():
self.order[dish] = amount
dialogue += str(amount) +" " + str(dish)
dialogue += " is added to your order. "
else:
dialogue = " Sorry, please use numbers for quantity. "
except:
dialogue = " Sorry, I didn't get that. "
self.state = 0;
return dialogue- Similarly, we need to write code for all our callbacks, some syntaxes for sending dialogue to tts are:
hermes.publish_continue_session(intent_message.session_id,"Say this" ,\
["intents of next dialogue"],"")
hermes.publish_end_session(intent_message.session_id,"Say this")
hermes.publish_start_session_notification("site.id","Say this","")- More on managing sessions can be found here.
- First make sure you have open cv4 installed, if not you can get help some here.
- To confirm the installation run python in shell, and import cv2.
import cv2
>>> cv2.__version__
'4.0.0'- We will be doing three things in this script, first, we will detect a human face by Haar Cascade classifier, if the detected face cannot be identified by the LBPH face recognizer using current model, we will capture some images of the face to train our model and give it some unique id and save it to our database.
- Face Recognition script publishes detected face ids to the topic camera/recognisedIds" and subscribes to topic camera/addId in case when a new user gives the order, the "main script" publishes a unique user id to
camera/addIdand the face recognition script trains its model to add this new id. - Here all functions are squeezed in one script for a more detailed explanation please folow this project "Real-Time Face Recognition: An End-to-End Project".
'''
Based on code by Marcelo Rovai - MJRoBot.org and on code by Anirban Kar:
https://github.com/thecodacus/Face-Recognition
'''
def on_connect(client, userdata, flags, rc):
print("[faceRecognition]: Connected")
client.subscribe("camera/addId")
def on_message(client, userdata, msg):
global face_detected
global face_add
global face_id
if str(msg.topic) == "camera/addId":
face_id = str(msg.payload)
face_add = 1
def initMqtt():
client.on_connect = on_connect
client.on_message = on_message
client.connect("localhost", 1883)
client.loop_start()
def getImagesAndLabels(path):
imagePaths = [os.path.join(path,f) for f in os.listdir(path)]
faceSamples=[]
ids = []
for imagePath in imagePaths:
PIL_img = Image.open(imagePath).convert('L') # convert it to grayscale
img_numpy = np.array(PIL_img,'uint8')
id = int(os.path.split(imagePath)[-1].split(".")[1])
faces = detector.detectMultiScale(img_numpy)
for (x,y,w,h) in faces:
faceSamples.append(img_numpy[y:y+h,x:x+w])
ids.append(id)
return faceSamples,ids
def trainer():
path = 'dataset'
recognizer = cv2.face.LBPHFaceRecognizer_create()
print ("\n [FaceRecognition] Training faces. It will take a few seconds. Wait ...")
faces,ids = getImagesAndLabels(path)
recognizer.train(faces, np.array(ids))
# Save the model into trainer/trainer.yml
recognizer.write('trainer/trainer.yml')
def faceAddition(face_id):
face_detector = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
count = 0
while(count<30):
count += 1
ret, img = cam.read()
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
faces = face_detector.detectMultiScale(gray, 1.3, 5)
#cv2.imshow('image', img)
for (x,y,w,h) in faces:
cv2.rectangle(img, (x,y), (x+w,y+h), (255,0,0), 2)
# Save the captured image into the datasets folder
cv2.imwrite("dataset/User." + str(face_id) + '.' + str(count) + ".jpg", gray[y:y+h,x:x+w])
print("[FaceRecognition]: Face capture count = " + str(count))
trainer()
def faceDetection():
global cam
#print('[FaceRecognition]: Face Detection ON')
recognizer = cv2.face.LBPHFaceRecognizer_create()
try:
recognizer.read('trainer/trainer.yml')
except:
return [0]
cascadePath = "haarcascade_frontalface_default.xml"
faceCascade = cv2.CascadeClassifier(cascadePath);
font = cv2.FONT_HERSHEY_SIMPLEX
#iniciate id counter
id = 0
# Define min window size to be recognized as a face
minW = 0.1*cam.get(3)
minH = 0.1*cam.get(4)
counter = 0
recognised_ids = []
while counter<=20:
counter+=1
ret, img =cam.read()
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
faces = faceCascade.detectMultiScale(
gray,
scaleFactor = 1.2,
minNeighbors = 5,
minSize = (int(minW), int(minH)),
)
for(x,y,w,h) in faces:
cv2.rectangle(img, (x,y), (x+w,y+h), (0,255,0), 2)
id, confidence = recognizer.predict(gray[y:y+h,x:x+w])
if (confidence < 100):
pass
else:
id = 0
# storing list of recognised/unrecognised faces
if not id in recognised_ids:
recognised_ids.append(id)
return recognised_ids- First, make sure you have NFC Reader Library for PN512 is installed if not follow this.
- In our project payment is done by reading nfc tags, data.txt file contains the information regarding UID and their respective balance.
- We read nfc data and extract UID in hex format and call function that opens data.txt file and searches for required UID and its balance if the UID has sufficient balance the file is updated with a new balance.
- Payment also gets failed if user is unable to scan card within 60 seconds.
int scan(double amount){
matrix_hal::NFC nfc;
matrix_hal::NFCData nfc_data;
std::cout << "[NFC]: NFC started!" << std::endl;
int sucess = 0;
auto past_time = std::chrono::system_clock::now();
auto current_time = std::chrono::system_clock::now();
std::chrono::duration<double> duration = (current_time-past_time);
while(duration.count()<60){
current_time = std::chrono::system_clock::now();
duration = current_time-past_time;
nfc.Activate();
nfc.ReadInfo(&nfc_data.info);
nfc.Deactivate();
if (nfc_data.info.recently_updated) {
std::cout << "[NFC] : " + nfc_data.info.ToString() << std::endl;
std::string user_id = nfc_data.info.UIDToHex();
sucess = payment(user_id, amount);
break;
}
std::this_thread::sleep_for(std::chrono::microseconds(10000));
}
return sucess;
}- In data.txt file formate for stored cards should be
<UID in HEX> <BALANCE>- NFC program subscribe to the topic
payment/startwhich contains bill amount and publishes to the topicpayment/statuswhich represents whether the payment was successful or not.
- Before we continue, make sure you have MATRIX HAL installed, if not follow this link.
- In everloop we have to set RGB values of each leds in everloop_image object before finally writing it to the bus. for example:
for (matrix_hal::LedValue &led : everloop_image.leds) {
led.red = 0;
// Set green to 100
led.green = 100;
led.blue = 0;
led.white = 0;
}
// Updates the Everloop on the MATRIX device
everloop.Write(&everloop_image);- You also need to make sure we have paho mqtt c installed because we will be subscribing to the topic
"everloop"so that we can change the colors of everloop according to the situation. - The functions that are used for mqtt communication are written in mqtthelper.cpp file and this should be added while compiling the code.
#include <iostream>
#include <string.h>
#include "MQTTClient.h"
#include "mqtthelper.h"
volatile double msg = 0;
MQTTClient_deliveryToken deliveredtoken;
MQTTClient client;
MQTTClient_message pubmsg = MQTTClient_message_initializer;
MQTTClient_deliveryToken token;
int msgarrvd(void *context, char *topicName, int topicLen, MQTTClient_message *message)
{
msg = std::stod((char*)message->payload);
MQTTClient_freeMessage(&message);
MQTTClient_free(topicName);
return 1;
}
void delivered(void *context, MQTTClient_deliveryToken dt)
{
deliveredtoken = dt;
}
void connlost(void *context, char *cause)
{
printf("\n[Everloop]: Connection lost\n");
printf(" cause: %s\n", cause);
}
void initMqtt(char *ADDRESS,char *CLIENTID,char *TOPIC,int QOS){
MQTTClient_connectOptions conn_opts = MQTTClient_connectOptions_initializer;
int rc;
int ch;
MQTTClient_create(&client, ADDRESS, CLIENTID,MQTTCLIENT_PERSISTENCE_NONE, NULL);
conn_opts.keepAliveInterval = 20;
conn_opts.cleansession = 1;
MQTTClient_setCallbacks(client, NULL, connlost, msgarrvd, delivered);
if ((rc = MQTTClient_connect(client, &conn_opts)) != MQTTCLIENT_SUCCESS)
{
printf("[Everloop]: Failed to connect, return code %d\n", rc);
exit(EXIT_FAILURE);
}
std::cout<<"[Everloop]: Connected"<<std::endl;
MQTTClient_subscribe(client, TOPIC, QOS);
}
void publishStatus(char *topic,char *payload){
int rc;
pubmsg.payload = payload;
pubmsg.payloadlen = 1;
pubmsg.qos = 0;
pubmsg.retained = 0;
deliveredtoken = 0;
MQTTClient_publishMessage(client,topic, &pubmsg, &token);
printf("Waiting for publication of %s\n"
"on topic %s \n",
payload, topic);
while(deliveredtoken != token);
printf("[paymentny]Message with delivery\n");
}- Compile file using the command
g++ -o everloop everloop.cpp mqtthelper.cpp -std=c++11 -lmatrix_creator_hal -lpaho-mqtt3c- Run your everloop program and check its working by publishing to topic
everloopby running this python script.
import paho.mqtt.client as mqttClient
import time
def on_connect(client, userdata, flags, rc):
if rc == 0:
print("Connected to broker")
global Connected #Use global variable
Connected = True #Signal connection
else:
print("Connection failed")
Connected = False #global variable for the state of the connection
broker_address= "raspberrypi.local"
port = 1883
user = ""
password = ""
client = mqttClient.Client("Python") #create new instance
client.on_connect= on_connect #attach function to callback
client.connect(broker_address, port=port) #connect to broker
client.loop_start() #start the loop
while Connected != True: #Wait for connection
time.sleep(0.1)
try:
while True:
value = raw_input()
client.publish("everloop",value)
except KeyboardInterrupt:
client.disconnect()
client.loop_stop()everloop
- First, make sure you have PyQt installed if not run this command
sudo apt-get install python-qt4- In front GUI we display the Menu and the current order, the script gets information from the topic
guiFront/textthat is published by "main script".
front GUI
- The GUI for the kitchen contains orders that are yet to be served. Once the order is ready, and the "completed" button beneath the order is pressed it publishes orderid to the topic
guiBack/completedOrderwhich is subscribed by "main script" that then calls the user and deliver the order after recognizing them through facial recognition.
GUI Kitchen
- The ideal delivery mechanism will ensure that every order reaches its correct customer. The main problem arises when people don't reach the counter in the same order as they were called, catering to this kind of problem is out of the scope of this project.
- Currently, motor.cpp will run two motors whenever the user called to collect order is recognized by the camera. One motor delivers bottles from the vending machine and the other motor opens the window to collect the order from the counter.
Thanks to MJRoBot.
Comments