AutoPing

This is AutoPing. We have been using it in our restaurant and I would say it can accurately detect "busy" around 90% of the time.

Introduction: How did we get here?

Working at my family's take-out restaurant, business can be unpredictable. Some days, there would be literally no one while other days the restaurant would be jam packed. As a student in high school essentially working part-time there, I have to strike a balance between my academics and work, and herein lies the problem.

There were many times when I would be at home working on my homework, completely unaware of the status of the restaurant until I receive a phone call, with my mom on the other end of the line yelling at me to come help, with the phone ringing in the background.

Other times, I would be in the restaurant substituting for my parents while they were out replenishing supplies when business would suddenly peak. During those times, it would be extremely inconvenient to grab my phone with my greasy hands just to notify my parents of the situation while simultaneously picking up an order.

On the other hand, there were times where we would expect it to be busy, however business would be extremely slow. During these times, I would not be able to leave because I simply didn't know if business would randomly pick up later, costing precious time for me to do my homework.

This problem obviously stems from personal experience, but this is a complex problem for many other mom-and-pop small businesses across the country, specially the ones with first-generation children.

The Solution

I want to build a device that would constantly listen to its surroundings and predict customer traffic. A general device that can pick up all the cues in a bustling business environment with pinpoint accuracy is beyond the scope of a project like this, so in an effort to bring the scope of the project down to the realm of possibility and build a working minimum viable product that is deployable on an edge device, it is necessary to pre-define a metric to determining how short-staffed a business is.

Metric

There are only two ways to place an order at our restaurant: ordering in-person or ordering via the phone, with the vast majority of customers calling in to place an order. Depending on how many people are working at the moment, picking up a phone call sometimes meant dropping whatever they were doing to pick up the phone, ruining the efficiency of the restaurant.

Thus, as the order count increases and we become more and more short-staffed, the time that the phone will ring for will get longer and longer as we become more and more reluctant to pick up the phone. How long the phone rings for before it is picked up will be the metric used to determine how busy the restaurant is.

Our main goal: Build a device that would actively listen to the restaurant environment. If it detects that the phone is ringing slightly longer than usual, and send a message notifying backup employees on standby (like me) to help.

The AutoPing System

AutoPing is the name of the entire notification system and is made up of two parts: the EdgePing hardware, and the CompanionGUI software.

EdgePing

The EdgePing is developed around the Infineon PSoC 6 AI Evaluation Kit, which is perfect for this application due to its small size, high processing power, and ample onboard sensors, which greatly simplified hardware and firmware development.

CompanionGUI

The CompanionGUI runs on a standard laptop and connects with the EdgePing device. Since the EdgePing device has no native way of connecting to the internet and sending messages, it requires the assistance of an external device. The CompanionGUI is a custom app developed using PyQt which takes care of sending the text message to employees notifying them to come help and wraps everything up in an easy and intuitive user interface for interfacing with EdgePing.

EdgePing Development

Step 1: Update PSoC Firmware

When I first received my PSoC6, the firmware was outdated so the first thing to do was update it. To do so, I firstly had to go to this website: https://developer.imagimob.com/getting-started/infineon-ai-evaluation-kit

From there, I had to download both ModusToolbox Programmer and Streaming Firmware HEX File and followed the instructions found on the website to complete the firmware update.

Step 2: Collecting Ring Data on the PSoC6

To gain sufficient data, I need to train the PSoC on the phone's ringing to make it understand the sound. I simply set the PSoC aside and had it record the restaurant's noise environment. Unfortunately, DeepCraft only allowed me to record in 5 minute intervals which meant every 5 minutes, I had to go back and create a new recording file.

Now that I got all my required data, I must create a Classification Project to sort out all of my recorded audio. However when I uploaded my data files, there was a yellow warning sign under Status.

It turns out I was supposed to label my audio while it was recording. To fix this issue, I had to add some Predefined Labels like “Ring” and "No Ring" before going back to my original file and highlighting the audio track with the labels.

Designing the PSoC's Housing

Step 3: Building the PSoC’s housing.

I decided to make a simple housing with 4 holes. 2 would be for the red and green LEDs and the other 2 would be for the microphone. I want the green led to always be one and when the phase is detected, the red led would blink, letting the person know that the PSoC is going to send out the message before having it turn solid red, telling the person that a call has been sent out. With this idea in mind, I designed a housing in CAD.

Step 4: Wiring up everything

This was the most simple step so far. I hooked the green and red LED to pin 3 and 0 respectively, with a 120 ohm resistor in between while the negative of both LEDs went to the GND of the PSoC.

Step 5: Blinking the LEDs

After installing everything, I needed to make sure that the PSoC survived the surgery, so I wrote a small piece of code to flash the LEDs on and off, which can be found on the Github in EdgePing > Blinking_LED.zip

Unzip the file and open it up in ModusToolBox. In Quick Panel under the Launch tab, click Blinking_LED Program (KitProg3_MiniProg4).

Step 6: Training the Model

To train the model, I must assign each of my audio files between Training, Validation, and Testing. I decided to keep 80% for training, 10% for Validation, and 10% for Testing. This training part took the longest but once it was done, it showed me the best model to use.

Step 7: Deploying the ML Model on EdgePing

From there, it generated the model.c and model.h file in which I inserted into my code and I tested it. I downloaded it and generated code that could be used in ModusToolBox. All the files for EdgePing including the model I used are found in the EdgePing folder in the Github repo. This worked surprisingly well and I realized that this was because the phone ring is very consistent every time, making it easier for the ML model to recognize.

The code for EdgePing can be found in EdgePing > Detector.zip. Unzip the file and open it in ModusToolBox. In Quick Panel under Launches, launch the program like how Blinking_LED was launched.

Once EdgePing recognizes that it is getting busy, it will send a quick message via the debug channel which was configured to output through Serial, which CompanionGUI will pick up and act accordingly.

CompanionGUI Development

Originally, we planned to use SMS as the main form of communication between AutoPing and the employees using Twilio, but that proved to be too much of a hassle due to the restrictions, message limits, and verification hurdles that Twilio’s trial account provides.

So, we turned to the most developer-friendly messaging platform in the world: Discord!

Discord also proved to be a highly optimal form of owner to employee communication due to its inherent group chat, channel, and server formats. And most of all, there is a robust community around developing Discord bots!

Step 8: Setting up the messaging platform

The first step involves creating a Discord account and setting up a server for the AutoPing application to run in. If you frequent Hackster, there’s almost a 100% chance you already have Discord and are very well-versed with it, so I won’t bother describing the steps I took to do that here.

Step 9: Turn developer mode on

Log into discord and click on the gear icon next to your username, scroll down to Advanced and turn on developer options. This will give you the ability to make bots and control more aspects of Discord.

Step 10: Make your application

Go to https://discord.com/developers/applications and click on create new application.

Go to Bot and click "Reset Token" generate a secret token that you will need for the python code to interface with the discord bot.

Keep the token in a safe spot as you will never see it again!

Then, we go and give our bot the appropriate permissions and turn on intents, which allow the bot to send and receive messages as well as see who is in the server.

Step 11: Invite the bot to your server

Go to https://discord.com/oauth2/authorize?client_id=ID_HERE&scope=bot&permissions=8, where you replace ID_HERE in the URL with your Application ID found in your application page and then once you invite the bot to your server you are done with the Discord part!

While you grab your Application ID, you should also grab the Channel ID by right-clicking on the channel where AutoPing resides in and clicking "Copy Channel ID"

Step 12: Python and Testing Discord

The complete code for the discord client runner and the gui are found in the Github under the folder CompanionGUI. This code was fairly difficult to write and debug because it involved asynchronous operations and multithreading as the GUI, discord client, and serial monitor all needed to run simultaneously.

In order to do this, first run the following command in Terminal to install all appropriate libraries:

pip install discord.py PyQt5

After this, we can open up message.py and edit the following lines at the top of the code by providing your own Discord application token and channel ID.

TOKEN = "YOUR TOKEN HERE (WITH QUOTES)"
ID = CHANNEL ID HERE (NO QUOTES)

Then run the program using the following command

python message.py

Now, if you send a message in the server, it will also show up in the terminal.

Step 13: Run AutoPing

We have now verified that Discord is up and running, so next we can run the main GUI.

First, open up psoc_interface.py, add your Token and Channel ID just like what you did in step 12, but also add the Discord usernames of your employees to this line:

employees = []

After that has been set, run the following command in Terminal.

python psoc_interface.py

The following window should appear. Choose the COM port of the EdgePing device and click Next.

Once you have connected, you should see the main interface and you are ready to go!

Step 14: Using CompanionGUI

CompanionGUI is very easy to use. The color indicator square shows a visual representation of the EdgePing state. Ready means that the EdgePing is on standby and listening for a "busy" event.

Once EdgePing notifies CompanionGUI that it is busy, a countdown timer will start and the Abort button will become clickable. If the Abort button is pressed, it will cancel the message send, but if the timer runs out, it will send a Discord message tagging the employee to come to work.

Finally, close connection is used when you want to disconnect and close the application.

Future Development

AutoPing merely scratches the surface of the possibility of customer traffic prediction.

With a more advanced ML model as well as access to more historical customer trend data, EdgePing may even become possible to predict customer surges well in advance and give more notice to employees.

Future updates of CompanionGUI will allow owners to directly use the GUI to input employee names, schedule priorities (aka prioritize contacting one employee over the other), connect to Discord more seamlessly, etc.

But for now, this application works very well for my personal use case, and hopefully this will be of value to some other small business out there!