Hardware components | ||||||
 |
| × | 1 | |||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
 |
| |||||
In this project, we will show you how we made a sensor that can tell you if a bar or pub is full of people or not.
IntroductionHave you ever been in a situation, with friends or alone, when you wanted to go to a bar/pub and didn't know if the bar/pub was full and there is a party or empty and you could have a drink and talk? With this app, you won't have any problems like this in the future.
DescriptionThis project consists of an Arduino Nano board, which acts as our sensor or microphone, and a mobile app, that acts as our search engine for finding pubs and bars in our area.
Information about how full the pub/bar is is sampled by the microphone on the Arduino board. It then processes this information using machine learning algorithms to determine whether the pub/bar is full of people or not. The information is then sent via Bluetooth Low Energy (BLE) to the connected phone, where we can check for the information of any pub/bar we want to go to.
Machine learningEdge Impulse is a free development platform for machine learning. We created an account using a shared email and created a new project. This is important because making a project public or sharing it with others will not enable them to make adjustments to the model or help you collect data.
- Data acquisition
The best way or the easiest way to collect data was using our phones (and not the Arduino board) as it was simpler to set up and faster to acquire the data. With the Sound recorder app on our phones, we collected 3 different data (Voice, Music, and Silence).
We decided that each of us will contribute at least 20 minutes of data. Our sample duration is 10 seconds, which means we each contributed at least 120 samples of different data. As we did not have access to pubs/bars, because of the pandemic, we had to imitate different scenes with family members.
When we collected enough data we uploaded it to the Edge Impulse data acquisition. Edge Impulse automatically split the data between the training set and the testing set. Usually, the ratio is 80% (training set) and 20% (testing set), but you can use our own ratios by assigning samples to their respective sets.
- Impulse design
The next step is creating an impulse. After selecting "Create impulse" you add a processing block and a learning block and then save the impulse. For the processing block, we selected the Audio (MFCC) block, which extracts features from audio signals using Mel Frequency Cepstral Coefficients and is great for the recognition of the human voice. For the learning block, we selected the Neural Network (Keras) block, which learns patterns from data, and can apply these to new data. This block is great for categorizing movement or recognizing audio.
You can check/change/save the parameters under ''MFCC'' and then generate features to see the representation of the data you collected. To train your model select "NN Classifier" and click "Start training". When the training is complete you will get the accuracy of the last training performance, the confusion matrix, and a 3D model of the results.
After this, you can test your model under the ''Model testing'' tab. Should you want to improve the accuracy of your model, you will likely have to play with the parameters of the MFCC and NN Classifier blocks, but you have to be careful because with higher accuracy the complexity of the model will get greater and with that, it is possible, that the Arduino Nano won't be able to run the model.
DeploymentFor using the Arduino Nano board you need to create the right library. Under the "Deployment" tab we selected the "Arduino library" and Edge Impulse created our own library, which was made on the basis of our project. You will be presented with a zip library containing your model, which you can then compile onto your Arduino Nano.
Now we can check if the model works properly. In the serial monitor, you should be getting feedback, for 3 different things. If you talk with someone, the serial monitor should print out the Talk label, if you are listening to music, the label should be Music and if there is silence the label should be Silence.
Because we need to connect the Arduino via BLE, we slightly modified the exported code from Edge Impulse. It also automatically connects the app without selecting the proper device from a list of available Bluetooth devices.
We modified the code so that the Arduino can send the right label to the app. If the Arduino read more talking than music and silence then the app would tell us that this pub/bar is full of people, if the music was louder then the communication of the people present or the silence the app would tell us that the pub/bar is open, but there are not many people there, and if there was no music nor people talking, the app should tell us that in the pub/bar there is silence and that means, that it is probably closed. Furthermore, we modified our code so that the Arduino takes an average of the last 5 minutes (last 30 samples) and refreshes every 5 minutes or the instance there are any drastic changes.
Mobile applicationThe mobile application, we decided to use is the Mit2app inventor for android devices, because it is practical in the meaning of the programming interface, fast, and easy to use for beginners. The downside however is in graphical user interface design. The blocking diagram for the codding is seen in the picture below.
The application is simple to use. You just log in to the app and then go to the maps, where you (in the future, because this is only a prototype) will see marks of the pubs, and on the base of the color, you can decide where to go.
Here is a video that shows you the use of the app and how it works.
ConclusionThe app works pretty well for a prototype. This app was made for only one pub/bar, to see if it would work, so there are many improvements to be made, which would significantly enhance the user experience and should be considered before trying to put the app on the market.
- improvements
The first thing that limited us was, that because of the pandemic all of the pubs/bars were closed so we could not test this in a real situation. Tests were made, but with a small number of people in one place.
Furthermore, you should consider including more pubs/bars, but the problem is to store all the data that is collected. For this, you would need a lot more Arduinos or some microphones, an online server for the Arduinos to connect to, and where the data, that is sent via services from the pubs/bars with different IDs, would be stored.
The app itself is functional, but it could be made more appealing to the eye. Before any introduction to the market, a switch to native code should be considered as it enables much more flexibility in app design.
Comments