Software apps and online services
Hand tools and fabrication machines
Imagine a chef is cooking something while preparing salad or making dough, now he/she might need to adjust the heating on the stove. This is a scenario where voice controlled stove/oven can be very useful for a cook/chef. In any situation where user's both hand are busy doing something is the best scenario to utilize voice control applications.
Unfortunately existing solutions are connected solutions, one of those Alexa or Google cloud connected oven thingies. But, why let them know what you cook or when you cook ? Also, user needs to setup wifi connection for the device and download another app ! This is a big NO for many users.
This is not an "Internet of Things" but an "Internet OFF Things", so it's still an IoT (pun intended).Features
- Voice controlled Ignition/Flame activation
- Voice controlled flame/heating adjustment
- Voice controlled cooking/burner timer setup
- Interactive Voice Feedback UI for confirmation
- Voice controlled Power off
- Edge AI voice recognition, no cloud connectivity
- Natural Language multi-commands operation
These hardware are used to built this project :
PSoC™ 6 62S2 Wi-Fi BT Pioneer Kit (CY8CKIT-062S2-43012)
This is the development board for PSoC™ 62 dual core MCU CY8C624ABZI-S2D44 (150-MHz Arm Cortex-M4 and 100-MHz Arm Cortex-M0+ cores) with 2MB of Flash. It is powerful enough to run voice recognition AI without internet connectivity
IoT sense expansion kit (CY8CKIT-028-SENSE)
The IoT sense expansion kit is an add-on (shield) board that can be used with the Pioneer Kit, this board has analog/digital microphone for audio input and audio codec for audio output, which is required for this project
Voice Playback Module
My original goal was to use the IoT sense kit's audio codec for playing Voice UI messages and 2MB flash on PSoC6 to store the wave files. Unfortunately, I was unable to use the I2S Audio demo code provided in Modustoolbox. That's why I decided to use a project I made last year which can store up to 1000 separate audio messages in mp3 format up to 40 minutes long.
Here is the project in details : Countdown Voice Timer & Fire Starter
I have changed the original firmware (attached below), so this module can receive serial messages sent from PSoC 6 and playback appropriate audio messages accordingly.
This module consists of an LM7809 Linear Voltage Regulator and DC-DC Buck Converter module that can output 5 volts. I have used the 9 Vdc output to power the PSoC6 Pioneer. The 5 Vdc from the Buck Converter is used to supply adequate power to the servo motors and the Voice Playback module.
Metal Gear Servo Motors
There are two metal gear servo motors used for this project. One of the servo motor controls the (fuel) gas flow valve by partially opening is as needed. This servo motor can fully close the valve whenever the burner needs to be turned off. The other servo motor pulls the lever of peizo-igniter to generate spark for starting a flame.
Portable Gas Stove
The portable gas stove is a tiny single burner stove with build-in peizo igniter fire starter and a gas flow adjustment valve. I have picked this stove for ease of prototyping since it's easy to bring the whole setup near my computer for developing the firmware with a tiny stove. Of course, this project can be scaled for larger, multi-burner stove setup.
Just like any quick and low budget prototyping, I have assembled the hardware with super glue, hot glue, cable tie and jumper wires. Main goal was to cut down cost, time and effort.
All the pieces are connected according to this schematic :
One of the servo motor is superglued to the gas adjustment valve like this :
The other servo motor is threaded to the peizo ignition spark generator lever like this :-
This is how the Serial TX line is hacked out without modifying the Dev kit:-
All the parts are fastened with cable tie on a piece of hardboard. Jumper wires are used to interconnect all the hardware pieces.
To program the PSoC6 Pioneer Kit, please download and install the Modustoolbox(MTB) SDK. I have used MTB version 2.4, any version prior to version 2.1 of MTB should be avoided for this project.
Modustoolbox download link : https://www.infineon.com/cms/en/design-support/tools/sdk/modustoolbox-software/
You might face some difficulties with MTB after installation. Make sure to add path to tools for MTB to work properly.
Following installation guide from Infenion explains this setup process :-Modustoolbox Installation Guide
Once the Modustoolbox(MTB) is installed, let's get started with the voice recognition demo first.
Run MTB and click on New Application from Quick Start menu:
Make sure your Internet connection is active, as this process will download the example application demo from Github
Select the appropriate platform (PSoC 6) and correct board (CY8CKIT-062S2-43012) from Board Support Package menu
Next. select the Voice template application and check the demo project Picovoice E2E Voice Recognition Demo and click create
Now, open the main.c file inside the Picovoice E2E Voice Recognition Demo code. Find the line that accepts Picovoice Access Key, we will come back here later
Keep the MTB running in the background and open your web browser app, it's time for Picovoice.
Picovoice is one of the most intuitive way to build a voice recognition model and implement on different platforms. Simply put, you just need to type wake words and speech to intent words, then build the platform specific voice recognition model. Once the model is generated, download and implement the voice recognition model in your embedded development project.
Following steps will explain the process :
Step 0: Go to https://console.picovoice.ai/, create a new account and login. This page will appear
Step 1: Access Key and UUID (Unique Identifier)
First click on "Show AccessKey" and it will provide an unique string, copy this string and save it on a text file. This Access Key will be required in the code.
The Access Key looks something like this (this is a dummy string) :
This string must go between " " and end with ; like this inside the code:-
static const char* ACCESS_KEY = "XIn8zpxLsfsdfsadVmTTiPxze+v1A3ESyCdfsdfsdhXXrWwOKS0orK1VtQK6kg==";
To make voice recognition models, one more information is needed, that is the device UUID ( In this case the UUID of the PSoC 6 (mcu) Pioneer Kit ).
Copy the Access Key string and Paste/Insert into the MTB demo code that was previously opened like this :
Now, select the demo project and click on the build button (hammer icon).
While, the build process is going on (it will take some time), connect the PSoC6 Pioneer Kit (+ IoT Sense Kit connected on it) to computer with an USB cable.
Once the build is completed, flash the code by clicking the Green Play icon, make sure to select the right programmer
Then say, loudly "Hey Pico"
You should get a response on serial monitor with an UUID of your PSoC6 Pioneer Kit, something like this (hexadecimal number) :
This UUID will be needed to make voice recognition models in Picovoice Console.
Now. let's go back to Picovoice console.
Step 2: Wake Word
Picovoice wake word is a special trigger word which will activate the device. I am going to use the word : Hey Stove as a wake word, because this is a voice controlled stove.
Go to Porcupine option on Picovoice console (https://console.picovoice.ai/ppn)
- Type Phrase: Hey Stove
- Select Language: English
- Select Platform: ARM Cortex-M
- Select Board: PSoC6
- Type the UUID/Unique Identifier from before: (hexadecimal number)
Next, click Train Wake Word, with in few seconds your wake word trained model will be ready. Download the trained model
You will get a downloaded zipped folder, unzip it and open the pv_porcupine_params.h file with any text editor (notepad). This is the wake word model, a large array of hexadecimal numbers.
More on this will be explained later.
Step 3: Speech to Intent Model
The wake word "Hey Stove" will activate the device but it won't perform any stove action. I want to turn on the stove, turn off the stove, change gas/flame, set cooking timer etc. I have intentions to do certain tasks with my stove. This is where the speech-to-intent engine is needed.
Picovoice Rhino Speech-to-Intent engine can extract multiple spoken commands, which can be used to perform different operations.
Go to Rhino option on Picovoice console (https://console.picovoice.ai/rhn)
Type name: Voice Controlled Stove
Select Template: Empty
Then click Create Context, this will take you to a new page where the actual speech-to-intent model 'coding' is done.
Here is a quick overview of Intents, Slots and Macros for Picovoice:-
Intents are certain actions that you want to perform with voice control. Think of them as verbs used in grammar. Intents have expressions, which closely follows English grammar for sentences with a verb.
Intent Expressions= Action (do something) + Subject/Object (about something)
This voice control stove has 4 separate intent:
- StartBurner : To start a fire on the stove's burner
- StopBurner : To stop the fire
- SetFlame : To change the flame by adjusting gas
- SetTimer : To set a cooking timer to turn off the fire when time elapsed
Each intent has it's own expression which you must type. An expression should be like this :
(polite word) @Macro(i.e. Verb/Action) (article words) $Slot(i.e. Noun/Variables)
Please Start the Burner
Can you Set Timer to Five Minutes?
It may seem a little confusing at first, but you will get it eventually (don't worry)
Slots are the variables in an intent. Slots can hold multiple elements representing different variables or different names for one variable
Think of it as the subject/object in English grammar on which the action needs to be performed.
For example, if we want to set a cooking timer, the time will be different depending on the command.
- Set timer to one minute
- Set timer to three minutes
- Set timer to five minutes
Slots = Variables in actions (usually)
Speech to Intent engine can extract these differences.
Slot will have $ symbol before them when when used inside an expression
Picovoice has some built-in slots, like pv.Percent which can directly extract a percent value from user's intent.
Macros are exactly used for different words with same meaning (intent). It is equivalent to Synonyms in English Grammar.
Macros will have @ symbol before them when used inside an expression
Macros = Synonyms for verbs
For example: Start can be said as turn on/power up/enable/activate any of these words.
So, how do you put everything together?
First, think of allthe possible ways a human can say the 'same thing' which represents a specific intent. Then list all the verbs/nouns in Macros, list all the variables in Slots and create an expression for the intent like this :
(please) @Start (the) $Flame:Flame
The Voice Recognition model will work better if it can detect wide variety of human phrases with same intention. Properly listed Macro elements and better expressions are the key to success.
Please read this to get a better understanding : https://picovoice.ai/docs/tips/syntax-cheat-sheet/
Step 4: Training the Model
Once all the Intents, Slots, Macros are done, click on "Train" on the bottom right corner:
Select as follows:
- Model Platform: ARM Cortex-M
- Board Type: PSoC6
- Unique Identifier (UUID) : That Hexadecimal Number
Then click Train "Voice Controlled Stove" for ARM Cortex-M
Go to https://console.picovoice.ai/rhn and download the speech to intent model.
Unzip the downloaded file.
Then, open the Voice-Controlled-Stove_en_cortexm_v2_1_0.h
There is this large array of hex numbers inside the file, this is the speech to intent model.
Step 5: Editing main.c and param.h in Modustoolbox
Now, go back to Modustoolbox. Copy the main.c code attached here.
Replace the entire original demo code from the main.c of Picovoice E2E Voice Recognition Demo
Next, edit the param.h by including the wake word model's array and speech to intent model's array. Both models (wake word + speech to intent) are combined together into param.h.
I have attached the modified param.h below. Copy it's content and replace the original contents of param.h to avoid any confusion.
If you want to understand how the code works, please read it. There are tons of comments in the code explaining most of the things
Finally, click the hammer icon to build and green play button icon to flash the firmware into PSoC6 mcu, as explained before (see step 1).Preparing Audio Files for Voice Playback
Step1: I have used https://ttsreader.com/ text-to-speech service to make the audio feedbacks.
Step 2: While the speech was playing, I have recorded the sound with Audacity . Select Windows WASPAPI and Source as Headphone/Speaker, then click the Record icon to capture audio playback in your computer.
Step 3: After recording, the audios are segmented into multiple mp3 files. To separate each segment of the audio (sound for every number), select that part with mouse, then use shortcut Ctrl+C (copy), Ctrl+N (new), Ctrl+V (paste) that audio segment. Repeat this step for each audio voice segment.
Step 4: Go to Files > Export as MP3, select 48 kbps quality and rename as FILE0000.MP3, FILE0001.MP3, FILE0002.MP3 up to FILE0100.MP3
(all the audio files are attached below, you can skip step 1-4, just download and copy the files according to step 5)
Step 5: Voice module is connected to computer with a USB C cable, 16 MB Flash Drive appeared. It will require formatting (FAT) when using for the first time. Next, all the mp3 files are copied to a DF_REC directory inside the voice module's flash drive.
You can put maximum of 1000 MP3 files with total audio length of 40 minutes @ 48kbps. Higher sampling frequency song/audio voice will work but the audio length will be less due to larger file size
How the audio playback works ?
PSoC6 mcu's serial TX and Beetle board mcu's serial RX are connected for half duplex serial communication. PSoC6 mcu simply send's printf("some number"), which is transmitted to Beetle board's mcu. Depending on the number that was received from PSoC6 mcu, Beetle board mcu will send command to the Voice Playback Module to play the mp3 file representing that number.
For example: printf("040") command will play 0040.mp3 file. Check this project of mine, which explains this in full details :Device Operation
To understand the device operation, let's have a look at all the PSoC6 Application Resources (PSoC6 mcu internal hardware)
Depending on the voice commands from user, some of these resources will activate certain modules to control servos, enable timer, send printf messages to voice playback module etc.
- Two PWM modules will generate 50 Hz PWM signals. By adjusting the duty cycles of these signals, both servo motors are controlled.
- LPTIMER will measure time elapsed when cooking timer function is active
- UART TX will send printf() messages to Beetle board to play mp3 files
- GPIO will drive Green LED on PSoC6 Pioneer Kit board when wake word is detected. Blue LED will glow when speech to intent is detected.
Here are some example voice commands and hardware interactions:
Of course, these are not the only voice commands. For example:
"Hey Stove, turn on the burner" can also be said as :
"Hey Stove, start the fire"
"Hey Stove, turn on the flame"
"Hey Stove, activate the burner"
Gas/Flame adjustment constrains
Gas or Flame can be adjusted with any value between 10% to 100%, in step of 5%. Which means, 10%, 15%, 20%, 25%,....... 100%
Here are some valid voice commands examples :
"Hey Stove, change gas to 40%"
"Hey Stove, increase flame to 65%"
"Hey Stove, reduce gas to 25%"
For any values which is not divisible by 5 will have a default gas valve setting of 30%
Timer setting constrains
Cooking duration timer can be : 1 minute, 2 minutes, 3 minutes, 5 minutes, 10 minutes, 15 minutes, 20 minutes, 25 minutes, 30 minutes only
Of course, more or less options can be added by editing the main.c code.
Here are some valid timer setting voice commands:
"Hey Stove, set timer to 1 minute"
"Hey Stove, change timer to 5 minutes"
Existing timer setting can be overridden with new timer setting commands. For any value of minutes which is not listed above, default timer setting is one minute.
I have made this project with a limited budget (around 50$ of extra hardware) + PSoC6 Pioneer Kit which is provided by Infineon and Picovoice developer account which is provided by Picovoice for free. But if anyone wants to turn this into a product, more features may be added.
Here are few ideas for additional features:
- Flame and Gas sensors for safety
- Temperature Sensor for closed loop controlled heating
- Built in cooking profiles with multi stage heating
- Spillage detection on stove's surroundings
- Overcook protection/food anti-burn protection
- Multi burners cooking operation
Flame sensor will protect against failed ignition and turn off the gas after few retries, gas sensor will notify user if there is gas leakage in absence of flame or there is no gas available.
Built in cooking profiles can take care of temperature adjustments for cooking different foods. User will say something like this : "Hey Stove, hard boil my eggs" or "Hey Stove, I want extra crispy French fry !"
Spillage detection will automatically adjust heating to avoid/reduce spillage of boiling milk or soup.
Same with overcook protection, no more burnt cooking. Some sensors like Acrylamide sensor may be needed. Multi burner cooking will allow chefs to cook lots of dishes at once.
All these features will have voice control and voice feedback as usual. My demo project is just the starting point for anyone willing to turn this into a real world product.