Software apps and online services
Petoi's Bittle is a palm-sized, open-source, programmable robot dog for STEM and fun. Bittle can connect with Raspberry Pi and can be easily extended. This project is done during my intern in Petoi. The goal was to develop a real-time voice control module for Bittle and command Bittle to do some actions.Demo VideoAbstraction
The conclusion is that I use VAD(Voice Activity Detection) + DTW + Vosk
Use Python to record
I used PyAudio at the beginning, but it is an old library. So I used Sounddevice and Soundfile instead.
Command/Key Words Recognition
From a functional point of view, the methods to do this can be divided into：
- Speech to Text. And then look up the commands in the text. One good thing is that this can be combined with NLP applications but this is an overkill for Speech2Text.
- Use acoustic features to do analysis and detect commands.
DTW (Dynamic Time Warping) (Used)
This belongs to the second category and it's similar to template matching. DTW can calculate the cost to match one piece of audio with template audio. We can pick the audio with the lowest cost. This method does not need training and is also applicable even if you want to add new commands. The bad thing is that the calculation is time-consuming. But at least the command audios are short in time and we can find ways to eliminate the silence and extract MFCC(Mel-frequency Cepstral Coefficients) feature.
CNN for Command/Key Word Recognition
This is a demo Speech Command Recognition with torchaudio — PyTorch Tutorials which is done by PyTorch Official. But we need to re-train the model when we have new commands coming in.Voice Activity Detection
I was inspired by a blog Audio Handling Basics: Process Audio Files In Command-Line or Python | Hacker Noon. The blog mentions that we can eliminate the silence part of an audio recording according to the short-term energy of audio data. A Python library called librosa provides some functions for doing that.Speech Recognition
I tried some open source methods:
Offline recognition, provides lightweight tflite models for low-resource devices.
Requires 16bit 16KHz mono channel audio. A new version supports Chinese.
I tested it by using non-strip and stripped audios with both large and small size models but it did not do well. For example:
- 起立 -> 嘶力/成立
- 向前跑 -> 睡前跑
- 向前走 -> 当前走
So I tested it again using English:
- Hey Bittle
- Stand up
- Walk forward
- Run forward
I have used 16 recordings for now. An empty result is shown when it encounters OOV(out of vocabulary) words. "Bittle" would be recognized as "be to". After silence elimination, some results have changed from wrong to correct, and some have changed from correct to the wrong (this may be due to the reduction of the silence between the pronunciation of words).
16 English Tests, 9 were correct &16 Chinese Tests, 3 were correct.
It does not have lightweight models and the models are near 900MB. It's too big for a Raspberry Pi.
It provides multiple ways such as using Google/MS Api. The only one method to offline recognition is no longer being maintained.
- alphacep/vosk （Used）
Vosk provides offline recognition and light-weighted models for both Chinese and Chinese. The documentation is not complete.
A test result for the Chinese model
Non-strip Correct----------|----------Stripped Correct----------|----------Total correct
16/21----------------------------|-----------16/21--------------------------|-----------32/42Prepare environmentSimple Version (Run on Pi):
1. Enter Pi's terminal:
sudo apt-get update && sudo apt-get upgrade
sudo apt-get install make build-essential libportaudio0 libportaudio2 libportaudiocpp0 portaudio19-dev libatlas-base-dev
3. Create a new virtual environment with
python==3.7.3 and activate it.
- If you use
python3 -m venv /path/to/environment
4. Install Scipy. Download the wheel file
pip install your_filename
Make sure you download the one with
cp37-cp37m-linux_armv7l in its name.
- Deactivate the environment and then:
sudo apt install libblas-dev llvm llvm-dev
- Activate the environment and then: Note: The correct version number of
llvmlitecan be found at pypi. You should determine according to the llvm version installed by the first command above. Same for
pip install llvmlite==[correct version number]
pip install numba==[correct version number]
- Download and unzip librosa source code. Then cd into the directory.
python setup.py build
python setup.py install
Example for version number:
llvm==7.0.1 corresponds to
6. Install the remaining dependencies.
pip install -r requirements_pi.txt
7. Download vosk model. Download
vosk-model-small-cn. The former is for English(US) and another one is for Chinese. Both of them are small and portable models.
8. Current codes use English Model for default, download and extract the model. Move the folder you just got into
./models. Make sure the folder name is the same as
vosk_model_path in your config.
1. IMPORTANT: Let terminal enter
my_vosk and input:
2. You can skip the recording step. A pre-recorded file is saved at ./recordings/template_1.wav A file with
raw in its name means the wave file has not been stripped(silence) yet.
3. Finetune the
threshold for wakeup recognition. In config.yml, the value is now 0 for easy debug. After "开始监听(start listening)", check the console for
DTW.normalizedDistance. For example：
- Before you say the "wakeup word", the value is between 115-125;
- After you say the "wakeup word", the value should decrease. Maybe between 105-117.For this case, you can set
thresholdas 120(strict) or 115(not so strict).
4. You should go into command recognition after waking up Petoi. There are many pre-defined commands in cmd_lookup.py. You can say "stand up", for example.
5. If you want to use Chinese model(similar for other languages):
- Unzip the Chinese model and put the folder into
- Set the
cmd_tablein config.yml as below: