Software apps and online services
Hand tools and fabrication machines
In my previous project, I showed how to control a few LEDs using an Arduino board and BitVoicer Server. In this project, I am going to make things a little more complicated. I am also going to synthesize speech using the Arduino DUE digital-to-analog converter (DAC). If you do not have an Arduino DUE, you can use other Arduino boards, but you will need an external DAC and some additional code to operate the DAC (the BVSSpeaker library will not help you with that).
In the video below, you can see that I also make the Arduino play a little song and blink the LEDs as if they were piano keys. Sorry for my piano skills, but that is the best I can do :) . The LEDs actually blink in the same sequence and timing as real C, D and E keys, so if you have a piano around you can follow the LEDs and play the same song. It is a jingle from an old retailer (Mappin) that does not even exist anymore.
The following procedures will be executed to transform voice commands into LED activity and synthesized speech:
- 1. Audio waves will be captured and amplified by the Sparkfun Electret Breakout board;
- 2. The amplified signal will be digitalized and buffered in the Arduino using its analog-to-digital converter (ADC);
- 3. The audio samples will be streamed to BitVoicer Server using the Arduino serial port;
- 4. BitVoicer Server will process the audio stream and recognize the speech it contains;
- 5. The recognized speech will be mapped to predefined commands that will be sent back to the Arduino. If one of the commands consists in synthesizing speech, BitVoicer Server will prepare the audio stream and send it to the Arduino;
- 6. The Arduino will identify the commands and perform the appropriate action. If an audio stream is received, it will be queued into the BVSSpeaker class and played using the DUE DAC and DMA.
- 7. The SparkFun Mono Audio Amp will amplify the DAC signal so it can drive an 8 Ohm speaker.
- Arduino DUE: ~U$ 50.00
- SparkFun Mono Audio Amp Breakout: U$ 7.95
- BitVoicer Server 1.0: U$ 9.90
- 8 Ohm speaker: ~U$ 2.00
- Breadboard: ~U$ 10.00
- 3 x LEDs: ~U$ 1.00
- 3 x 330 Ohm resistors: ~U$ 0.75
- Jumper wires: ~U$ 0.50
The first step is to wire the Arduino and the breadboard with the components as shown in the pictures below. I had to place a small rubber underneath the speaker because it vibrates a lot and without the rubber the quality of the audio is considerably affected.
Here we have a small but important difference from my previous project. Most Arduino boards run at 5V, but the DUE runs at 3.3V. Because I got better results running the Sparkfun Electret Breakout at 3.3V, I recommend you add a jumper between the 3.3V pin and the AREF pin IF you are using 5V Arduino boards. The DUE already uses a 3.3V analog reference so you do not need a jumper to the AREF pin. In fact, the AREF pin on the DUE is connected to the microcontroller through a resistor bridge. To use the AREF pin, resistor BR1 must be desoldered from the PCB.STEP 2: Uploading the code to the Arduino
Now you have to upload the code below to your Arduino. For convenience, the Arduino sketch is also available in the Attachments section at the bottom of this post. Before you upload the code, you must properly install the BitVoicer Server libraries into the Arduino IDE (Importing a .zip Library).
Arduino Sketch: BVS_Demo2.ino
This sketch has seven major parts:
- Library references and variable declaration: The first four lines include references to the BVSP, BVSMic, BVSSpeaker and DAC libraries. These libraries are provided by BitSophia and can be found in the BitVoicer Server installation folder. The DAC library is included automatically when you add a reference to the BVSSpeaker library. The other lines declare constants and variables used throughout the sketch. The BVSP class is used to communicate with BitVoicer Server, the BVSMic class is used to capture and store audio samples and the BVSSpeaker class is used to reproduce audio using the DUE DAC.
- Setup function: This function performs the following actions: sets up the pin modes and their initial state; initializes serial communication; and initializes the BVSP, BVSMic and BVSSpeaker classes. It also sets “event handlers” (they are actually function pointers) for the frameReceived, modeChanged and streamReceived events of the BVSP class.
- Loop function: This function performs five important actions: requests status info to the server (keepAlive() function); checks if the server has sent any data and processes the received data (receive() function); controls the recording and sending of audio streams (isSREAvailable(), startRecording(), stopRecording() and sendStream() functions); plays the audio samples queued into the BVSSpeaker class (play() function); and calls the playNextLEDNote() function that controls how the LEDs should blink after the playLEDNotes command is received.
- BVSP_frameReceived function: This function is called every time the receive() function identifies that one complete frame has been received. Here I run the commands sent from BitVoicer Server. Commands that controls the LEDs contains 2 bytes. The first byte indicates the pin and the second byte indicates the pin value. I use the analogWrite() function to set the appropriate value to the pin. I also check if the playLEDNotes command, which is of Byte type, has been received. If it has been received, I set playLEDNotes to true and mark the current time. This time will be used by the playNextLEDNote function to synchronize the LEDs with the song.
- BVSP_modeChanged function: This function is called every time the receive() function identifies a mode change in the outbound direction (Server --> Arduino). WOW!!! What is that?! BitVoicer Server can send framed data or audio streams to the Arduino. Before the communication goes from one mode to another, BitVoicer Server sends a signal. The BVSP class identifies this signal and raises the modeChanged event. In the BVSP_modeChanged function, if I detect the communication is going from stream mode to framed mode, I know the audio has ended so I can tell the BVSSpeaker class to stop playing audio samples.
- BVSP_streamReceived function: This function is called every time the receive() function identifies that audio samples have been received. I simply retrieve the samples and queue them into the BVSSpeaker class so the play() function can reproduce them.
- playNextLEDNote function: This function only runs if the BVSP_frameReceived function identifies the playLEDNotes command. It controls and synchronizes the LEDs with the audio sent from BitVoicer Server. To synchronize the LEDs with the audio and know the correct timing, I used Sonic Visualizer. This free software allowed me to see the audio waves so I could easily tell when a piano key was pressed. It also shows a time line and that is how I got the milliseconds used in this function. Sounds like a silly trick and it is. I think it would be possible to analyze the audio stream and turn on the corresponding LED, but that is out of my reach.
Now you have to set up BitVoicer Server to work with the Arduino. BitVoicer Server has four major solution objects: Locations, Devices, BinaryData and Voice Schemas.
Locations represent the physical location where a device is installed. In my case, I created a location called Home.
Devices are the BitVoicer Server clients. I created a Mixed device, named it ArduinoDUE and entered the communication settings. IMPORTANT: even the Arduino DUE has a small amount of memory to store all the audio samples BitVoicer Server will stream. If you do not limit the bandwidth, you would need a much bigger buffer to store the audio. I got some buffer overflows for this reason so I had to limit the Data Rate in the communication settings to 8000 samples per second.
BinaryData is a type of command BitVoicer Server can send to client devices. They are actually byte arrays you can link to commands. When BitVoicer Server recognizes speech related to that command, it sends the byte array to the target device. I created one BinaryData object to each pin value and named them ArduinoDUEGreenLedOn, ArduinoDUEGreenLedOff and so on. I ended up with 18 BinaryData objects in my solution, so I suggest you download and import the objects from the VoiceSchema.sof file below.
Voice Schemas are where everything comes together. They define what sentences should be recognized and what commands to run. For each sentence, you can define as many commands as you need and the order they will be executed. You can also define delays between commands. That is how I managed to perform the sequence of actions you see in the video.
One of the sentences in my Voice Schema is “play a little song.” This sentence contains two commands. The first command sends a byte that indicates the following command is going to be an audio stream. The Arduino then starts “playing” the LEDs while the audio is being transmitted. The audio is a little piano jingle I recorded myself and set it as the audio source of the second command. BitVoicer Server supports only 8-bit mono PCM audio (8000 samples per second) so if you need to convert an audio file to this format, I recommend the following online conversion tool: http://audio.online-convert.com/convert-to-wav.
You can import (Importing Solution Objects) all solution objects I used in this project from the files below. One contains the DUE Device and the other contains the Voice Schema and its Commands.
Solution Object Files:STEP 4: Conclusion
There you go! You can turn everything on and do the same things shown in the video.
As I did in my previous project, I started the speech recognition by enabling the Arduino device in the BitVoicer Server Manager. As soon as it gets enabled, the Arduino identifies an available Speech Recognition Engine and starts streaming audio to BitVoicer Server. However, now you see a lot more activity in the Arduino RX LED while audio is being streamed from BitVoicer Server to the Arduino.
In my next project, I will be a little more ambitious. I going to add WiFi communication to one Arduino and control two other Arduinos all together by voice. I am thinking of some kind of game between them. Suggestions are very welcome!