Voice Control Toy and Display using XIAO and Edge Impulse

I'm a Genshin Impact player and I like one character who's name is HuTao. The game helps relax. By chance I found a very interesting GIF image published by acumo which is cute and kind of inspired me. Hence, I was planning to build a voice control toy based on it, and thanks to RENZO MISCHIANTI, MJRoBot (Marcelo Rovai) and Kason who make this actually come to reality.

These are I am going to do:

1. Display the GIF image on my 0.66 OLED screen.

2. Make the toy active.

3. Use XIAO nRF52840 and PDM on it, to train an embedded machine model to classify "hutao", "shake" and regular environment voice.

4. Deploy the model on XIAO nRF52840 Sense board, connecting Grove-Relay and the toy to achieve voice control toy.

5. Deploy the model on XIAO nRF52840 Sense board, connecting a battery and Grove 0.66 OLED screen to achieve voice control animations.

This is my first time writing a project, and when there are something that's not clear please contact me.

And... while I am just about finishing the project, I realize that I can do it with one XIAO nRF52840 Sense board and oneXIAO Expansion board... Maybe later I can update...

Display the GIF Image on the 0.66 OLED Screen

I display many discrete and static images to reach continuous and dynamic animations.

Step1. First change the GIF image to static images. I use the MacBook here so it is pretty easy.

Step2. Then convert images into byte arrays using Image2cpp.

Because of Grove - 0.66 OLED screen, I set the images as 64 x 48 pixel and because the original images are larger, I set up the scaling as "scale to fit, keeping proportions".

The rest settings are default and output are like(0.66 display are pretty small, but it kind of cute):

Select "plain bytes" and click "Generate code" to get my display code.

Step3. Fill the code with plain bytes below where "6" represents 6 overall images and "3072" represents "64x48" as the size of single image.

const unsigned char myBitmap [6] [3072] PROGMEM = 
{
// 1st image
{
},
// 2nd image
{
},
// 3rd image
{
}
...
};

Since the Grove 0.66 OLED display is based on the 128×64 resolution screen, the code of width and height are still 128 and 64. And range is from (31, 16) to (95, 63) which means the start point at (31, 16) instead of (0, 0).

This will be the code:

#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#define SCREEN_WIDTH 128 // OLED display width, in pixels
#define SCREEN_HEIGHT 64 // OLED display height, in pixels
// ------------------- For i2c -------------------
//// Declaration for an SSD1306 display connected to I2C (SDA, SCL pins)
#define OLED_RESET     -1 // Reset pin # (or -1 if sharing Arduino reset pin)
Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, OLED_RESET);

int count = 5; 

const unsigned char myBitmap [6] [3072] PROGMEM = 
{
//here are the "myBitmap" code
};

void setup() {
Serial.begin(115200);
delay(2000);
Serial.println(F("Starting!"));
//  ------------------- For i2c -------------------
//  // SSD1306_SWITCHCAPVCC = generate display voltage from 3.3V internally
if(!display.begin(SSD1306_SWITCHCAPVCC, 0x3C)) {
Serial.println(F("SSD1306 allocation failed"));
for(;;); // Don't proceed, loop forever
}
Serial.println(F("Initialized!"));
// Show initial display buffer contents on the screen --

}

void loop() {

display.clearDisplay();
display.drawBitmap(31, 16, myBitmap[count], 64, 48, WHITE); // start point at (31,16)
display.display();
count--;
if(count<0) count = 5; // continuous display
delay(70); //control speed
}
}

Step4. Finally upload the code to XIAO nRF52840 Sense with Arduino and the output should be like:

Make the Toy Active

The toy is assembled by 3D printing components. The original author Kason authorizes the original files and I upload here, and the chart here. However all files are all written by Chinese and so does the assemble tutorial video.

So... good luck! XD

The assembled components(without chart) should be like this:

Embedded ML Voice Control Function - Audio Data Capture

There is a PDM microphone on XIAO nRF52840 Sense, and I am going to use it and Seeed Studio XIAO Expansion board to capture audio then save them as WAV file(Seeed Wiki). Next I will upload them to Edge Impulse as train & test dataset. The dataset are classified as "HuTao", "Shake", and regular environment voice.

Thanks to MJRoBot (Marcelo Rovai), there are more methods to do it.

Step1. Connect the XIAO nRF52840 Sense on the Expansion Board and insert an SD card into the SD card slot at the back.

Step2. Download the Seeed_Arduino_FS Library as a zip file and add it to the Arduino library. Run the code Xiao_mic_Saved_OnSDcard.ino.

For each time running the code there will be recording 5 seconds and writing to a WAV file. So it requires reset button to be pushed every time.

The monitor on the Arduino and files on the SD card should be like:

Embedded ML Voice Control Function - Model Train

Now I am going to train a model base these data by using Edge Impulse.

Step 1. Create a new project on the Edge Impulse website.

Step 2. Select the Data Acquisition section and click the Upload Existing Data tool. Then upload the WAV files which are collected before, like here:

The samples then will show on the Data acquisition section.

Step 3. Click on three dots after the sample name and select Split sample. Split the data into 1-second records (try to avoid noise):

Step 4. After splitting samples, click create impulse on the left column and I can start creating my models.

There are a few things should be considered during the process, like "window increase" here should not be higher than "window size" and "MFCC" processing block is better deal with human voice.

Step 5. Click MFCC(when selected "MFCC" last step) on the left column and pre-process the data here. I keep the default parameters value here and proceed to generate the features.

Step 6. Continue to choose Classifier, and use Convolution Neural Network to build the model. I still keep the default settings.

The model has been built and the model testing shows OK, then I can proceed.

Step 7. Finally, click "Deployment" and select "Arduino library" to build a library based on the data above.

The output library is like:

Embedded ML Voice Control Function - Deploy model on board

Now it is time for the real test.

Step 1. Add the.zip file to the Arduino library, and go to the File/Examples tab then look for the project after the project name.

XIAO nRF52840 Sense share the same chip and PDM with nano ble33, so I can directly use select nano_ble33_sense_microphone_continuous example code to test the model first.

Step 2. After uploading the code to XIAO nRF52840 Sense, click "Serial Monitor" on the top right corner and the test will begin. The result are like:

Assemble Voice Control Function with Toy

After successfully testing the model, now I can connect with my shake toy and voice control it.

Step 1. Use XIAO nRF52840 Sense to control Grove Relay, here I am using A1 port to set as digital output. And use Grove Relay to control the toy motor and the motor is powered by a battery.

Step 2. Add the Grove Relay code into the previous code (nano_ble33_sense_microphone_continuous).

void shake_hutao(int pred_index){
  switch (pred_index){
    case 0: // "hutao"
      digitalWrite(D1, LOW);
      break;

    case 1: // "shake"
      digitalWrite(D1, HIGH);
      break;
    
    case 2: // environment voice
       ei_printf("not voice");
      digitalWrite(D1, LOW);
      break;
  }
}

Step 3. Add the comparison function code in the classification output code.

The added code that needs to be added is marked.

void setup()
{
 pinMode(D1, OUTPUT); //added, output port
...
if (++print_results >= (EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW)) 
{
        // print the predictions
        ei_printf("Predictions ");
        ei_printf("(DSP: %d ms., Classification: %d ms., Anomaly: %d ms.)",
            result.timing.dsp, result.timing.classification, result.timing.anomaly);
        ei_printf(": \n");
        
        int pred_index = 2;     // added, Initialize pred_index
        float pred_value = 0.8;   // added, Initialize pred_value

        
        for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
            ei_printf("    %s: %.5f\n", result.classification[ix].label,
                      result.classification[ix].value);

    // added, comparison function
          if (result.classification[ix].value > pred_value)
        {
               pred_index = ix;
                pred_value = result.classification[ix].value;

                           
        }

          shake_hutao(pred_index); //added, active function
          ei_printf(": \n");  // test
          ei_printf("%d",pred_index);// test
}

Step 4. Complete the code and test the devices if functioning as expected.

Assemble Voice Control Function with OLED Screen

Here I am doing some same steps like above, that I connect XIAO nRF52840 Sense with an OLED screen and a battery then voice control the display.

The code here have some conflicts and sometimes stuck, see "Little Problem" below for more information.

Step 1. The hardware connection should be like this(from Motion Recognition project by Seeed Studio):

Step 2. Generate the image byte arrays with one image under "hutao" voice and the one under regular environment voice, then upload to test with XIAO expansion board.

The left one is the image under "hutao" voice and the right one is the image under regular environment voice. Set the code of left one name as "hutao" and set the right one as "hutao_idle"

Step 3. Add the Grove 0.66 OLED display function into previous code.(nano_ble33_sense_microphone_continuous).

void shake_hutao_display(int pred_index){
  switch (pred_index){
    case 0:  // display when predict "hutao" voice

       display.clearDisplay();
       display.drawBitmap(31, 16, hutao, 64, 48, WHITE); // start point at (31,16) 
       display.display();

      break;

    case 1: // animations with "shake" voice

       display.clearDisplay();
       display.drawBitmap(31, 16, myBitmap[count], 64, 48, WHITE); // start point at (31,16) display whtn predict "shake"
       display.display();
       count--;
       if(count<0) count = 5; // continuous display
       delay(70); //control speed

      break;
    
    case 2: display when predict regular environment voice

       display.clearDisplay();
       display.drawBitmap(31, 16, hutao, 64, 48, WHITE); // start point at (31,16) 
       display.display();

      break;
  }
}

Step 4. Add the comparison function code in the classification output code.

The added The code that needs to be added is marked.

#include <Wire.h> //added
#include <Adafruit_GFX.h> //added
#include <Adafruit_SSD1306.h> //added
#define SCREEN_WIDTH 128 // addedOLED display width, in pixels
#define SCREEN_HEIGHT 64 //added OLED display height, in pixels
// ------------------- For i2c -------------------
//// Declaration for an SSD1306 display connected to I2C (SDA, SCL pins)
#define OLED_RESET     -1 // Reset pin # (or -1 if sharing Arduino reset pin)
Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, OLED_RESET);

int count = 5; //added

void setup()
{

if (++print_results >= (EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW)) 
{
        // print the predictions
        ei_printf("Predictions ");
        ei_printf("(DSP: %d ms., Classification: %d ms., Anomaly: %d ms.)",
            result.timing.dsp, result.timing.classification, result.timing.anomaly);
        ei_printf(": \n");
        
        int pred_index = 2;     // added, Initialize pred_index
        float pred_value = 0.8;   // added, Initialize pred_value

        
        for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
            ei_printf("    %s: %.5f\n", result.classification[ix].label,
                      result.classification[ix].value);

    // added, comparison function
          if (result.classification[ix].value > pred_value)
        {
               pred_index = ix;
                pred_value = result.classification[ix].value;

                           
        }

          shake_hutao_diplay(pred_index); //added, active function
          ei_printf(": \n");  // test
          ei_printf("%d",pred_index);// test
}

Step 4. Complete the code and test the devices if functioning as expected.

The Final(not really...)

Let me put them together and the display like the video here:

Since the detect and predict are fast, I should keep talking the key words "hutao", "shake". And two boards are surprisingly not working at the same time. Perhaps one of them I already used for like a year.

Little Problem(maybe)

Sometimes the code of "Voice Control Function with OLED Screen" may cause some conflicts and stuck. With the code below and assembled in the Edge Impulse model library, it doesn't have any troubles. But the images display with the code are strange. You can always use it to display words, but not images. Hope someone can help me here.

#include <Arduino.h>
#include <U8g2lib.h>
#include <Wire.h>
U8G2_SSD1306_128X64_NONAME_F_HW_I2C u8g2(U8G2_R0, /* clock=*/ D5, /* data=*/ D4, /* reset=*/ U8X8_PIN_NONE);   // All Boards without Reset of the Display
static const uint8_t  PROGMEM mymap0[] = 
{
...
};

void setup()
{
u8g2.begin();
}

void loop() {
u8g2.clearBuffer();
u8g2.drawXBM(0, 0, 64, 48, mymap0); //(x,y,image_width,image_length,image)
u8g2.sendBuffer();
}

Anyway, this is just for the demonstration, showing voice controlling devices like relay, motor, OLED screen, etc.

Hope you guys like it. Thank you!