Team Vistelilabs: Stephanie Vicarte, Elizabeth Vicarte, Irak Mayer

Published April 16, 2020 © GPL3+

MusBall

An interactive touch less instrument interface.

IntermediateFull instructions provided5 hours1,083

Things used in this project

Hardware components

Raspberry Pi 3 Model B

SparkFun ESP32 Thing

VL6180X Time-of-Flight Distance Sensor Carrier with Voltage Regulator, 60cm max

Plastic sphere ball

Story

MusBall

Motivation

Music has always been a big part of life. Without realizing it, we often look to music to express unspoken feelings or to help unwind after a long day. Because of the big influence music has had on us, we wanted to combine our two greatest passions, music and making, into a new project: MusBall. We felt that the idea of an instrument with a touchless-interface was a unique idea that we wanted to explore.The touchless surface also allows multiple people to play the instrument without having to worry about sanitation.

The basic idea is to use TOF sensors to measure the distance from your palms to the sensor and map those values to specific instruments and notes.

Materials

We use a Raspberry Pi 3B+, two Sparkfun ESP32 Thing, four Pololu VL6180 TOF sensors and a plastic ball.

1 / 4 • Raspberry Pi 3B+

ESPNOW

We use the ESPNOW protocol to establish the communication with the instrument to the device that can perform the sound execution (RPi 3B+).

The ESPNow web defines the protocol as

ESP-NOW is yet another protocol developed by Espressif, which enables multiple devices to communicate with one another without using Wi-Fi. The protocol is similar to the low-power 2.4GHz wireless connectivity that is often deployed in wireless mouses. So, the pairing between devices is needed prior to their communication. After the pairing is done, the connection is safe and peer-to-peer, with no handshake being required

The protocol is an easy to use, designed for low range, fast data transmission.

We use one of the Sparkfun ESP32 Thing devices as an AP and the other one as a slave.

Using the example programs installed with the ESP32 board support, ESPNOW\Master.ino and ESPNOW\Slave.ino as templates we developed our version to transmit the sensor data.

We initialize both devices by using the espnow API as follows

esp_now_init()

On the Sparkfun ESP32 Thing device connected to the distance sensors, we scan for the network for available AP

int8_t scanResults = WiFi.scanNetworks();

For each found network we get their SSID and compare it to the AP we are looking to connect into.

String SSID = WiFi.SSID(i);
      int32_t RSSI = WiFi.RSSI(i);
      String BSSIDstr = WiFi.BSSIDstr(i);

      // Check if the current device starts with `Slave`
      if (SSID.indexOf("Slave") == 0) {

Once we found a suitable AP we save its MAC, and channel information so we can connect to them.

The program then registers the AP as a peer for each loop it wants to send data, by executing

const esp_now_peer_info_t *peer = &slaves[i];
      const uint8_t *peer_addr = slaves[i].peer_addr;
      bool exists = esp_now_is_peer_exist(peer_addr);
      if (!exists) {
        esp_err_t addStatus = esp_now_add_peer(peer);
        delay(10);
      }

Then it sends its data package by calling

const uint8_t *peer_addr = slaves[i].peer_addr;
    esp_err_t result = esp_now_send(peer_addr, data, DATASIZE);

Where DATASIZE is the package length in bytes and data is a byte array with the payload.

On the Sparkfun ESP32 Thing device connected to the RPi 3B+, we start an AP on the slave of the device that connects to the RPi 3B+ as

char* SSID = "Slave_1";
  bool result = WiFi.softAP(SSID, "Slave_1_Password", CHANNEL, 0);

Registering the callback function to receive data from the master.

esp_now_register_recv_cb(OnDataRecv);

And the callback function forwards the received data to the serial port.

void OnDataRecv(const uint8_t *mac_addr, const uint8_t *data, int data_len) {
  char macStr[18];
  //Split mac address
  snprintf(macStr, sizeof(macStr), "%02x:%02x:%02x:%02x:%02x:%02x",
           mac_addr[0], mac_addr[1], mac_addr[2], mac_addr[3], mac_addr[4], mac_addr[5]);

  //Forward each byte to serial port, replace the data delimiter with a carriage return.
  for (int i=0;i<data_len;i++)
  {
    Serial2.write((data[i]));
    if (data[i] == '|')
    {
      Serial2.write('\n');
    }
  }

}

The function breaks the received package and adds a carriage return after the divider character "|" on the data set for easy consumption on the Python program.

TOF sensor array

The data that envelopes the payload is a package formatted with the distance taken from the four TOF sensors located on the ball separated by a 90 degree metric.

Setting the I2C addresses on the Pololu TOF sensors was not a trivial task. We have to do a bit of research in order to assign the new addresses to the devices.

Create independent global variables for each sensor available.

VL6180X sensor;
VL6180X sensor1;
VL6180X sensor2;
VL6180X sensor3;

First all the sensors start on an idle state, you turn them on by sending a HIGH setting to the GPIO0/CE of the sensor connected to a digital pin (in the example below pin 12). Then initialize the sensor, configure to the default settings, set some basic parameters and finally set the new address (in the example below 0x31).

In code it translates to:

//Initilize sensor and assign address.
  digitalWrite(12, HIGH);
  sensor1.init();
  sensor1.configureDefault();
  sensor1.setScaling(SCALING);
  sensor1.writeReg(VL6180X::SYSRANGE__MAX_CONVERGENCE_TIME, 30);
  sensor1.writeReg16Bit(VL6180X::SYSALS__INTEGRATION_PERIOD, 50);  
  sensor1.setTimeout(500);
  sensor1.setAddress(0x31);

Repeat these same steps in that order for each of the other connected sensors.

Once the connections have being established start collecting data by reading the sensors output.

dLeft  = sensor.readRangeContinuousMillimeters();
  dFront = sensor1.readRangeContinuousMillimeters();
  dBack  = sensor2.readRangeContinuousMillimeters();
  dRight = sensor3.readRangeContinuousMillimeters();

And encode the data into a package

//Format data into one package to be sent.
  sprintf((char *)myData,"%05d,%05d,%05d,%05d,2|",dLeft,dFront,dBack,dRight);

Note the package delimiter "|". The last 2 indicates the ball id in case there is more than one ball sending data to the connected AP.

Python service

On the Raspberry Pi 3B+ we create a program than runs on the background reading from the serial port, mapping the data into one of the loaded instrument nodes.

To play our notes we use pygame mixer libraries

#Initialize the pygame sound engine
pygame.mixer.init()

The program opens the serial port to read

#Initialize the serial port for communication with the Matrix Voice ESP32
ser = serial.Serial(
        port='/dev/ttyS0',
        baudrate = 115200,
        parity=serial.PARITY_NONE,
        stopbits=serial.STOPBITS_ONE,
        bytesize=serial.EIGHTBITS,
        timeout=300
)

Playing the notes, we took a similar approach to our previous project GloMusic. The following are parts of that page explanation.

We request 15 Channels to play our instrument sounds simultaneously.

# Set the number of channels to allow multiple simultaneous sounds
pygame.mixer.set_num_channels(15)

We took a logistic approach to handle the resources as easily as possible, by creating an array of file paths and an array that would create the sound objects so we would not need to load them every time we want to play. A snip of the chorus object is shown below. Notice that the chsndObj array contains all the sound objects for the chorus after loading the corresponding file.

chCEGmfPath = ['/home/pi/MusicOrch/Chorus/chorus-female-c5-PB-loop.wav',
            '/home/pi/MusicOrch/Chorus/chorus-female-e5-PB-loop.wav',
            '/home/pi/MusicOrch/Chorus/chorus-female-g5-PB-loop.wav']
# Create an array of Sound objects for each instrumment. This is the array for chorus.
# Chorus
chsndObj = [pygame.mixer.Sound(chADFmfPath[0]), pygame.mixer.Sound(chADFmfPath[1]),
         pygame.mixer.Sound(chADFmfPath[2]),
         pygame.mixer.Sound(chCEGMmfPath[0]), pygame.mixer.Sound(chCEGMmfPath[1]),
         pygame.mixer.Sound(chCEGMmfPath[2]),
         pygame.mixer.Sound(chCEGmfPath[0]), pygame.mixer.Sound(chCEGmfPath[1]),
         pygame.mixer.Sound(chCEGmfPath[2])]

The selectSoundObj and playNote functions takes the task to select the correct sound according to the active instrument and play on the independent channel.

Finally, the data decoding is envelope in

#Read a line of data from the ESP32 device
x=ser.readline()
#Split elements into data.
group = re.split(r'\|',x.decode('ascii'))
singleData = re.split(r',+',group[0])
singleData[0] = singleData[0][1:]
singleData2 = re.split(r',+',group[1])

We read a line from the serial stream and split the package by using the delimiter "|", and further more by the ", " delimiter.

The data is then mapped to the available array elements, and the corresponding notes are played.

Results

Although sound was possible, the creation of a melody still requires some more work.

We are looking forward to see what other projects are inspired by this work.

Thanks for reading through.

Code

//This is a modified version of the sample code that comes with the ESP32 board.
//Vistelilabs. 2019
//This program reads the distance data and packs them together to be sent thru the ESPNOW protocol.

#include <Wire.h>
#include <VL6180X.h>
#include <esp_now.h>
#include <WiFi.h>

VL6180X sensor;
VL6180X sensor1;
VL6180X sensor2;
VL6180X sensor3;

#define NUMSLAVES 1
esp_now_peer_info_t slaves[NUMSLAVES] = {};
int SlaveCnt = 0;

#define SCALING 1

#define CHANNEL 3
#define PRINTSCANRESULTS 0

#define PIN            13
#define NUMPIXELS      7
#define BRIGHTNESS 250

// Init ESP Now with fallback
void InitESPNow() {
  WiFi.disconnect();
  if (esp_now_init() == ESP_OK) {
    Serial.println("ESPNow Init Success");
  }
  else {
    Serial.println("ESPNow Init Failed");
    ESP.restart();
  }
}

// Scan for slaves in AP mode
void ScanForSlave() {
  int8_t scanResults = WiFi.scanNetworks();
  //reset slaves
  memset(slaves, 0, sizeof(slaves));
  SlaveCnt = 0;
  Serial.println("");
  if (scanResults == 0) {
    Serial.println("No WiFi devices in AP Mode found");
  } else {
    Serial.print("Found "); Serial.print(scanResults); Serial.println(" devices ");
    for (int i = 0; i < scanResults; ++i) {
      // Print SSID and RSSI for each device found
      String SSID = WiFi.SSID(i);
      int32_t RSSI = WiFi.RSSI(i);
      String BSSIDstr = WiFi.BSSIDstr(i);

      if (PRINTSCANRESULTS) {
        Serial.print(i + 1); Serial.print(": "); Serial.print(SSID); Serial.print(" ["); Serial.print(BSSIDstr); Serial.print("]"); Serial.print(" ("); Serial.print(RSSI); Serial.print(")"); Serial.println("");
      }
      delay(10);
      // Check if the current device starts with `Slave`
      if (SSID.indexOf("Slave") == 0) {
        // SSID of interest
        Serial.print(i + 1); Serial.print(": "); Serial.print(SSID); Serial.print(" ["); Serial.print(BSSIDstr); Serial.print("]"); Serial.print(" ("); Serial.print(RSSI); Serial.print(")"); Serial.println("");
        // Get BSSID => Mac Address of the Slave
        int mac[6];

        if ( 6 == sscanf(BSSIDstr.c_str(), "%x:%x:%x:%x:%x:%x%c",  &mac[0], &mac[1], &mac[2], &mac[3], &mac[4], &mac[5] ) ) {
          for (int ii = 0; ii < 6; ++ii ) {
            slaves[SlaveCnt].peer_addr[ii] = (uint8_t) mac[ii];
          }
        }
        slaves[SlaveCnt].channel = CHANNEL; // pick a channel
        slaves[SlaveCnt].encrypt = 0; // no encryption
        SlaveCnt++;
      }
    }
  }

  if (SlaveCnt > 0) {
    Serial.print(SlaveCnt); Serial.println(" Slave(s) found, processing..");
  } else {
    Serial.println("No Slave Found, trying again.");
  }

  // clean up ram
  WiFi.scanDelete();
}

// Check if the slave is already paired with the master.
// If not, pair the slave with master
void manageSlave() {
  if (SlaveCnt > 0) {
    for (int i = 0; i < SlaveCnt; i++) {
      const esp_now_peer_info_t *peer = &slaves[i];
      const uint8_t *peer_addr = slaves[i].peer_addr;
      bool exists = esp_now_is_peer_exist(peer_addr);
      if (!exists) {
        esp_err_t addStatus = esp_now_add_peer(peer);
        delay(10);
      }
    }
  } else {
    // No slave found to process
    Serial.println("No Slave found to process");
  }
}

//Number of bytes in the package
#define DATASIZE 27
uint8_t data[DATASIZE];
uint64_t pos=0;

// send data
void sendData(char *inData) {
  pos++;
  sprintf((char *)data,"%s",inData);
  for (int i = 0; i < SlaveCnt; i++) {
    const uint8_t *peer_addr = slaves[i].peer_addr;
    esp_err_t result = esp_now_send(peer_addr, data, DATASIZE); //sizeof(data));
    delay(10);
  }
}

// callback when data is sent from Master to Slave
void OnDataSent(const uint8_t *mac_addr, esp_now_send_status_t status) {
  char macStr[18];
  snprintf(macStr, sizeof(macStr), "%02x:%02x:%02x:%02x:%02x:%02x",
           mac_addr[0], mac_addr[1], mac_addr[2], mac_addr[3], mac_addr[4], mac_addr[5]);
}

void setup()
{
  Serial.begin(115200);
  Wire.begin();

  //Set device in STA mode to begin with
  WiFi.mode(WIFI_STA);
  Serial.println("ESPNow/Multi-Slave/Master Example");
  // This is the mac address of the Master in Station Mode
  Serial.print("STA MAC: "); Serial.println(WiFi.macAddress());
  // Init ESPNow with a fallback logic
  InitESPNow();
  // Once ESPNow is successfully Init, we will register for Send CB to
  // get the status of Trasnmitted packet
  esp_now_register_send_cb(OnDataSent);

  // In the loop we scan for slave
  while (SlaveCnt < 1)
  {
    ScanForSlave();
    delay(20);
  }

  pinMode(12, OUTPUT);
  pinMode(13, OUTPUT);
  pinMode(27, OUTPUT);
  pinMode(33, OUTPUT);

  //Initilize sensor and assign address.
  digitalWrite(12, HIGH);
  sensor1.init();
  sensor1.configureDefault();
  sensor1.setScaling(SCALING);
  sensor1.writeReg(VL6180X::SYSRANGE__MAX_CONVERGENCE_TIME, 30);
  sensor1.writeReg16Bit(VL6180X::SYSALS__INTEGRATION_PERIOD, 50);  
  sensor1.setTimeout(500);
  sensor1.setAddress(0x31);
  
  //Initilize sensor and assign address.
  digitalWrite(13, HIGH);
  sensor.init();
  sensor.configureDefault();
  sensor.setScaling(SCALING);
  sensor.writeReg(VL6180X::SYSRANGE__MAX_CONVERGENCE_TIME, 30);
  sensor.writeReg16Bit(VL6180X::SYSALS__INTEGRATION_PERIOD, 50);  
  sensor.setTimeout(500);
  sensor.setAddress(0x30);

  //Initilize sensor and assign address.
  digitalWrite(27, HIGH);
  sensor2.init();
  sensor2.configureDefault();
  sensor2.setScaling(SCALING);
  sensor2.writeReg(VL6180X::SYSRANGE__MAX_CONVERGENCE_TIME, 30);
  sensor2.writeReg16Bit(VL6180X::SYSALS__INTEGRATION_PERIOD, 50);  
  sensor2.setTimeout(500);
  sensor2.setAddress(0x32);

  //Initilize sensor and assign address.
  digitalWrite(33, HIGH);
  sensor3.init();
  sensor3.configureDefault();
  sensor3.setScaling(SCALING);
  sensor3.writeReg(VL6180X::SYSRANGE__MAX_CONVERGENCE_TIME, 30);
  sensor3.writeReg16Bit(VL6180X::SYSALS__INTEGRATION_PERIOD, 50);  
  sensor3.setTimeout(500);
  sensor3.setAddress(0x33);

  // Start continuous back-to-back mode (take readings as
  // fast as possible).  To use continuous timed mode
  // instead, provide a desired inter-measurement period in
  // ms (e.g. sensor.startContinuous(100)).
  sensor.startInterleavedContinuous(100);
  sensor1.startInterleavedContinuous(100);
  sensor2.startInterleavedContinuous(100);
  sensor3.startInterleavedContinuous(100);

  Serial.print("End Setup");

}

void loop()
{
  int dLeft,dFront,dBack,dRight,dUp;
  char myData[DATASIZE+2];

  //Read sensor data
  dLeft  = sensor.readRangeContinuousMillimeters();
  dFront = sensor1.readRangeContinuousMillimeters();
  dBack  = sensor2.readRangeContinuousMillimeters();
  dRight = sensor3.readRangeContinuousMillimeters();

  //Format data into one package to be sent.
  sprintf((char *)myData,"%05d,%05d,%05d,%05d,2|",dLeft,dFront,dBack,dRight);
  
  if (sensor.timeoutOccurred())  { Serial.print("TIMEOUT"); }
  if (sensor1.timeoutOccurred()) { Serial.print(" TIMEOUT"); }
  if (sensor2.timeoutOccurred()) { Serial.print(" TIMEOUT"); }
  if (sensor3.timeoutOccurred()) { Serial.print(" TIMEOUT"); }

  if (SlaveCnt > 0) { // check if slave channel is defined
    manageSlave();
    sendData(myData);
  }

  delay(10);
}

//This is a modified version of the sample code that comes with the ESP32 board.
//Vistelilabs. 2019
//This program forwards the received data to the serial port.
#include <esp_now.h>
#include <WiFi.h>

#define CHANNEL 1

// Init ESP Now with fallback
void InitESPNow() {
  WiFi.disconnect();
  if (esp_now_init() == ESP_OK) {
    Serial.println("ESPNow Init Success");
  }
  else {
    Serial.println("ESPNow Init Failed");
    ESP.restart();
  }
}

// config AP SSID
void configDeviceAP() {
  char* SSID = "Slave_1";
  bool result = WiFi.softAP(SSID, "Slave_1_Password", CHANNEL, 0);
  if (!result) {
    Serial.println("AP Config failed.");
  } else {
    Serial.println("AP Config Success. Broadcasting with AP: " + String(SSID));
  }
}

void setup() {
  Serial.begin(115200);
  Serial.println("ESPNow/Basic/Slave Example");
  //Set device in AP mode to begin with
  WiFi.mode(WIFI_AP);
  // configure device AP mode
  configDeviceAP();
  // This is the mac address of the Slave in AP Mode
  Serial.print("AP MAC: "); Serial.println(WiFi.softAPmacAddress());
  // Init ESPNow with a fallback logic
  InitESPNow();
  // Once ESPNow is successfully Init, we will register for recv CB to
  // get recv packer info.
  esp_now_register_recv_cb(OnDataRecv);

  //Initialize port to forward received data.
  Serial2.begin(115200,SERIAL_8N1, 16, 17);

}

// callback when data is recv from Master
void OnDataRecv(const uint8_t *mac_addr, const uint8_t *data, int data_len) {
  char macStr[18];
  //Split mac address
  snprintf(macStr, sizeof(macStr), "%02x:%02x:%02x:%02x:%02x:%02x",
           mac_addr[0], mac_addr[1], mac_addr[2], mac_addr[3], mac_addr[4], mac_addr[5]);

  //Forward each byte to serial port, replace the data delimiter with a carriage return.
  for (int i=0;i<data_len;i++)
  {
    Serial2.write((data[i]));
    if (data[i] == '|')
    {
      Serial2.write('\n');
    }
  }

  //Print data for DEBUG purposes
  Serial.print((char *)data);
  Serial.println("");
}

void loop() {
  // Chill
}

#!/usr/bin/env python

#MusBall.py
#4/01/2020
#Vistelilabs (www.vistelilabs.com)
#This program reads from the Serial port data coming from the ESPNOW device and plays the mapped value with the supported instrument
# Copyright <2020> <VisteliLabs>
#Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
#The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


import time
import serial
import re
import pygame
import time

#Initialize the pygame sound engine
pygame.mixer.init()
#Set the number of channels to allow multiple simultaneous sounds
pygame.mixer.set_num_channels(15)

#Initialize the file array for the instruments. The array format is intrument (v violin, ...), scale (A B C D ...), music file (mf), Path
#Cello
cCmfPath = ['/home/pi/MusicOrch/Cello/c2-stac-rr2-PB.wav',
            '/home/pi/MusicOrch/Cello/c4-stac-rr2-PB.wav',
            '/home/pi/MusicOrch/Cello/c5-stac-rr2-PB.wav']

cEmfPath = ['/home/pi/MusicOrch/Cello/e2-stac-rr2-PB.wav',
            '/home/pi/MusicOrch/Cello/e4-stac-rr2-PB.wav',
            '/home/pi/MusicOrch/Cello/e5-stac-rr2-PB.wav']

cGmfPath = ['/home/pi/MusicOrch/Cello/g2-stac-rr2-PB.wav',
            '/home/pi/MusicOrch/Cello/g3-stac-rr2-PB.wav',
            '/home/pi/MusicOrch/Cello/g4-stac-rr2-PB.wav']

#Create an array of Sound objects for each instrumment. This is the array for chorus.
csndObj = [pygame.mixer.Sound(cCmfPath[0]), pygame.mixer.Sound(cCmfPath[1]),
           pygame.mixer.Sound(cCmfPath[2]),
           pygame.mixer.Sound(cEmfPath[0]), pygame.mixer.Sound(cEmfPath[1]),
           pygame.mixer.Sound(cEmfPath[2]),
           pygame.mixer.Sound(cGmfPath[0]), pygame.mixer.Sound(cGmfPath[1]),
           pygame.mixer.Sound(cGmfPath[2])]

#Violin
vAmfPath = ['/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_A2_v2_rr1-PB.wav',
            '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_A3_v2_rr1-PB.wav',
            '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_A4_v2_rr1-PB.wav',
            '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_A5_v2_rr1-PB.wav']

vCmfPath = ['/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_C3_v2_rr1-PB.wav',
            '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_C4_v2_rr1-PB.wav',
            '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_C5_v2_rr1-PB.wav',
            '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_C6_v2_rr1-PB.wav']

vEmfPath = ['/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_E3_v2_rr1-PB.wav',
           '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_E4_v2_rr1-PB.wav',
           '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_E5_v1_rr1-PB.wav',
           '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_E5_v2_rr1-PB.wav']

vGmfPath = ['/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_G2_v2_rr1-PB.wav',
           '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_G3_v2_rr1-PB.wav',
           '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_G4_v2_rr1-PB.wav',
           '/home/pi/MusicOrch/Solo Violin/spic/LLVln_spic_G5_v2_rr1-PB.wav']

#Create an array of Sound objects for each instrumment. This is the array for chorus.
vsndObj = [pygame.mixer.Sound(vAmfPath[0]),pygame.mixer.Sound(vAmfPath[1]),
          pygame.mixer.Sound(vAmfPath[2]),pygame.mixer.Sound(vAmfPath[3]),
          pygame.mixer.Sound(vCmfPath[0]),pygame.mixer.Sound(vCmfPath[1]),
          pygame.mixer.Sound(vCmfPath[2]),pygame.mixer.Sound(vCmfPath[3]),
          pygame.mixer.Sound(vEmfPath[0]),pygame.mixer.Sound(vEmfPath[1]),
          pygame.mixer.Sound(vEmfPath[2]),pygame.mixer.Sound(vEmfPath[3]),
          pygame.mixer.Sound(vGmfPath[0]),pygame.mixer.Sound(vGmfPath[1]),
          pygame.mixer.Sound(vGmfPath[2]),pygame.mixer.Sound(vGmfPath[3])]

#Flute
fAmfPath = ['/home/pi/MusicOrch/Flute/stac-PB/LDFlute_stac_A3_v2_rr1-PB.wav',
            '/home/pi/MusicOrch/Flute/stac-PB/LDFlute_stac_A4_v3_rr2-PB.wav',
            '/home/pi/MusicOrch/Flute/stac-PB/LDFlute_stac_A5_v1_rr1-PB.wav']
            
fCmfPath = ['/home/pi/MusicOrch/Flute/stac-PB/LDFlute_stac_C4_v4_rr1-PB.wav',
            '/home/pi/MusicOrch/Flute/stac-PB/LDFlute_stac_C5_v1_rr1-PB.wav',
            '/home/pi/MusicOrch/Flute/stac-PB/LDFlute_stac_C6_v2_rr2-PB.wav']
            
fEmfPath = ['/home/pi/MusicOrch/Flute/stac-PB/LDFlute_stac_E3-PB.wav',
            '/home/pi/MusicOrch/Flute/stac-PB/LDFlute_stac_E4_v2_rr1-PB.wav',
            '/home/pi/MusicOrch/Flute/stac-PB/LDFlute_stac_E5_v1_rr2-PB.wav']
            
#Create an array of Sound objects for each instrumment. This is the array for chorus.
fsndObj = [pygame.mixer.Sound(fAmfPath[0]),pygame.mixer.Sound(fAmfPath[1]),
          pygame.mixer.Sound(fAmfPath[2]),
          pygame.mixer.Sound(fCmfPath[0]),pygame.mixer.Sound(fCmfPath[1]),
          pygame.mixer.Sound(fCmfPath[2]),
          pygame.mixer.Sound(fEmfPath[0]),pygame.mixer.Sound(fEmfPath[1]),
          pygame.mixer.Sound(fEmfPath[2])]

#Initialize the serial port for communication with the Matrix Voice ESP32
ser = serial.Serial(
        port='/dev/ttyS0',
        baudrate = 115200,
        parity=serial.PARITY_NONE,
        stopbits=serial.STOPBITS_ONE,
        bytesize=serial.EIGHTBITS,
        timeout=300
)

PauseExec = False

#Define the instrument constants
ViolinIns = 'A'
ChorusIns = 'B'
CelloIns = 'C'
FluteIns = 'D'
PianoIns = 'E'
MixOneIns = 'F'
TimpaniIns = 'G'
ExitIns = 'X'

#Select the instrument to play
activeInstrument = MixOneIns 
#activeInstrument = FluteIns 
#activeInstrument = ViolinIns

#Normalize attributes
baseNote = 40
anchorVal = 3
threshold = 120

#Function that selects the sound file according to the pressure value and scale index
#returns the Sound object
def selectSoundObj(objArray,indexId,noteValue):
    offsetVal = int(noteValue / baseNote)
    return (objArray[indexId*anchorVal+offsetVal])

#Function that plays the sound in the corresponding channel
#Parameters are the select instruments, and the pressure values from each sensor
def playNote(selIns,note1,note2,note3,note4=0):
    global vsndObj
    global fsndObj
    if (selIns == ViolinIns):
        #print('Violin')
        if (note1 > 0):
            pygame.mixer.Channel(0).play(selectSoundObj(vsndObj,0,note1))
        if (note2 > 0):
            pygame.mixer.Channel(1).play(selectSoundObj(vsndObj,1,note2))
        if (note3 > 0):
            pygame.mixer.Channel(2).play(selectSoundObj(vsndObj,2,note3))
        if (note4 > 0):
            pygame.mixer.Channel(3).play(selectSoundObj(vsndObj,3,note4))
    if (selIns == FluteIns):
        #print('Flute')
        if (note1 > 0):
            pygame.mixer.Channel(3).play(selectSoundObj(fsndObj,0,note1))
        if (note2 > 0):
            pygame.mixer.Channel(4).play(selectSoundObj(fsndObj,1,note2))
        if (note3 > 0):
            pygame.mixer.Channel(5).play(selectSoundObj(fsndObj,2,note3))
            
    if (selIns == CelloIns):
        #print('Cello')
        if (note1 > 0):
            pygame.mixer.Channel(6).play(selectSoundObj(csndObj,0,note1))
        if (note2 > 0):
            pygame.mixer.Channel(7).play(selectSoundObj(csndObj,1,note2))
        if (note3 > 0):
            pygame.mixer.Channel(8).play(selectSoundObj(csndObj,2,note3))

    if (selIns == ChorusIns):
        #print('Chorus')
        if (note1 > 0):
            pygame.mixer.Channel(9).play(selectSoundObj(chsndObj,0,note1),maxtime=4000)
        if (note2 > 0):
            pygame.mixer.Channel(10).play(selectSoundObj(chsndObj,1,note2),maxtime=4000)
        if (note3 > 0):
            pygame.mixer.Channel(11).play(selectSoundObj(chsndObj,2,note3),maxtime=4000)
            
    if (selIns == TimpaniIns):
        #print('Timpani')
        if (note1 > 0):
            pygame.mixer.Channel(12).play(selectSoundObj(tsndObj,0,note1))
        if (note2 > 0):
            pygame.mixer.Channel(13).play(selectSoundObj(tsndObj,1,note2))
        if (note3 > 0):
            pygame.mixer.Channel(14).play(selectSoundObj(tsndObj,2,note3))

    if (selIns == MixOneIns):
        #print('MixOne')
        if (note1 > 0):
            pygame.mixer.Channel(0).play(selectSoundObj(vsndObj,0,note1),fade_ms=50)
            pygame.mixer.Channel(3).play(selectSoundObj(fsndObj,0,note1),fade_ms=60)            
            pygame.mixer.Channel(6).play(selectSoundObj(csndObj,0,note1))
        if (note2 > 0):
            pygame.mixer.Channel(1).play(selectSoundObj(vsndObj,1,note2),fade_ms=50)
            pygame.mixer.Channel(4).play(selectSoundObj(fsndObj,1,note2))
            pygame.mixer.Channel(7).play(selectSoundObj(csndObj,1,note2),fade_ms=60)
        if (note3 > 0):
            pygame.mixer.Channel(2).play(selectSoundObj(vsndObj,2,note3))
            pygame.mixer.Channel(5).play(selectSoundObj(fsndObj,2,note3),fade_ms=50)            
            pygame.mixer.Channel(8).play(selectSoundObj(csndObj,2,note3),fade_ms=60)

#main loop
counter=0;
while 1:
        #Read a line of data from the ESP32 device
        x=ser.readline()
        #Split elements into data.
        group = re.split(r'\|',x.decode('ascii'))
        singleData = re.split(r',+',group[0])
        singleData[0] = singleData[0][1:]
        singleData2 = re.split(r',+',group[1])
        #print(singleData)
        if (counter > 2):
            data = []
            #print(singleData)
            #Process data
            if (int(singleData[3]) < 500):
                data1 = int(singleData[3])%128
                data1 = int(singleData[3]) % threshold
            if (int(singleData[1]) < 500):
                data2 = (int(singleData[1])%64)+50
                data2=int(singleData[1]) % threshold
            if (int(singleData[0]) < 500):
                data3 = (int(singleData[0])%32)+50
                data3=int(singleData[0]) % threshold
            if (int(singleData[2]) < 500):
                data4 = (int(singleData[2])%16)+50
                data4 = int(singleData[2]) % threshold
            if (int(singleData[3]) == 255):
                    data1 = 0;
            if (int(singleData[2]) == 255):
                    data4 = 0 ;
            if (int(singleData[1]) == 255):
                    data2 = 0;
            #print(singleData[0])
            if (int(singleData[0]) == 255):
                    data3 = 0;
            #play note
            if (data1 != 0 or data2 != 0 or data3 != 0 or data4 != 0):
                print(data1,data2,data3,data4)
                playNote(activeInstrument,data1,data2,data3,data4)

            
            counter = 0;
        counter=counter+1

#On exit close the sound engine.
pygame.mixer.quit()

MusBall

Things used in this project

Hardware components

Story

Motivation

Materials

ESPNOW

TOF sensor array

Python service

Results

Schematics

Distance array sensor diagram

Code

MusicBallSense60.ino

MusicBallSerialSlave.ino

MusBall.py

Credits

Stephanie Vicarte

Elizabeth Vicarte

Irak Mayer

Comments

Embed the widget on your own site

MusBall

MusBall

Things used in this project

Hardware components

Story

Motivation

Materials

ESPNOW

TOF sensor array

Python service

Results

Schematics

Distance array sensor diagram

Code

MusicBallSense60.ino

MusicBallSerialSlave.ino

MusBall.py

Credits

Stephanie Vicarte

Elizabeth Vicarte

Irak Mayer

Comments

Related channels and tags