Audio Hacking on the ESP8266

The ESP8266 is quite powerful for audio applications with a CPU frequency of 160MHz and 4MB flash.

The goal for this tutorial is to build synthesizers on the ESP8266 platform so we will also go through adding MIDI inputs.

The PDM DAC can also be used for webradio and other audio streaming applications.

We will use the Wemos D1 Mini board with the Arduino IDE.

ESP8266 i2s interface

The ESP8266 handles audio through something called i2s.i2s is high speed shifting out of 2 16-bit serial words, left and right channel,  and a shift clock powered by DMA.

This interface normally requires an external i2s DAC that converts the serial stream to analog signals.

To make it more easy we are going to build a PDM (Pulse Density Modulation) DAC based on the i2s interface.

PDM is a high rate bitstream and at 44.1KHz sample rate it will be 32 times higher or about 1.4MHz.

Pulse Density Modulation being a 1-bit DAC gives us a dynamic range of 6dB.That will generate ALOT of noise or 90dB to be exact.

The good thing is that the noise is in a frequency range far above the audio spectrum and can easily be filtered off with a lowpass filter leaving us just the audio signal.

So delta-sigma coding our 16-bit sample words to PDM will give us one 16-bit DAC output with only an external passive filter.

This is the schematics for the audio output:

But why is it connected to the RX pin?Isn’t that the serial input pin?

It’s also the i2s data output pin.

Lets show some code

This is the setup() for our first test.

It turns off the WiFi radio to reduce power to about 15mA and setting up the pins and DMA for the i2s subsystem at a 44100Hz sample rate:

#include "Arduino.h"
#include "ESP8266WiFi.h"
#include "i2s.h"
#include "i2s_reg.h"
void setup() {
  //WiFi.forceSleepBegin();
  //delay(1);
  system_update_cpu_freq(160);
  i2s_begin();
  i2s_set_rate(44100);
}

Here is the writeDAC function that outputs the samples:

void writeDAC(uint16_t DAC) {
 for (uint8_t i=0;i<32;i++) { 
  i2sACC=i2sACC<<1;
  if(DAC >= err) {
    i2sACC|=1;
    err += 0xFFFF-DAC;
  }
    else
  {
    err -= DAC;
  }
 }
 bool flag=i2s_write_sample(i2sACC);
}

To test the DAC we generate a slow sine wave:

uint8_t phase;
  void loop() {
  writeDAC(0x8000+sine[phase++]);
}

And the sinewave data:

int16_t sine[256] = {
  0x0000, 0x0324, 0x0647, 0x096a, 0x0c8b, 0x0fab, 0x12c8, 0x15e2,
  0x18f8, 0x1c0b, 0x1f19, 0x2223, 0x2528, 0x2826, 0x2b1f, 0x2e11,
  0x30fb, 0x33de, 0x36ba, 0x398c, 0x3c56, 0x3f17, 0x41ce, 0x447a,
  0x471c, 0x49b4, 0x4c3f, 0x4ebf, 0x5133, 0x539b, 0x55f5, 0x5842,
  0x5a82, 0x5cb4, 0x5ed7, 0x60ec, 0x62f2, 0x64e8, 0x66cf, 0x68a6,
  0x6a6d, 0x6c24, 0x6dca, 0x6f5f, 0x70e2, 0x7255, 0x73b5, 0x7504,
  0x7641, 0x776c, 0x7884, 0x798a, 0x7a7d, 0x7b5d, 0x7c29, 0x7ce3,
  0x7d8a, 0x7e1d, 0x7e9d, 0x7f09, 0x7f62, 0x7fa7, 0x7fd8, 0x7ff6,
  0x7fff, 0x7ff6, 0x7fd8, 0x7fa7, 0x7f62, 0x7f09, 0x7e9d, 0x7e1d,
  0x7d8a, 0x7ce3, 0x7c29, 0x7b5d, 0x7a7d, 0x798a, 0x7884, 0x776c,
  0x7641, 0x7504, 0x73b5, 0x7255, 0x70e2, 0x6f5f, 0x6dca, 0x6c24,
  0x6a6d, 0x68a6, 0x66cf, 0x64e8, 0x62f2, 0x60ec, 0x5ed7, 0x5cb4,
  0x5a82, 0x5842, 0x55f5, 0x539b, 0x5133, 0x4ebf, 0x4c3f, 0x49b4,
  0x471c, 0x447a, 0x41ce, 0x3f17, 0x3c56, 0x398c, 0x36ba, 0x33de,
  0x30fb, 0x2e11, 0x2b1f, 0x2826, 0x2528, 0x2223, 0x1f19, 0x1c0b,
  0x18f8, 0x15e2, 0x12c8, 0x0fab, 0x0c8b, 0x096a, 0x0647, 0x0324,
  0x0000, 0xfcdc, 0xf9b9, 0xf696, 0xf375, 0xf055, 0xed38, 0xea1e,
  0xe708, 0xe3f5, 0xe0e7, 0xdddd, 0xdad8, 0xd7da, 0xd4e1, 0xd1ef,
  0xcf05, 0xcc22, 0xc946, 0xc674, 0xc3aa, 0xc0e9, 0xbe32, 0xbb86,
  0xb8e4, 0xb64c, 0xb3c1, 0xb141, 0xaecd, 0xac65, 0xaa0b, 0xa7be,
  0xa57e, 0xa34c, 0xa129, 0x9f14, 0x9d0e, 0x9b18, 0x9931, 0x975a,
  0x9593, 0x93dc, 0x9236, 0x90a1, 0x8f1e, 0x8dab, 0x8c4b, 0x8afc,
  0x89bf, 0x8894, 0x877c, 0x8676, 0x8583, 0x84a3, 0x83d7, 0x831d,
  0x8276, 0x81e3, 0x8163, 0x80f7, 0x809e, 0x8059, 0x8028, 0x800a,
  0x8000, 0x800a, 0x8028, 0x8059, 0x809e, 0x80f7, 0x8163, 0x81e3,
  0x8276, 0x831d, 0x83d7, 0x84a3, 0x8583, 0x8676, 0x877c, 0x8894,
  0x89bf, 0x8afc, 0x8c4b, 0x8dab, 0x8f1e, 0x90a1, 0x9236, 0x93dc,
  0x9593, 0x975a, 0x9931, 0x9b18, 0x9d0e, 0x9f14, 0xa129, 0xa34c,
  0xa57e, 0xa7be, 0xaa0b, 0xac65, 0xaecd, 0xb141, 0xb3c1, 0xb64c,
  0xb8e4, 0xbb86, 0xbe32, 0xc0e9, 0xc3aa, 0xc674, 0xc946, 0xcc22,
  0xcf05, 0xd1ef, 0xd4e1, 0xd7da, 0xdad8, 0xdddd, 0xe0e7, 0xe3f5,
  0xe708, 0xea1e, 0xed38, 0xf055, 0xf375, 0xf696, 0xf9b9, 0xfcdc
};

And the resulting waveform output, a sine wave at 172Hz:

Two important things to keep in mind:

The esp8266 is a RTOS system and other things happen in the background.So don’t use delay() or other blocking functions.Use yield() if something takes a long time.

The DMA buffer is 512 samples long and will exhaust in 11.5mSTo have uninterrupted audio output you need to feed it samples before it exhaust.

Feel free to try and get this running and I’ll be back with a sample player.

(* Someone claimed that feeding the PDM algorithm a flatline 0x0001 DAC value will make it fail with a 22Hz hum. While that is true it is an extreme case that seldom happens. Normal flatline is 0x8000 which produces a 50/50 squarewave at 700KHz and the occasional DAC values that dip into values below 1000 wont last long so its not a problem in real world wave data.)

(* The PDM bitrate runs at 1.4MHz. To make it run at the more professional bitrate of 3MHz just bump the sample rate up from 44.1KHz to 96KHz.)

A simple 909 drum synth

Using our sampling knowledge we are going to do a simple 909 drum sample player.

The sample player is a 11-voice fully polyphonic 44.1KHz 16-bit 1-shot wave player.

For this we need about 300Kbyte worth of 44.1KHz drum samples.

const uint16_t BD16[3796] PROGMEM = {
40, 85, 137, 144, -30, -347, -609, -785, // 0-7
const uint16_t CP16[4445] PROGMEM = {
-42, 74, -1236, -2741, -3134, -11950, -13578, -7572, // 0-7

The definitions above are only a small sample of the 16-bit wave data. The full arrays of the 11 sounds are included in the downloadable sketch.

We also need some declares for the sample engine.

uint32_t BD16CNT;
uint32_t CP16CNT;
uint32_t CR16CNT;
uint32_t HH16CNT;
uint32_t HT16CNT;
uint32_t LT16CNT;
uint32_t MT16CNT;
uint32_t CH16CNT;
uint32_t OH16CNT;
uint32_t RD16CNT;
uint32_t RS16CNT;
uint32_t SD16CNT;
#define BD16LEN 3796UL
#define CP16LEN 4445UL
#define CR16LEN 48686UL
#define HH16LEN 1734UL
#define HT16LEN 5802UL
#define LT16LEN 7061UL
#define MT16LEN 7304UL
#define OH16LEN 4772UL
#define RD16LEN 52850UL
#define RS16LEN 1316UL
#define SD16LEN 5577UL

This defines the sample counters and their lenght.

To keep the sample engine running a function needs to be defined that calculates the drum sounds.

 uint16_t SYNTH909() {
 int32_t DRUMTOTAL=0;
 if (BD16CNT<BD16LEN) DRUMTOTAL+=(pgm_read_word_near(BD16 + BD16CNT++)^32768)-32768;
 if (CP16CNT<CP16LEN) DRUMTOTAL+=(pgm_read_word_near(CP16 + CP16CNT++)^32768)-32768;
 if (CR16CNT<CR16LEN) DRUMTOTAL+=(pgm_read_word_near(CR16 + CR16CNT++)^32768)-32768;
 if (HH16CNT<HH16LEN) DRUMTOTAL+=(pgm_read_word_near(HH16 + HH16CNT++)^32768)-32768;
 if (HT16CNT<HT16LEN) DRUMTOTAL+=(pgm_read_word_near(HT16 + HT16CNT++)^32768)-32768;
 if (LT16CNT<LT16LEN) DRUMTOTAL+=(pgm_read_word_near(LT16 + LT16CNT++)^32768)-32768;
 if (MT16CNT<MT16LEN) DRUMTOTAL+=(pgm_read_word_near(MT16 + MT16CNT++)^32768)-32768;
 if (OH16CNT<OH16LEN) DRUMTOTAL+=(pgm_read_word_near(OH16 + OH16CNT++)^32768)-32768;
 if (RD16CNT<RD16LEN) DRUMTOTAL+=(pgm_read_word_near(RD16 + RD16CNT++)^32768)-32768;
 if (RS16CNT<RS16LEN) DRUMTOTAL+=(pgm_read_word_near(RS16 + RS16CNT++)^32768)-32768;
 if (SD16CNT<SD16LEN) DRUMTOTAL+=(pgm_read_word_near(SD16 + SD16CNT++)^32768)-32768;
 if (DRUMTOTAL>32767) DRUMTOTAL=32767;
 if (DRUMTOTAL<-32767) DRUMTOTAL=-32767;
 DRUMTOTAL+=32768;
 return DRUMTOTAL;
} 

In the main loop we add a call to the sample engine.

void loop() {
DAC=SYNTH909();
//Pulse Density Modulated 16-bit I2S DAC
for (uint8_t i=0;i<32;i++) { 
  i2sACC=i2sACC<<1;
  if(DAC >= err) {
    i2sACC|=1;
    err += 0xFFFF-DAC;
  }
    else
  {
    err -= DAC;
  }
 }
 bool flag=i2s_write_sample(i2sACC);

And finally the MIDI drum trigger function.

void MidiNoteOn(uint8_t channel, uint8_t note, uint8_t velocity) {
/* 909 MIDI Triggers
Bass Drum MIDI-35
Bass Drum MIDI-36
Rim Shot MIDI-37
Snare Drum MIDI-38
Hand Clap MIDI-39
Snare Drum MIDI-40
Low Tom MIDI-41
Closed Hat MIDI-42
Low Tom MIDI-43
Closed Hat MIDI-44
Mid Tom MIDI-45
Open Hat MIDI-46
Mid Tom MIDI-47
Hi Tom MIDI-48
Crash Cymbal MIDI-49
Hi Tom MIDI-50
Ride Cymbal MIDI-51
*/
if (channel==10) {
if(note==35) BD16CNT=0;
if(note==36) BD16CNT=0;
if(note==37) RS16CNT=0;
if(note==38) SD16CNT=0;
if(note==39) CP16CNT=0;
if(note==40) SD16CNT=0;
if(note==41) LT16CNT=0;
if(note==42) HH16CNT=0;
if(note==43) LT16CNT=0;
if(note==44) HH16CNT=0;
if(note==45) MT16CNT=0;
if(note==46) OH16CNT=0;
if(note==47) MT16CNT=0;
if(note==48) HT16CNT=0;
if(note==49) CR16CNT=0;
if(note==50) HT16CNT=0;
if(note==51) RD16CNT=0;
}
}

MIDI data for this can come from edge triggers on GPIO’s, serial MIDI or rtpMIDI.

You can easily add velocity data to scale the samples in the engine to make accented drums.

Download the sketch here:

ESP909.ino

Rearrange the code for ISR

Using the CPU to fill the DMA buffer is not nice if you have a DMA and not good if you want to run MIDI input events.

So we are going to rearrange the code into a ISR serviced at a 2mS intervall.

The definitions are the same except for the added Ticker library.

#include <Arduino.h> 
#include "ESP8266WiFi.h"
#include <i2s.h>
#include <i2s_reg.h>
#include <pgmspace.h>
#include <Ticker.h>
uint32_t i2sACC;
uint8_t i2sCNT=32;
uint16_t DAC=0x8000;
uint16_t err;

And our test sine waveform.

int16_t sine[256] = {
0x0000, 0x0324, 0x0647, 0x096a, 0x0c8b, 0x0fab, 0x12c8, 0x15e2,
0x18f8, 0x1c0b, 0x1f19, 0x2223, 0x2528, 0x2826, 0x2b1f, 0x2e11,
0x30fb, 0x33de, 0x36ba, 0x398c, 0x3c56, 0x3f17, 0x41ce, 0x447a,
0x471c, 0x49b4, 0x4c3f, 0x4ebf, 0x5133, 0x539b, 0x55f5, 0x5842,
0x5a82, 0x5cb4, 0x5ed7, 0x60ec, 0x62f2, 0x64e8, 0x66cf, 0x68a6,
0x6a6d, 0x6c24, 0x6dca, 0x6f5f, 0x70e2, 0x7255, 0x73b5, 0x7504,
0x7641, 0x776c, 0x7884, 0x798a, 0x7a7d, 0x7b5d, 0x7c29, 0x7ce3,
0x7d8a, 0x7e1d, 0x7e9d, 0x7f09, 0x7f62, 0x7fa7, 0x7fd8, 0x7ff6,
0x7fff, 0x7ff6, 0x7fd8, 0x7fa7, 0x7f62, 0x7f09, 0x7e9d, 0x7e1d,
0x7d8a, 0x7ce3, 0x7c29, 0x7b5d, 0x7a7d, 0x798a, 0x7884, 0x776c,
0x7641, 0x7504, 0x73b5, 0x7255, 0x70e2, 0x6f5f, 0x6dca, 0x6c24,
0x6a6d, 0x68a6, 0x66cf, 0x64e8, 0x62f2, 0x60ec, 0x5ed7, 0x5cb4,
0x5a82, 0x5842, 0x55f5, 0x539b, 0x5133, 0x4ebf, 0x4c3f, 0x49b4,
0x471c, 0x447a, 0x41ce, 0x3f17, 0x3c56, 0x398c, 0x36ba, 0x33de,
0x30fb, 0x2e11, 0x2b1f, 0x2826, 0x2528, 0x2223, 0x1f19, 0x1c0b,
0x18f8, 0x15e2, 0x12c8, 0x0fab, 0x0c8b, 0x096a, 0x0647, 0x0324,
0x0000, 0xfcdc, 0xf9b9, 0xf696, 0xf375, 0xf055, 0xed38, 0xea1e,
0xe708, 0xe3f5, 0xe0e7, 0xdddd, 0xdad8, 0xd7da, 0xd4e1, 0xd1ef,
0xcf05, 0xcc22, 0xc946, 0xc674, 0xc3aa, 0xc0e9, 0xbe32, 0xbb86,
0xb8e4, 0xb64c, 0xb3c1, 0xb141, 0xaecd, 0xac65, 0xaa0b, 0xa7be,
0xa57e, 0xa34c, 0xa129, 0x9f14, 0x9d0e, 0x9b18, 0x9931, 0x975a,
0x9593, 0x93dc, 0x9236, 0x90a1, 0x8f1e, 0x8dab, 0x8c4b, 0x8afc,
0x89bf, 0x8894, 0x877c, 0x8676, 0x8583, 0x84a3, 0x83d7, 0x831d,
0x8276, 0x81e3, 0x8163, 0x80f7, 0x809e, 0x8059, 0x8028, 0x800a,
0x8000, 0x800a, 0x8028, 0x8059, 0x809e, 0x80f7, 0x8163, 0x81e3,
0x8276, 0x831d, 0x83d7, 0x84a3, 0x8583, 0x8676, 0x877c, 0x8894,
0x89bf, 0x8afc, 0x8c4b, 0x8dab, 0x8f1e, 0x90a1, 0x9236, 0x93dc,
0x9593, 0x975a, 0x9931, 0x9b18, 0x9d0e, 0x9f14, 0xa129, 0xa34c,
0xa57e, 0xa7be, 0xaa0b, 0xac65, 0xaecd, 0xb141, 0xb3c1, 0xb64c,
0xb8e4, 0xbb86, 0xbe32, 0xc0e9, 0xc3aa, 0xc674, 0xc946, 0xcc22,
0xcf05, 0xd1ef, 0xd4e1, 0xd7da, 0xdad8, 0xdddd, 0xe0e7, 0xe3f5,
0xe708, 0xea1e, 0xed38, 0xf055, 0xf375, 0xf696, 0xf9b9, 0xfcdc
};
uint8_t phase=0; //Sine phase counter

The setup function now has some Timer code added.

void setup() {
i2s_begin(); //Start the i2s DMA engine
i2s_set_rate(44100); //Set sample rate
pinMode(2, INPUT); //restore GPIOs taken by i2s
pinMode(15, INPUT);
timer1_attachInterrupt(onTimerISR); //Attach our sampling ISR
timer1_enable(TIM_DIV16, TIM_EDGE, TIM_SINGLE);
timer1_write(2000); //Service at 2mS intervall
}

The main loop is now as you see empty.

void loop() {
}

This is because the DMA engine has moved to a ISR.

void ICACHE_RAM_ATTR onTimerISR(){ //Code needs to be in IRAM because its a ISR
while (!(i2s_is_full())) { //Don’t block the ISR if the buffer is full
DAC=0x8000+sine[phase++];
//Pulse Density Modulated 16-bit I2S DAC
for (uint8_t i=0;i<32;i++) { 
  i2sACC=i2sACC<<1;
  if(DAC >= err) {
    i2sACC|=1;
    err += 0xFFFF-DAC;
  }
    else
  {
    err -= DAC;
  }
 }
 bool flag=i2s_write_sample(i2sACC);
}
timer1_write(2000);//Next in 2mS
}

This does the same as the first example but you are now free to put what ever you like in the main loop because the timer takes care of loading data to the DMA.

The DMA is automatically serviced at a 2mS intervall and you can process MIDI data in the main loop instead.

Reading serial MIDI data

How do we actually read the MIDI data? Our serial port is used by the i2s stream so cant be used as a serial port.

We do this by moving the RX and TX pins to the alternate pins.

Serial.swap();

This moves the RX pin to GPIO13 and the TX pin to GPIO15.

You need to setup the serial port before you start the i2s engine because the serial setup will destroy the i2s GPIO setup.

void setup() {
  Serial.begin(31250); //Start the serial port with default MIDI baudrate
  Serial.swap(); //Move the TX and RX GPIOs to 15 and 13
  i2s_begin(); //Start the i2s DMA engine
  i2s_set_rate(44100); //Set sample rate
  pinMode(2, INPUT); //restore GPIOs taken by i2s
  pinMode(15, INPUT);
  timer1_attachInterrupt(onTimerISR); //Attach our sampling ISR
  timer1_enable(TIM_DIV16, TIM_EDGE, TIM_SINGLE);
  timer1_write(2000); //Service at 2mS intervall
}

Add the MIDI process definitions.

uint8_t MIDISTATE=0;
uint8_t MIDIRUNNINGSTATUS=0;
uint8_t MIDINOTE;
uint8_t MIDIVEL;

And the MIDI processor.

void processMIDI(uint8_t MIDIRX) {
/*
Handling “Running status”
1.Buffer is cleared (ie, set to 0) at power up.
2.Buffer stores the status when a Voice Category Status (ie, 0x80 to 0xEF) is received.
3.Buffer is cleared when a System Common Category Status (ie, 0xF0 to 0xF7) is received.
4.Nothing is done to the buffer when a RealTime Category message is received.
5.Any data bytes are ignored when the buffer is 0.
*/
if ((MIDIRX>0xBF)&&(MIDIRX<0xF8)) {
 MIDIRUNNINGSTATUS=0;
 MIDISTATE=0;
 return;
}
if (MIDIRX>0xF7) return;
if (MIDIRX & 0x80) {
  MIDIRUNNINGSTATUS=MIDIRX;
  MIDISTATE=1;
return;
}
if (MIDIRX < 0x80) {
  if (!MIDIRUNNINGSTATUS) return;
  if (MIDISTATE==1) {
  MIDINOTE=MIDIRX;
  MIDISTATE++;
  return;
}
if (MIDISTATE==2) {
  MIDIVEL=MIDIRX;
  MIDISTATE=1;
  //if (MIDIRUNNINGSTATUS==0x80) handleMIDInoteOFF(MIDIRUNNINGSTATUS,MIDINOTE,MIDIVEL);
  //if (MIDIRUNNINGSTATUS==0x90) handleMIDInoteON(MIDIRUNNINGSTATUS,MIDINOTE,MIDIVEL);
  //if (MIDIRUNNINGSTATUS==0xB0) handleMIDICC(MIDINOTE,MIDIVEL);
}
}
}

You need to add the handlers for noteOFF, noteON and MIDICC.

Now we can process incoming MIDI bytes in the main loop.

void loop() {
  if (Serial.available()) processMIDI(Serial.read());
}

Now you can apply our new DMA engine and serial MIDI processor on the simple drum player and play it from a keyboard or sequencer.

rtpMIDI on the ESP8266

What about rtpMIDI, or Apple-MIDI over WiFI?

Well it works pretty well on our drum machine.

To use it you need to download and install the Apple-MIDI library:https://github.com/lathoub/Arduino-AppleMIDI-Library

Our drum machine defines.

#include <Arduino.h> 
#include "ESP8266WiFi.h"
#include <WiFiClient.h>
#include <WiFiUdp.h>
#include <i2s.h>
#include <i2s_reg.h>
#include <pgmspace.h>
#include "AppleMidi.h"
#include <Ticker.h>
extern “C” {
#include “user_interface.h”
}
char ssid[] = "YourSSID"; //  your network SSID (name)
char pass[] = "YourKEY";    // your network password (use for WPA, or use as key for WEP)
APPLEMIDI_CREATE_INSTANCE(WiFiUDP, AppleMIDI); // see definition in AppleMidi_Defs.h
// Forward declaration
void OnAppleMidiConnected(uint32_t ssrc, char* name);
void OnAppleMidiDisconnected(uint32_t ssrc);
void OnAppleMidiNoteOn(byte channel, byte note, byte velocity);
void OnAppleMidiNoteOff(byte channel, byte note, byte velocity);
uint32_t i2sACC;
uint8_t i2sCNT=32;
uint16_t DAC=0x8000;
uint16_t err;
uint32_t BD16CNT;
uint32_t CP16CNT;
uint32_t CR16CNT;
uint32_t HH16CNT;
uint32_t HT16CNT;
uint32_t LT16CNT;
uint32_t MT16CNT;
uint32_t CH16CNT;
uint32_t OH16CNT;
uint32_t RD16CNT;
uint32_t RS16CNT;
uint32_t SD16CNT;
#define BD16LEN 3796UL
#define CP16LEN 4445UL
#define CR16LEN 48686UL
#define HH16LEN 1734UL
#define HT16LEN 5802UL
#define LT16LEN 7061UL
#define MT16LEN 7304UL
#define OH16LEN 4772UL
#define RD16LEN 52850UL
#define RS16LEN 1316UL
#define SD16LEN 5577UL
const uint16_t BD16[3796] PROGMEM = {
40, 85, 137, 144, -30, -347, -609, -785, // 0-7

The defines above are the same as the original drum sampler code but with the Apple MIDI added.

The 909 synth engine is still the same.

uint16_t SYNTH909() {
  int32_t DRUMTOTAL=0;
  if (BD16CNT<BD16LEN) DRUMTOTAL+=(pgm_read_word_near(BD16 + BD16CNT++)^32768)-32768;
  if (CP16CNT<CP16LEN) DRUMTOTAL+=(pgm_read_word_near(CP16 + CP16CNT++)^32768)-32768;
  if (CR16CNT<CR16LEN) DRUMTOTAL+=(pgm_read_word_near(CR16 + CR16CNT++)^32768)-32768;
  if (HH16CNT<HH16LEN) DRUMTOTAL+=(pgm_read_word_near(HH16 + HH16CNT++)^32768)-32768;
  if (HT16CNT<HT16LEN) DRUMTOTAL+=(pgm_read_word_near(HT16 + HT16CNT++)^32768)-32768;
  if (LT16CNT<LT16LEN) DRUMTOTAL+=(pgm_read_word_near(LT16 + LT16CNT++)^32768)-32768;
  if (MT16CNT<MT16LEN) DRUMTOTAL+=(pgm_read_word_near(MT16 + MT16CNT++)^32768)-32768;
  if (OH16CNT<OH16LEN) DRUMTOTAL+=(pgm_read_word_near(OH16 + OH16CNT++)^32768)-32768;
  if (RD16CNT<RD16LEN) DRUMTOTAL+=(pgm_read_word_near(RD16 + RD16CNT++)^32768)-32768;
  if (RS16CNT<RS16LEN) DRUMTOTAL+=(pgm_read_word_near(RS16 + RS16CNT++)^32768)-32768;
  if (SD16CNT<SD16LEN) DRUMTOTAL+=(pgm_read_word_near(SD16 + SD16CNT++)^32768)-32768;
  if  (DRUMTOTAL>32767) DRUMTOTAL=32767;
  if  (DRUMTOTAL<-32767) DRUMTOTAL=-32767;
  DRUMTOTAL+=32768;
  return DRUMTOTAL;
}

Setup includes some new code to add the ESP8266 to your WiFi network.

void setup() {
//WiFi.forceSleepBegin();
//delay(1);
system_update_cpu_freq(160);
//Serial.begin(9600);
WiFi.begin(ssid, pass);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
}
//Serial.print(F(“IP address is “));
//Serial.println(WiFi.localIP());
AppleMIDI.begin(“ESP909”); // ‘ESP909’ will show up as the session name
AppleMIDI.OnReceiveNoteOn(OnAppleMidiNoteOn);
i2s_begin();
i2s_set_rate(44100);
timer1_attachInterrupt(onTimerISR); //Attach our sampling ISR
timer1_enable(TIM_DIV16, TIM_EDGE, TIM_SINGLE);
timer1_write(2000); //Service at 2mS intervall
}

The main loop now has the Apple MIDI status code in it.

void loop() {
AppleMIDI.run();
}

And our main sampling ISR is the same.

void ICACHE_RAM_ATTR onTimerISR(){
  
  while (!(i2s_is_full())) { //Don't block the ISR
    
    DAC=SYNTH909();
    //----------------- Pulse Density Modulated 16-bit I2S DAC --------------------
     for (uint8_t i=0;i<32;i++) { 
      i2sACC=i2sACC<<1;
      if(DAC >= err) {
        i2sACC|=1;
        err += 0xFFFF-DAC;
      }
        else
      {
        err -= DAC;
      }
     }
     bool flag=i2s_write_sample(i2sACC);
    //-----------------------------------------------------------------------
  }
  
  timer1_write(2000);//Next in 2mS
} 

But now there is a new function added to handle MIDI events.

void OnAppleMidiNoteOn(byte channel, byte note, byte velocity) {
/* Triggers
Bass Drum MIDI-35
Bass Drum MIDI-36
Rim Shot MIDI-37
Snare Drum MIDI-38
Hand Clap MIDI-39
Snare Drum MIDI-40
Low Tom MIDI-41
Closed Hat MIDI-42
Low Tom MIDI-43
Closed Hat MIDI-44
Mid Tom MIDI-45
Open Hat MIDI-46
Mid Tom MIDI-47
Hi Tom MIDI-48
Crash Cymbal MIDI-49
Hi Tom MIDI-50
Ride Cymbal MIDI-51
*/
if (channel==10) {
if(note==35) BD16CNT=0;
if(note==36) BD16CNT=0;
if(note==37) RS16CNT=0;
if(note==38) SD16CNT=0;
if(note==39) CP16CNT=0;
if(note==40) SD16CNT=0;
if(note==41) LT16CNT=0;
if(note==42) HH16CNT=0;
if(note==43) LT16CNT=0;
if(note==44) HH16CNT=0;
if(note==45) MT16CNT=0;
if(note==46) OH16CNT=0;
if(note==47) MT16CNT=0;
if(note==48) HT16CNT=0;
if(note==49) CR16CNT=0;
if(note==50) HT16CNT=0;
if(note==51) RD16CNT=0;
}
}

To make this work you need to set up Apple rtpMIDI on your Mac, iPad or PC.

I cant show you how to do this because it is up to your platform how it is done.

And you need to find the IP address of your ESP8266 to pair it with your MIDI computer.

Enable the serial debug code and watch the serial console for your IP address.

But once that is done it works pretty well.

Download the sketch here:

rtpMIDI909.ino

This is the most basic sample player.As long as the samples fits into your flash space you can play whatever you want and control it over MIDI or WiFi.

Compared to the samplers of the 90’s this is way better. 16-bit audio and 4Mbyte memory in a tiny space is way to good.

Sample playing keyboard

With our newly found knowledge of the ESP8266 we now go on with creating a sample playing keyboard or Rompler.

The difference to our drum sampler is that this plays the sample chromatically and polyphonically.

The EMU-II in the picture above used 8-bit DPCM samples at the strange sample rate of 27.7KHz. To make it more simple we are going to use 16-bit signed samples at a rate of 32KHz.

The EMU-II was 8-voice polyphonic but I’m no good at writing voice assigners so our sampler will be 128-voice fully polyphonic.

While you can in theory play all the 128 MIDI keys at once with individual envelopes the polyphony will be less because of limits in the processing power.

Our definitions

#include <Arduino.h> 
#include "ESP8266WiFi.h"
#include <i2s.h>
#include <i2s_reg.h>
#include <pgmspace.h>
#include <Ticker.h>
uint32_t i2sACC;
uint8_t i2sCNT=32;
uint16_t DAC=0x8000;
uint16_t err;
  
//Envelope and VCA parameters
volatile ENVcnt=8; //16mS env resolution
int16_t VCA[128]; //VCA levels
volatile uint8_t ATTACK=30; // ENV Attack rate 1-255
volatile uint8_t RELEASE=3; // ENV Release rate 1-255
  
//Sample parameters and tables
uint32_t FREQ[128]; //Phase accumulators
uint32_t SPNT[128]; //Sample pointers
uint32_t LOOP1[128]; //Start of loop segment in sample
uint32_t LOOP2[128]; //End of loop segment in sample
uint32_t SLEN[128]; //Length of sample 

There are as you can see 128 tables of each parameter.

The parameters for each key on the keyboard is:

• Phase accumulator (more explained later)
• Sample pointer (The linear counter for time inside the sample)
• LOOP1 (The starting point of a sustain loop)
• LOOP2 (The end point of the loop and where it jumps back to LOOP1)
• Length (How long or number of words the total sample is)

The setup

This is our setup routine using MIDI input to play the samples.

void setup() {
 WiFi.forceSleepBegin(); //Turn off WiFi radio
 delay(1); //Wait for it to turn off
 system_update_cpu_freq(160);
 Serial.begin(31250); //Start the serial port with default MIDI baudrate
 Serial.swap(); //Move the TX and RX GPIOs to 15 and 13
 i2s_begin(); //Start the i2s DMA engine
 i2s_set_rate(32000); //Set sample rate
 pinMode(2, INPUT); //restore GPIOs taken by i2s
 pinMode(15, INPUT);
 timer1_attachInterrupt(onTimerISR); //Attach our sampling ISR
 timer1_enable(TIM_DIV16, TIM_EDGE, TIM_SINGLE);
 timer1_write(2000); //Service at 2mS intervall
}

It turns off the WiFi radio, bumps the CPU freq up to 160MHz, sets up the UART for MIDI, start the i2s DMA engine and turns on the Timer.

We also need the Timer interrupt that takes care of loading the DMA at a 2mS interval.

void ICACHE_RAM_ATTR onTimerISR(){ //Code needs to be in IRAM because its a ISR
 while (!(i2s_is_full())) { //Don’t block the ISR if the buffer is full
   DAC=samplerTick(); //Calculate current sample value
   //Pulse Density Modulated 16-bit I2S DAC
   for (uint8_t i=0;i<32;i++) { 
     i2sACC=i2sACC<<1;
     if(DAC >= err) {
       i2sACC|=1;
       err += 0xFFFF-DAC;
     }
       else
     {
       err -= DAC;
     }
   }
   bool flag=i2s_write_sample(i2sACC);
}
//Envelope handler
if (!(ENVcnt--)) { //Calculate ENV every 16mS
 ENVcnt==8;
 for (envcnt=0;envcnt<128;envcnt++) { //128 VCA's
 if ((MIDItable[envcnt]>0)&&(VCA[envcnt]<255)) {
   VCA[envcnt]+=ATTACK;
   if (VCA[envcnt]>255) VCA[envcnt]=255;
 }
 if ((MIDItable[envcnt]==0)&&(VCA[envcnt]>0)) {
   VCA[envcnt]-=RELEASE;
   if (VCA[envcnt]<0) VCA[envcnt]=0;
 }
}
 timer1_write(2000);//Next in 2mS
}

Inside the Timer handler we also run our envelope generators at a 16mS interval.

Each key has its own Attack/Decay volume envelope. Sustain is always at full level.

MIDI handler

Our loop() takes care of checking if serial data is available and if so runs the MIDI processor.

void loop() {
  if (Serial.available()) processMIDI(Serial.read());
}

If MIDI data is available it processes it.

void processMIDI(uint8_t MIDIRX) { //MIDI processor
  
/*
Handling “Running status”
1.Buffer is cleared (ie, set to 0) at power up.
2.Buffer stores the status when a Voice Category Status (ie, 0x80 to 0xEF) is received.
3.Buffer is cleared when a System Common Category Status (ie, 0xF0 to 0xF7) is received.
4.Nothing is done to the buffer when a RealTime Category message is received.
5.Any data bytes are ignored when the buffer is 0.
*/
  
 if ((MIDIRX>0xBF)&&(MIDIRX<0xF8)) {
   MIDIRUNNINGSTATUS=0;
   MIDISTATE=0;
   return;
 }
 if (MIDIRX>0xF7) return;
 if (MIDIRX & 0x80) {
   MIDIRUNNINGSTATUS=MIDIRX;
   MIDISTATE=1;
   return;
 }
 if (MIDIRX < 0x80) {
   if (!MIDIRUNNINGSTATUS) return;
   if (MIDISTATE==1) {
     MIDINOTE=MIDIRX;
     MIDISTATE++;
     return;
   }
   if (MIDISTATE==2) {
     MIDIVEL=MIDIRX;
     MIDISTATE=1;
     if (MIDIRUNNINGSTATUS==0x80) handleMIDInoteOFF(MIDIRUNNINGSTATUS,MIDINOTE,MIDIVEL);
     if (MIDIRUNNINGSTATUS==0x90) handleMIDInoteON(MIDIRUNNINGSTATUS,MIDINOTE,MIDIVEL);
     //if (MIDIRUNNINGSTATUS==0xB0) handleMIDICC(MIDINOTE,MIDIVEL);
   }
 }
}
void handleMIDInoteON(MIDIRUNNINGSTATUS,MIDINOTE,MIDIVEL);
 MIDItable[MIDINOTE]=MIDIVEL;
}
void handleMIDInoteOFF(MIDIRUNNINGSTATUS,MIDINOTE,MIDIVEL);
 MIDItable[MIDINOTE]=0;
}

The handleMIDInoteON/OFF writes the the MIDI mapping table showing which keys are depressed.

The sample engine

This is where all the sample counters are handled and all the different samples are looped and summed.

As each key has a frequency that has a twelfth root relationship of the next key or multiplied/divided by 1.05 how do we get the frequency of each key as they are all processed at 32KHz?

The answer is the phase accumulator. It actually count fractions of one sample tick.

The counters are 15-bit and for the highest key we add 0x80000000 to it and if it overflows we have a full tick.

For one octave below we add 0x40000000 and at overflow we have half the frequency. 1 seminote below is 0x80000000 / 1.05 and so on.

The ticker for the C3 octave looks like this:

void samplerTick() //Calculate total sample value for each playing note
 int32_t total=0;
 if ((VCA[48+0])&&(SPNT[48+0]<SLEN[48+0])) { //If VCA is active and the sample has not reached end
   FREQ[48+0]+=1073741824; //Add frequency to the phase accumulator for C3 key
   if (FREQ[48+0]&0x8000000) { //If phase accumulator overflows
     FREQ[48+0]&=0x7FFFFFFF; //Trim off MSB
     if ((SPNT[48+0]>LOOP2[48+0])&&(MIDItable[48+0])) SPNT[48+0]=LOOP1[48+0]; //Check if we're in a loop
     total+=(((pgm_read_word_near(SAMPLE + SPNT[48+0])^32768)-32768)*VCA[48+0])>>8; //Add the sample value to total with ENV scaling
     SPNT[48+0]++; //Increment sample pointer
   }
 }
 if ((VCA[49+0])&&(SPNT[49+0]<SLEN[49+0])) {
   FREQ[49+0]+=1137589835; //Add frequency to counter for C3# key
   if (FREQ[49+0]&0x8000000) {
     FREQ[49+0]&=0x7FFFFFFF;
     if ((SPNT[49+0]>LOOP2[49+0])&&(MIDItable[49+0])) SPNT[49+0]=LOOP1[49+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[49+0])^32768)-32768)*VCA[49+0])>>8;
     SPNT[49+0]++;
   }
 }
 if ((VCA[50+0])&&(SPNT[50+0]<SLEN[50+0])) {
   FREQ[50+0]+=1205234447; //Add frequency to counter for D3 key
   if (FREQ[50+0]&0x8000000) {
     FREQ[50+0]&=0x7FFFFFFF;
     if ((SPNT[50+0]>LOOP2[50+0])&&(MIDItable[50+0])) SPNT[50+0]=LOOP1[50+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[50+0])^32768)-32768)*VCA[50+0])>>8;
     SPNT[50+0]++;
   }
 }
 if ((VCA[51+0])&&(SPNT[51+0]<SLEN[51+0])) {
   FREQ[51+0]+=1276901416; //Add frequency to counter for D3# key
   if (FREQ[51+0]&0x8000000) {
     FREQ[51+0]&=0x7FFFFFFF;
     if ((SPNT[51+0]>LOOP2[51+0])&&(MIDItable[51+0])) SPNT[51+0]=LOOP1[51+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[51+0])^32768)-32768)*VCA[51+0])>>8;
     SPNT[51+0]++;
   }
 }
 if ((VCA[52+0])&&(SPNT[52+0]<SLEN[52+0])) {
   FREQ[52+0]+=1352829926; //Add frequency to counter for E3 key
   if (FREQ[52+0]&0x8000000) {
     FREQ[52+0]&=0x7FFFFFFF;
     if ((SPNT[52+0]>LOOP2[52+0])&&(MIDItable[52+0])) SPNT[52+0]=LOOP1[52+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[52+0])^32768)-32768)*VCA[52+0])>>8;
     SPNT[52+0]++;
   }
 }
 if ((VCA[53+0])&&(SPNT[53+0]<SLEN[53+0])) {
   FREQ[53+0]+=1433273379; //Add frequency to counter for F3 key
   if (FREQ[53+0]&0x8000000) {
     FREQ[53+0]&=0x7FFFFFFF;
     if ((SPNT[53+0]>LOOP2[53+0])&&(MIDItable[53+0])) SPNT[53+0]=LOOP1[53+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[53+0])^32768)-32768)*VCA[53+0])>>8;
     SPNT[53+0]++;
   }
 }
 if ((VCA[54+0])&&(SPNT[54+0]<SLEN[54+0])) {
   FREQ[54+0]+=1518500249; //Add frequency to counter for G3 key
   if (FREQ[54+0]&0x8000000) {
     FREQ[54+0]&=0x7FFFFFFF;
     if ((SPNT[54+0]>LOOP2[54+0])&&(MIDItable[54+0])) SPNT[54+0]=LOOP1[54+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[54+0])^32768)-32768)*VCA[54+0])>>8;
     SPNT[54+0]++;
   }
 }
 if ((VCA[55+0])&&(SPNT[55+0]<SLEN[55+0])) {
   FREQ[55+0]+=1608794973; //Add frequency to counter for G3# key
   if (FREQ[55+0]&0x8000000) {
     FREQ[55+0]&=0x7FFFFFFF;
     if ((SPNT[55+0]>LOOP2[55+0])&&(MIDItable[55+0])) SPNT[55+0]=LOOP1[55+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[55+0])^32768)-32768)*VCA[55+0])>>8;
     SPNT[55+0]++;
   }
 }
 if ((VCA[56+0])&&(SPNT[56+0]<SLEN[56+0])) {
   FREQ[56+0]+=1704458900; //Add frequency to counter for A3 key
   if (FREQ[56+0]&0x8000000) {
     FREQ[56+0]&=0x7FFFFFFF;
     if ((SPNT[56+0]>LOOP2[56+0])&&(MIDItable[56+0])) SPNT[56+0]=LOOP1[56+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[56+0])^32768)-32768)*VCA[56+0])>>8;
     SPNT[56+0]++;
   }
 }
 if ((VCA[57+0])&&(SPNT[57+0]<SLEN[57+0])) {
   FREQ[57+0]+=1805811301; //Add frequency to counter for A3# key
   if (FREQ[57+0]&0x8000000) {
     FREQ[57+0]&=0x7FFFFFFF;
     if ((SPNT[57+0]>LOOP2[57+0])&&(MIDItable[57+0])) SPNT[57+0]=LOOP1[57+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[57+0])^32768)-32768)*VCA[57+0])>>8;
     SPNT[57+0]++;
   }
 }
 if ((VCA[58+0])&&(SPNT[58+0]<SLEN[58+0])) {
   FREQ[58+0]+=1913190429; //Add frequency to counter for B3 key
   if (FREQ[58+0]&0x8000000) {
     FREQ[58+0]&=0x7FFFFFFF;
     if ((SPNT[58+0]>LOOP2[58+0])&&(MIDItable[58+0])) SPNT[58+0]=LOOP1[58+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[58+0])^32768)-32768)*VCA[58+0])>>8;
     SPNT[58+0]++;
   }
 }
 if ((VCA[59+0])&&(SPNT[59+0]<SLEN[59+0])) {
   FREQ[59+0]+=2026954652; //Add frequency to counter for B3# key
   if (FREQ[59+0]&0x8000000) {
     FREQ[59+0]&=0x7FFFFFFF;
     if ((SPNT[59+0]>LOOP2[59+0])&&(MIDItable[59+0])) SPNT[59+0]=LOOP1[59+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[59+0])^32768)-32768)*VCA[59+0])>>8;
     SPNT[59+0]++;
   }
 }
 if ((VCA[60+0])&&(SPNT[60+0]<SLEN[60+0])) {
   FREQ[60+0]+=2147483648; //Add frequency to counter for C4 key, this overflows every tick thus 32KHz
   if (FREQ[60+0]&0x8000000) {
     FREQ[60+0]&=0x7FFFFFFF;
     if ((SPNT[60+0]>LOOP2[60+0])&&(MIDItable[60+0])) SPNT[60+0]=LOOP1[60+0];
     total+=(((pgm_read_word_near(SAMPLE + SPNT[60+0])^32768)-32768)*VCA[60+0])>>8;
     SPNT[60+0]++;
   }
 }
 if (total>32767) total=32767; //Clip to max
 if (total<-32767) total=-32767; //Clip to min
 total+=32768; //Center value
 return total;
}

For each key in the octave, if it’s still sounding (VCA>0) and the sample has not past the end we add the frequency to the phase accumulator for that key.

Then we check if the key is held and we pass a loop point and in that case jump the sample pointer to the beginning of the loop.

Then we fetch the sample value and scale it’s volume to the VCA value.

Now you probably figured out that we can have multi-samples if we want because each key has unique sample parameters.

Finally since we are adding samples up we could pass the dynamic range of our signal so we clip it to the limits.

I showed the example for the C3 octave because C4 is 32KHz or one tick and the key we make our samples in.

But we are going to need 9 octaves more of this.