Team AnimalComms: Avik Ghosh, Mario De Los Santos

Published September 28, 2022

Correlations in Animal Movement and Communication (CAMC)

CAMC will build a large dataset for Machine Learning specialists to find possible correlations between animal speech and movement.

IntermediateShowcase (no instructions)133

Correlations in Animal Movement and Communication (CAMC)

Things used in this project

Hardware components

Radxa Zero

This was the SBC that was used to work on the project. Generally, any SBC will work! Radxa seems to have some of the best support among Raspberry Pi Competitors. An RPi would be recommended but acquiring a new one nowadays is difficult. Also, because this project is a proof of concept, keeping GPIO pins would be suggested for future versions that can be expanded to carry more sensor data.

Samson Go Mic

Any microphone can be used! This was simply the microphone I had at home. USB-A microphones are recommended because of the necessary USB-C to A hub that will be used.

Micro HDMI to HDMI cable

A Monitor

Any monitor can be used for the initial setup. If not possible, borrowing a monitor would be perfectly fine because, after the setup, the project can be worked on remotely.

USB C Hub

There is only one USB C port on the Radxa Zero apart from power. Using a USB C hub allows the user to utilize a keyboard and mouse.

Seeed Studio PCB Board From Seeed Fusion

Software apps and online services

Debian Buster

balena Etcher

IotFlows

Story

As a child, I was always fascinated by animals! They had a form of communication that was not understood by humans. Some believed them to be drastically inferior to humanity, but I was not yet convinced. Admittedly, that may have had something to do with the many animated movies I had grown up watching, rather than any scientific basis.

Nevertheless, this project is my homage to the childhood dream I had. It's an attempt to decipher whether animal communication is complex enough to have correlations to movement. The project itself will not work on any deciphering – the point of the project is to build a dataset. Later on, a professional would be able to look for correlations between communication between the animal’s sounds, the GPS data, and the time.

First, this document will discuss how it would be attached to the animal. Next, we will review the program itself (more detail can be obtained by looking at the actual code). Lastly, we will discuss two methods to extract the data.

This system would be attached as a collar to an animal. I have not provided a design for the attachment mechanism, but a simple collar with a box should be enough. See the image below for an example.

Elephant with tracking collar. Thank you to https://www.worldwildlife.org/magazine/issues/summer-2017/articles/tracking-elephant-migrations!

The core of this project is deciding which data to transmit back and making the most efficient usage of data transmission and storage limits.

But first, how does it work? There are two main functions (there’s another one which is more of a helper function). recording() and checkRecording() are the two functions, and understanding their slightly complicated dynamic will make the project easier to understand.

The first function that will be called in the program will be recording(). recording() will record a.wav file for 0.1 seconds in the first run called recording1.wav. This sets up the dynamic that starts after the first run and stays for the rest of the program.

In the second run, recording() and checkRecording() will be called. The reason checkRecording() couldn’t be called in the first run is that there is no recording to check then! But now, checkRecording() will check the last recording of 0.1 seconds (recording1.wav) while the function recording() will save recording2.wav. checkRecording() will check if the audio should be saved or discarded – if saved, it will call storeRecording(). Say storeRecording() is not called because checkRecording() thinks no animals have talked in the last 0.5 seconds.

Then comes the third run. checkRecording() will check recording2.wav, while recording() works on making recording3.wav. Remember that recording() will always be one recording ahead of the file that checkRecording() will analyze.

This continues until checkRecording() thinks an animal did speak! But how does that happen? There are a few steps. First, we need to find a way to check whether or not it should be saved. A good way we found would be to find an average amplitude. If the average amplitude is high enough, checkRecording() calls storeRecording().

But the averages would occasionally be too low, or negative! So we had to do some preprocessing first. First, we make everything positive, and set values under 0.2 to 0.0001. All values below 0.0001 would be discarded in the average, and we would calculate the arithmetic mean over the remaining samples. If it’s above the minimum loudness threshold (we set it to be 0.4), then the program would know to record the next few seconds and store the data.

Because we want the process to run as quickly as possible (in 0.1 seconds), we vectorized the methods using NumPy as much as possible.

Next, what happens when the function calls storeRecording()?

Calling storeRecording() will pause recording() until completion. storeRecording() will record the next 7 seconds of audio and store it in a text file. Accompanying the audio will be GPS data and time data. This will allow the researchers to utilize this data to look for correlations between the audio and the movement of animals. For example, every time a cat makes a specific sound, the cat moves forward. This tells the researchers to investigate whether or not the specific sound can actually mean “move forward”, giving us a very useful tool to move forward in deciphering animal communication.

If words aren’t your thing, here’s a diagram!

Flowchart of program functions

Now the data we need is on the board. How do we get access to it though?

There are two simple ways to do so. The first is to use SCP. SCP is a file transfer system built on the SSH protocol and works over WiFi. The issue with this is that you would have to manually deal with downloading the files. On your local machine, run “scp board_name@ip_addr:location/of/file location/to/store” and input your password. A quick tip would be to cd into the directory you wish to save files in and replace location/to/store with a simple period (current directory)! (Look into bash’s wildcard system to streamline your file retrieval if this is the way you choose to go!)

The other system is using an IoT service such as IoTFlows. IoTFlows can be set up through a quick and fairly simple process and will allow users to view the full directories of the board itself. Below is an example of the user view. This is a much simpler process, and can actually be very helpful if your goal is to display your data on a website!

Example of IoTFlows website

Again, if words aren’t your thing, take a look at the simple diagram below.

For the audio itself, there were two systems to choose from. Realistically, a simple USB microphone would work, but we also have a PCB Board from Seeed’s Fusion service (https://www.seeedstudio.com/pcb-assembly.html) that has multiple microphones and would work just as well! I would suggest using a USB microphone if it’s already around – if not, the PCB service would work just fine. The gerber file (for the PCB) created by Mario De Los Santos is attached. Below is an image of how the project looks with the USB microphone.

And lastly, the internet. The system would do well with an LTE module, or simply being attached to someone’s pets! I wholeheartedly recommend this for any pet owners who wish to have an insight on the things their pets are saying all day long! Because the pets would primarily stay indoors, the system would work just off of the wifi connection that can be established.

That just about concludes the summary! Thank you for reading this far. I wish you the best of luck if you wish to make this project yourself, or improve it as well!

Code

Audio Processing File

#!/usr/bin/env python3
# getting ThinkDSP module for Digital Signal Processing
import os

if not os.path.exists('thinkdsp.py'):
    os.system(
        "wget 'https://github.com/AllenDowney/ThinkDSP/raw/master/code/thinkdsp.py'")

# import modules and methods needed
import sounddevice as sd  # allows users to record from microphones
from scipy.io.wavfile import write  # allows users to write out a .wav file
# allows users to run multiple functions at once
from multiprocessing import Process
import time  # allows users to store time data with the recording
import thinkdsp  # allows users to run the DSP section of the code
import numpy as np  # allows users to run rapid calculations
import subprocess  # allows users to run shell commands
import geocoder  # allows users to find their GPS coordinates

# global variables
AudioRec = 0
fileNum = 0

# setup recording
duration = 0.1
fs = 44100
sd.default.samplerate = fs
sd.default.channels = 2

'''
This function will actually record the last 0.1 seconds and save it as a .wav file. 
It's numbered progressively. In most cases, the recording would not be saved. 
'''


def recording():
    global duration
    global fs
    global fileNum

    myRec = sd.rec(int(duration * fs), dtype=np.int16)
    sd.wait()

    write("wav_files/recording" + str(fileNum) + ".wav",
          sd.default.samplerate, myRec.astype(np.int16))


'''
This function will check the recording that was saved by recording(). This function
will call storeRecording() if the recording passes the minimum amplitude threshold.
'''


def checkRecording():
    global fileNum

    # reading the actual saved by recording()
    rec = thinkdsp.read_wave("wav_files/recording" + str(fileNum - 1) + ".wav")

    # instantly delete it because it isn't needed after the data is stored in rec
    rm = "rm wav_files/recording" + str(fileNum - 1) + ".wav"
    subprocess.call(rm, shell=True)

    # implementation of the low pass filter on the absolute value of the samples
    ys_copy = np.abs(rec.ys)
    for i in np.argwhere(ys_copy < 0.2):
        ys_copy[i] = 0.000001

    # taking the average amplitude of only the values that are above the threshold
    cond = ys_copy > 0.000001
    extracted = np.extract(cond, ys_copy)
    score = np.sum(extracted) / extracted.size

    # minimum average amplitude before the program will store the 7 seconds of recording/data
    if score > 0.6:
        print("Score passed!")
        storeRecording()


'''
Function that's called by checkRecording() if the program determines some loud sound 
has been heard. This stores the GPS data, the time data, and the data necessary to 
actually restore the .wav files. 
'''


def storeRecording():
    print("\nStoring right now!\n")
    global AudioRec
    global fileNum

    # This is the system to have a saved system to access the number for the audio file
    # the text file will be constantly updated and saved in order to make the program
    # not have overlapping file names after the program is paused.
    f = open("sampleNumber.txt", "r")
    if f.mode == 'r':
        contents = f.read()
    sampleNum = int(contents)
    f.close()

    # list where GPS data will be saved
    GPS_data = []
    g = geocoder.ip('me')

    # stores the initial coordinates based on ip address of device
    GPS_data.append(g.latlng)

    time_data = time.localtime()
    current_time = time.strftime("%H:%M:%S", time_data)

    duration = 7
    my_rec = sd.rec(int(duration * fs), dtype=np.int16)
    sd.wait()  # waits until the recording is complete before moving on

    # stores the final coordinates after the recording
    GPS_data.append(g.latlng)

    # writing out the .wav file makes the saving a much simpler process
    write("sample" + str(sampleNum) + ".wav", fs, my_rec)

    # create the file that would store the data
    create_file = "touch text_files/file" + str(sampleNum) + ".txt"
    subprocess.call(create_file, shell=True)

    audio = thinkdsp.read_wave("sample" + str(sampleNum) + ".wav")
    ys = ""  # string that will contain all the samples in the audio recording
    ts = ""  # string that will contain all the time stamps for the samples in ys

    for i in audio.ys:
        ys += str(i) + ", "
    for i in audio.ts:
        ts += str(i) + ", "

    txt = open("saved_folder/file" + str(sampleNum) + ".txt", "w")

    # final GPS data that will be stored in the .txt file
    g = geocoder.ip('me')
    GPS_data.append(g.latlng)

    # write all the data into the file
    n = txt.write(str(ys) + "\n" + str(ts) + "\n" +
                  str(GPS_data) + "\n" + str(time_data) + "\n")
    txt.close()

    delete_old = "rm sample" + str(sampleNum) + ".wav"
    subprocess.call(delete_old, shell=True)

    print("\nRecording stored\n")
    print(sampleNum)

    # increment the sample number for the next time the recording data will be stored
    f = open("sampleNumber.txt", "w")
    f.write(str(sampleNum + 1))
    f.close()
    subprocess.call("clear", shell=True)


# THis is the first recording. This allows the first call of checkRecording() to not
# return errors from the lack of an audio file to analyze
recording()
fileNum += 1

# clears the shell so the program looks nicer
subprocess.call("clear", shell=True)


'''
The below lines are a core part of the project. Multiprocessing is a module
that allows users to run multiple functions at the same time. As the board records
the next 0.1 seconds, checkRecording() runs - allowing the board to analyze the last 0.1 
seconds of recording. If you think on the topic a little more, you could realize that 
this means there should be almost no time in which the recording is not analyzed - almost
immediately after it has been recording. Because realtime audio processing must be extremely fast 
(and Python not being known for its speed exactly), this allows for a much simpler program
that has much less need to be optimized. 
'''
while (True):
    if __name__ == '__main__':
        p1 = Process(target=recording)
        p2 = Process(target=checkRecording)
        p1.start()
        p2.start()

        p1.join()
        p2.join()
        fileNum += 1  # the fileNum is increased because the next recording needs a new name

Credits

Avik Ghosh

1 project • 1 follower

Mario De Los Santos

4 projects • 14 followers

Having fun with electronics and programming. Seeed CC

Thanks to Mario De Los Santos and Antony Trivet.

Correlations in Animal Movement and Communication (CAMC)