THEIA - Tracking Heuristic. Evaluation Intelligence Analyzer

Determining the movement activity of people, pets and even devices throughout the house can lead to innovations with home security and home automation. Situational awareness is key when it comes to protecting your home or needing to automate parts of the home for everyday use.

THEIA is an IoT device that uses artificial intelligence at the edge to classify objects as the pass through the PSOC6's RADAR field of view. In real-time, THEIA will collect, preprocess and down sample the BGT60 signal and provide the data to a custom model to identify the activity in the scene. Once as classification of significant confidence is made, THEIA will report the results to a client for further action. THEIA is mounted and trained in a doorway setting.

THEIA can also detect smaller, faster objects, like a ball.

Design

The XENSIV BGT60 radar sensor is an extremely capable device with the ability to produce radar measurements at various fidelity. Using measurements from the BGT60, THEIA will collect and down-sample the data for model training. Being that the BGT60 is capable of producing a high volume of data, the data needs to be down-sampled, and pre-processed before sending to the DEEPCRAFT machine learning pipeline. THEIA will average the 16 chirps in a radar frame, and select the first 8 samples. Using the higher data capture rate of the BGT60, this leads to 8 channel, 26 Hz. data. This data is timestamped and sent over UDP to a recorder for import into DEEPCRAFT (see software below).

Data

While this approach would work for any number of types of objects, given the short time frame we chose to limit our data set to; Person and Ball. Recordings were take of a person passing in and out of the sensors field of view at roughly.5 Hz. Also, similar readings were taken using a standard sized volleyball. In total. The following describes the data used to train the model in DEEPCRAFT.

Model

Many different models were considered. Given the relatively small dataset, we chose to use a CONV1dLSTM small model. LSTM is ideal and preferrable given we'd like to extend this tool to not only classify types of objects passing, but also direction and speed in which they are passing. The model results can be summarized as follows:

Architecture

There are 3 different applications to THEIA; theia-rec, theia-inf and theia-client. The applications work together to train, run inference and processes inference results.

THEIA-REC

This is a set of 2 applications that records and down samples data to be to used in training. The theia-rec application is a ModusToolbox application that uses FreeRTOS, the XENSIV drivers and UDP drivers to collect radar samples, downsample and send over UDP to a python script that writes the data in the format expected by DEEPCRAFT. All of the code is available in github. Here is the python script to record DEEPCRAFT samples:

import socket
import struct
import lz4.block
import time
import csv
import os
import math
import threading
import sys
import pathlib


# Define the server's address and port
SERVER_ADDRESS = ('10.0.0.62', 57345)

SAMPLE_DEPTH = 8
# Create a UDP socket
client_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

try:
    print(f"Starting capture {time.time()}")
    capturing = False
    rec_time = 0.
    capture_cnt=0

    message = sys.argv[1]
    label = sys.argv[2]
    client_socket.sendto(message.encode('utf-8'), SERVER_ADDRESS)
    client_socket.setblocking(0)
    print(f"Sent: {message}")
    # Receive response from the server
    out_dir = f'./data'
    new_dir = f'{out_dir}/live/{label}/{round(time.time(),5)}/'
    pathlib.Path(new_dir).mkdir(parents=True, exist_ok=True)
    h = ['# Time (seconds)']
    h.extend([f'f0_f0_f{_n}' for _n in range(SAMPLE_DEPTH)])
    time_step = 0.



    with open(f'{new_dir}/chirp.data','w') as f:
        w = csv.writer(f)
        w.writerow(h)
        t = 0.
        last_capture_time = None
        while 1:
            try:
                data, server_address = client_socket.recvfrom(2048)
                print ('recv d ')
                num_samples = int(len(data)/4) - 1
                samples = list(struct.unpack(f'<L{num_samples}f',data))
                capture_time = samples[0]

                if last_capture_time is None:
                    last_capture_time = capture_time
                    continue
                samples = samples[1:]
                ms_per_sample = (capture_time - last_capture_time) / (num_samples / SAMPLE_DEPTH)
                print ('data rate', (len(samples)/SAMPLE_DEPTH)/((capture_time-last_capture_time)/1000.))

                for _n in range(0,num_samples,SAMPLE_DEPTH):
                    time_step += ms_per_sample
                    n_time = last_capture_time + (_n * ms_per_sample/1000.)
                    w.writerow([time_step/1000.] + samples[_n:_n+SAMPLE_DEPTH])
                last_capture_time = capture_time

            except socket.error as e:
                if e.errno == 11 or e.errno == 10035:
                    continue
                else:
                    print ("exiting ", e)
                    time.sleep(.01)
                    continue

finally:
    # Close the socket
    client_socket.close()

THEIA-INF

This is the application that runs the THEIA inference. After a model has been generated in DEEPCRAFT (awesome site by the way, makes difficult tasks a breeze), the model is exported to a ModusToolbox application to be used for inference (see here for details on how to do this with DEEPCRAFT). As inferences are made, theia-inf will report the results over UDP to be processed by theia-client. Of course any UDP willing device could and would be used to achieve a more robust home automation or home security solution. The model inference is achieved by the following (see github for the full source):

//Average Chirps
arm_fill_f32(0, avg_chirp, NUM_SAMPLES_PER_CHIRP);

for (int chirp = 0; chirp < P3_XENSIV_BGT60TRXX_CONF_NUM_CHIRPS_PER_FRAME * 2; chirp++)
{
  arm_add_f32(avg_chirp, &radar.bgt60_send_buffer[NUM_SAMPLES_PER_CHIRP * chirp], avg_chirp, NUM_SAMPLES_PER_CHIRP);
}

for (int i = 0; i < NUM_SAMPLES_PER_CHIRP; i++)
{ 
  avg_chirp[i] *= 1.0f / P3_XENSIV_BGT60TRXX_CONF_NUM_CHIRPS_PER_FRAME;
}
//Enqueue the results to be used by ML inference model
imai_result_enqueue = IMAI_enqueue(&avg_chirp[sample_offset]);

if (xensiv_bgt60trxx_start_frame(&radar.bgt60_obj.dev, false) != XENSIV_BGT60TRXX_STATUS_OK)
{
    printf("start error\n");
}
int imai_result = IMAI_dequeue(model_out);
if (imai_result!= IMAI_RET_SUCCESS){
  printf("doh!!! dequeue error\n");
}
            
if (xensiv_bgt60trxx_start_frame(&radar.bgt60_obj.dev, true) != XENSIV_BGT60TRXX_STATUS_OK)
{
  printf("start frame error\n");
}

THEIA-CLIENT

The theia-client is a toy Arduino studio application that runs on the AWS M5Stack to process theia-inf results. The application listens for UDP messages from thia-inf and displays the information to the user (of course you could do many different things with this information like notify or otherwise react).

Conclusion

We had a lot of fun working on THEIA and we believe there are many interesting use cases especially with respect to home automation and home security. If you made it this far, thank you! You're a true hackster and we'd love your feedback.