The goal of the project is to create the ability to run a recurrent neural network (RNN) model on a Cortex M4 grade CPU, and integrate that functionality into an environmental sensing node to analyze the time series from a sensor on that node. The NXP K64 MCU in the Rapid IoT kit provides a good reference board to try out the concept. The analytics from the RNN can provide the ability to predict the future time series values and thereby the ability to provide early warnings for disaster events.
The exploration started with the web-based IDE for the kit but we switched to the desktop IDE so that we can add the neural network feature as the data came in from the sensors. We developed the Gated Recurrent Unit (GRU) using the available ARM CMSIS NN library that is optimized for the Cortex M series. The GRU code was developed using the FRDM K64 board.
The process consisted of developing the GRU model in an Azure Jupyter notebook using Keras where it was easy to change the model parameters and train the model. The next challenge was to import the weights and bias values of the trained model into a set of #DEFINE statements that could be compiled to create the ARM CMSIS NN GRU implementation for the Cortex M4. The Keras tensors had to reordered and restructured to meet the requirements of the CMSIS NN. That was first tested using a set of Excel macros and then finally coded as a c program that read the .h5 file and wrote the #DEFINE statements into a .h include file that was compiled on the FRDM board. The validity of that include file was tested by writing a small driver to create the same input data on the FRDM board as the GRU model was executed.
The goal for the testing was to be able predict a time-series that can mimics the daily temperature variation which is a single hump curve that repeats every 24 hours. The Keras model showed excellent ability to capture that behavior with just two GRU cells in a single GRU layer that was coupled with a dense layer. The ARM CMSIS NN leverages the q format for storing floats and we modified the code provided by ARM to convert our floating point weights from Keras into the q format numbers for CMSIS.
Then we tested whether the FRDM board generated the "single hump" curve for every 24 data points, and IT DID! That was the most exciting moment in the entire project execution. As far as we know, we are the very first group of people to implement a GRU on the ARM Cortex. The existing work only shows the CNN implementations for image recognition (like the CIFAR cats and dogs stuff). The sample GRU code provided by ARM used some random numbers to populated the weights, and also may have some bugs. We hope to interact with ARM CMSIS NN team to get some clarification.
The last piece of the puzzle was to modify the "example code for weather station" provided by NXP to include the GRU functionality. That was not too difficult after we figured out where in "sensors.c" the call the our GRU code had to be added. We decided to override the temperature reading by getting the "single-hump" data generated by the GRU and send that via bluetooth to the Android Weather Station Demo app (and forward into the cloud). That worked out just fine.
Our cover-page shows the "cloud output" and rest of the hardware.
It turned our the our idea of using GRU for time-series analysis was explored by a Korean team (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5712838/) in 2017 but where the GRU was executed in the cloud. We believe that we are first research team to explore GRU on an edge device.
This work supplements and complements the work done by the NXP team where two models can be trained: SVM (Support Vector Model) or GMM (Gaussian Mixture Model). They used the Rapid IoT Kit for anomaly detection on an electric drill where they mention that : "A few of Rapid IoT's sensors are utilized- gyro/accel/mag, Rapid IoT is trained to send an alert when an out of bound condition occurs, and the same principle of anomaly detection can be applicable for other verticals- e.g. temperature, humidity, air quality for Home and Building automation."
Same comments will apply to the GRU tool developed by our team with the exception that model training will take place externally. On the other hand, it is extremely easy to incorporate our code into the existing Rapid IoT Sensor Examples. There is also a great deal of existing research on using GRU's for speech recognition, and it will be interesting to add a microphone to the Rapid IoT kit and test ability of the kit to do real-time speech recognition.