Machine Learning with STM MEMS Sensor

Machine Learning in Sensor Silicon

STM had introduced one of their popular MEMS sensors (accelerator / gyro) with an extensions for machine learning (ML). This part is the LSM6DSOX a iNemo sensor that can combine the sensor results into a decision-tree, learned by machine learning.

What Does This Bring You?

Normally accel/gyro sensor data is read out by a MCU, doing all kinds of algorithmic calculations on the dataflow to decide what type of movement is ongoing. Sometimes this is complex and requires even FFT's, but most times it is slow and simple movements, that can be learned by machine learning... so decision making WITHOUT MCU procesing - WOW !!

The LSM6DSOX ML works as follows:

The sensor-chip is monitoring (filtered or raw) data of the sensors in a time window. In this time window it is able to calculate Features : simple specific data sets that are attributes of the signal in the time window. These Features are by example Mean- or Variance- calculation, Min/Max, Peak-Peak or Energy (for each axis and for the 3D-vector). It is even possible to calculate number of peaks or number of zero-crossings, to give a a time-window its key-features. These 'Features' are input to a decision tree, that calculates the expected pre-trained movement or can generate an interrupt. The decision trees are limited to max 256 nodes, but give descent results.

The Sensor has more interesting features that go beyond the scope of this demo, but worth to explore: a finite state machine: see YouTube video

Hardware Setup for Sports Demo

In this example we use sports-movements like curls, boxing, karate or frisbee-throwing (is this a sport??) and will train the decision tree for the right settings.To do so, we have two hardware components used: Generic MEMS platform eval-board called MKI109V3. This board has a STM32 processor that interfaces programs and monitors the sensor under test. Sensors under test are mounted on the DIL-PCB, and we use the STEVAL-MKI197V1 to evaluate the LSM6DSOX MEMS sensor via the DIL socket on the board. Please visit the links and download the required Datasheets and AppNotes.The Eval boards are stacked and for this demo we attach them to a glove. I used a Bike-sports glove and knitted the board on top of it :

Now you are almost ready to start testing the machine learning core :)

Sensor Tools: Unico & Weka

To train and program the sensor there are two software tools required: Unico (STM tool for controlling the MKI197V3 Mems board) and Weka, a Datamining Tool build by the University of NewZealand to generate your decision tree.

Important! Before continuing, play the following video demo of STMicro and University of Waikato that learn you how the tools work

Train yourselves:

Simple LSM6DSOX demo: YouTube
Weka preprocessing: YouTube
Weka documentation: Online

Important !!: Regarding Unico: program has a bug when encountering virtual usb ports with a high COM-number (ie COM30, Com70 etc). These are nor recognized by the program and you have to deselect automatic Port Detection and connect the board manualy in the tool.

This costed me half a day upgrading and debugging the program as the error message is not correct: it tells you to upgrade the firmware, which does not help at all.... :(

Training the Sports Movements

Now you have done the simple exercises of Unico and Weka, you are ready to make the sporty-part. For this example I created two decision trees: One that indicates the type of movement, and one that indicates the speed (pace) of movement. The Unico tool is a bit limited in the Data-base handling: it limits your to put only one class a time at a data-base set, and it also has no save-possibility for generating the ARFF file. (See the Zip-file of this project for the generated files.) To generate learning data we make use of the 'load/save' tab in Unico. Remember you create the name of the data file BEFORE you record it ( Unico streams data direct to the file). We use the following settings of the sensors:

• Accelerometer: 52Hz, 4g
• Gyrometer: 52Hz, 1000dps

Generate sets of data of minimum of 1 minute movement (est. 60 sample windows). Do this for Hoovering (no action) and the sports movements Boxing, Karate, Curls and Frisbee movement, the last ones two sets: one slow, one fast, So you have 9 sets of training data:

• Hoovering (1 set)
• Boxing (2 sets, one slow, one fast)
• Frisbee (2 sets, one slow, one fast)
• Karate (2 sets, one slow, one fast)
• Curls (2 sets, one slow, one fast)

Once data-sets are ready, load them with the right class: Hoovering, Boxing, Karate, Frisbee, Curls (each with slow and fast), plus add the sets for Fast and Slow, re-using the sets already created, but with class fast and slow.

For the ARFF generation select the following features:

• Time window: 2 seconds, so 104 samples
• Key features: PeakToPeak, MEAN, ENERGY, all signed - for all sensors Axis incl V (vector)
• 2 decision trees, both with same results:

Using Weka

Once the ARFF is ready and saved there are 2 ways to create the 2 decision trees:

1. Open/Edit the ARFF file and make 2 files, one with 'pace' (hoovering, fast, slow, delete the rest), second with 'type; (hoovering, boxing, frisbee, karate, curls, delete the rest). run each ARFF in Weka and copy the two trees to the weka.txt files to load back into Unico.

2. Load the full ARFF in Weka, delete the classes your dont need to use ('pace'-tree: hoovering, fast, slow, and 'type'-tree : hoovering, boxing, frisbee, karate, curls, ) and generate the the J48 trees.

We follow this second option. Open Weka Explorer and open the ARFF file. With the pre-processing filter unsupervised.instance.RemoveWithValues it is possible to delete the classes you don't need (do this 2 times for the decision 2 trees):

With the prepared data-sets, select in the classify tab the J.48 tree, and run it with a Cross-Validation with Folds of 10 (default setting). This gives already a 98%+ result, and would be acceptable for this demo. Stunning part is that the trees are pruned and small:

• type-of-sports: 98%, 9 Nodes, using only 4 feature sets
• pace-of-sports: 99%, 5 nodes , using only 2 feature sets

In case better or more accurate performance is required, options are to tweak Weka a bit more (less pruning or filter the dataset), or to generate more data samples to create a more accurate J48 calculation.

Testing the Results

Copy past the tree's to a weka.txt file, and load the decision trees back in Unico, and generate the UCF file. Now test the UCF file by loading it into the sensor. This is the result made with this quick demo: