Automatic monitoring of strength training in the gym

Let's look at how to build a gesture detection system on Nordic5340 + X-NUCLEO-IKS02A1 using TinyML and Edge Impulse.

ML (Machine Learning)-training a mathematical model on some data in order to predict an event or phenomenon on new data. That is, an attempt to force the algorithms of programs to perform actions based on previous experience, and not just on the basis of available data.

For training, you need real data (training sample) and the value of the target variable that corresponds to the specified data. The model observes and finds dependencies between the data and the target variable. These dependencies are used by the model for the new dataset to predict a target variable that is unknown.

What is machine learning like? It is divided into three main types:

With a teacher (Supervised machine learning).

Unsupervised machine learning.

Deep learning.

Deep learning involves the analysis of Big Data – a large array of information and uses neural networks to work. Which requires a lot of computing power.

TinyML is machine learning for microcontrollers, which have modest computing capabilities compared to computers. TinyML is an emerging area where there is still a lot of work to be done. But this will allow ML to be used in billions of microcontrollers in combination with all sorts of sensors, which can significantly change the world of embedded solutions.

We will use the Edge Impulse TinyML online platform to collect data, train the model, and deploy it to the device.

We will add a microcontroller to a regular dumbbell, which will automatically count the number of different exercises performed by a gym visitor and send data to a smartphone. Our dumbbell will count for three exercises.

Since the nRF5340 DK does not have built-in sensors, we use the X-NUCLEO-IKS02A1 board (with a MEMS accelerometer and a MEMS microphone).

We will use the Edge Impulse Tinyxml online platform (https://www.edgeimpulse.com) to collect data, train the model, and deploy it to the device. To get started, you need to create an account by going to https://studio.edgeimpulse.com/signup. Next, enter the profile and create a new project.

The platform supports several devices, one of which is the nRF5340 DK board. The number of devices increases over time, and you can also port other platforms yourself.

To set this device up in Edge Impulse, you will need to install the following software:

• Edge Impulse CLI.
• NodeJs

The development board does not come with the right firmware yet. To update the firmware:

• The development board is mounted as a USB mass-storage device (like a USB flash drive), with the name JLINK. Make sure you can see this drive.
• Download the latest Edge Impulse firmware.
• Drag the nrf5340-dk.bin file to the JLINK drive.
• Wait 20 seconds and press the BOOT/RESET button.

Or download it.hex file in nrfConnect Programmer

From a command prompt or terminal, run:

edge-impulse-daemon

This starts a wizard which asks you to log in and choose an Edge Impulse project.

The device is now connected to Edge Impulse. To make sure of this, go to the Edge Impulse project and select Devices.

Data collection, training, and creating an ML model on the Edge Impulse platform

Now that everything is set up, you can create your own machine learning model to create an exercise recognition system that runs on a microcontroller. This is a difficult task to solve with rule-based programming, since people don't perform the movements the same way every time. But machine learning can easily handle these changes. We will get high-frequency data from real sensors (IMU module), use signal processing to clean up the data, create a neural network classifier, and how to deploy the resulting model on the device.

Data collection first

We connect the board on the computer to the remote management API. On the site, go to the Data asquisition tab. We will collect data for 3 different variants (exercises) - bicepscurl, lateralraise, overheadpress. In the Record new data section, we set the duration values for the fragment of received data, the gesture label. And click on the Start sampling button. During the recording of the fragment, we constantly perform a certain exercise.

We write down several fragments for each exercise. It is important to make continuous movements. For each fragment, we can view raw data RAW DATA.

The model can only learn from what it will see in the training data, so it is important to create a dataset that represents the actual data that the model will see when deployed. If the dataset contains data from only one person, the model learns the exact movement patterns of that person. If the dataset contains data from many people, the model will examine the general characteristics of each exercise that exist in different people.

With a training set, you can build momentum. The pulse takes the raw data, splits it into smaller windows, uses signal processing blocks to extract features, and then uses the training block to classify the new data. Signal processing blocks always return the same values for the same input and are used to facilitate processing of raw data, while learning blocks learn from past experience.

We break the fragments into smaller windows. Setting the window size will depend on the duration of the event that the model learns to recognize. For data on periodic movements, such as repetitions of a dumbbell exercise, the window size should include at least one complete exercise.

The neural network classifies each data sample by inferring a confidence score for each label present in the training data set. The input to the network is an array of functions that is output from the spectral analysis block. The network type and structure can be configured in the NN classifier menu. We set the parameters for training the neural network and start training the neural network.

You can use the Model Testing tool to test the performance of a model across the entire test data set, or across test data packages as needed. It gives an accuracy score based on the expected result for each test data file. We get the test data in the Live classification menu item.

If you are satisfied with the recognition results, you need to generate a code to deploy the model on the device. Select the item nrf5340 and click the Build button. The generated library is downloaded to the computer from the zip archive.

Uploading it.hex file on the board nrf5340 DK

Uploading it.hex file on the nrf5340 DK board.

When performing exercises, the results are output to the serial port. This data will be received via the serial port of the Arduino Nano BLUE and sent to the application on the smartphone.

Creating a program for your smartphone

The smartphone app will be created in App Inventor 2, an

online visual programming editor for Android, which you can access on the page: http://ai2.appinventor.mit.edu. After authorization (you can use a Google profile) or registration, we get to your program profile, where we can create a new project