Hand gesture recognition using low-resolution thermal images

This project aims to develop a human machine interface (HMI) system that utilizes heat signatures to detect and classify hand gestures through machine learning. This system employs a 32×24 pixels low-resolution thermal camera and a low-powered microcontroller to capture thermal images of hand gestures, specifically rock, scissor, and paper. These images are then processed and classified using advanced machine learning algorithms to accurately identify the gestures.

The main goal of this project is to significantly improve user interaction by developing a seamless and contactless method for engaging with devices. This innovative approach will function effectively in various lighting conditions, including complete darkness and low-light environments, ensuring that users can effortlessly navigate and control their devices regardless of their surroundings. The system offers significant benefits, including efficiency due to the low-resolution thermal imaging, which reduces computational requirements while maintaining accuracy. It also provides an intuitive user experience, as natural hand gestures make the system easy to use and learn.

Hardware selection

This project needs a reliable, low-powered, and Wi-Fi enabled device that is also capable of running image classification models. We will be performing TensorFlow Lite model inference using Edge Impulse, and for this purpose, we will utilize the Particle Photon 2, which has ample memory for buffering thermal image data and storing ML models. We are using a thermal camera (MLX90640) with a 32×24 resolution, which is a far infrared array of 768 thermal sensors. Additionally, we are incorporating the Adafruit 2.8″ TFT touch shield for displaying thermal images and basic information during data collection and inferencing.

We used a stackable breadboard and jumper wires to assemble the hardware.

This breadboard has double-sided tie points. The Adafruit TFT display shield is mounted on the front with all connections using the jumper wires, and the Photon 2 and the thermal camera are mounted on the back.

Due to the inherently low sampling frequencies of thermal sensors, excessive movement can result in motion blur in the generated images based on the thermal readings. To mitigate this issue and ensure stable operation, we have opted to utilize a sturdy tripod stand.

We have added a Grove dual button to initiate certain actions during data collection.

Hardware wiring

We are providing clear pin maps for a better understanding of the connections. The Adafruit TFT shield and microSD card are connected via the SPI interface, and its pins are based on the Arduino Uno R3 pinout.

TFT shield & microSD card connections

The thermal sensor is connected via the I2C interface.

Thermal sensor (I2C interface)

The Grove dual button utilizes two digital pins on the Photon 2.

Grove dual button connections

The schematics are provided below.

Data collection

We will use Edge Impulse Studio to train and build a TensorFlow Lite model. We need to register an account and create a new project at Edge Impulse Studio.

We will collect thermal images for three hand gesture classes: rock, scissors, paper, and several background images called unknown. The thermal sensors capture real-time temperature readings, which are then converted into thermal images using color mappings. These images are saved onto a microSD card built into the TFT display, ensuring the data can be conveniently accessed later for training. Prior to uploading them to Edge Impulse Studio, the images are processed to ensure they conform to a uniform size, specifically being padded to become 32×32 pixel square images suitable for training purposes.

For demonstration purposes, the sample images below have been enlarged.

To upload image data, visit the Edge Impulse Data Acquisition page, click on the upload icon, and follow the provided instructions.

We have uploaded 379 images to Edge Impulse Studio to create a balanced dataset.

The complete application code for data collection is available in the GitHub repository at the following link:

GitHub Repository – Data Collection

Data collection demo

Building and training model

We will use Edge Impulse Studio to train and build a TensorFlow Lite model.

To create a machine learning pipeline, follow these steps:

• Go to the Impulse Design section, then select the Create Impulse page. We have opted for a 32×32 pixel image size in the “Image Data” form fields.
• Click on “Add a processing block” and choose “Image”. This step will pre-process and normalize the image data while also giving us the option to choose the color depth.
• Click on “Add a learning block” and choose “Classification”.
• Finally, click on the “Save Impulse” button to complete the process.

On the Image page, choose RGB as color depth and click on the Save parameters button. The page will be redirected to the Generate Features page.

Now we can initiate feature generation by clicking on the Generate features button. Once the feature generation is completed, the data visualization will be visible in the Feature Explorer panel.

On the Classifier page, we can define our model. We have developed a custom Convolutional Neural Network (CNN).

The architecture of this model is illustrated below.

Now, click the “Start training” button. The training process will take a few minutes to complete. Once the training is completed we can see the accuracy and metrics as shown below.

For such a small dataset 98.4% accuracy is pretty good so we will use this model.

Model testing

On the Model Testing page, click Classify All to initiate model testing with the trained model. The testing accuracy is 98.67%.

Model deployment

The Edge Impulse Studio officially supports Particle Photon 2, so on the Deployment page, select Particle Library.

For the Select Optimizations option, choose TensorFlow Lite and Quantized (Int8).

Click Build, and in a few seconds, the library bundle will be downloaded.

Set up Photon 2

Before starting to run the application make sure to set up your Photon 2. Please create a new account at https://login.particle.io/signup, then visit https://setup.particle.io/ and follow the instructions.

Building and flashing firmware

In the library bundle downloaded from the Edge Impulse Deployment page, delete the example main.cpp and add the file provided in the GitHub repository below.

https://github.com/metanav/thermal_hand_gesture_recognition/blob/main/thermal_hand_gesture_recognition_inferencing/src/thermal_hand_gesture_recognition_inferencing.cpp

To build the firmware and upload a binary to the Photon 2, Particle provides the Workbench Visual Studio Code extension. Please follow the instructions provided at the link below.

https://docs.particle.io/quickstart/workbench

• In Workbench, select Particle: Import Project and select the project.properties file from the project directory.
• Use Particle: Configure Project for Device and select deviceOS@5.9.0 and choose a target Photon 2 / P2.
• Compile with Particle: Compile application (local)
• Flash with Particle: Flash application & DeviceOS (local)

When the application starts running, we can view the inferencing logs in the serial monitor. The average inference time is about 90 ms, enabling the Photon 2 to process 11 thermal images per second, which is impressive for a low-powered 200MHz microcontroller

Predictions (DSP: 1 ms., Classification: 91 ms., Anomaly: 0 ms.):
Predictions:
Paper: 0.00000
Rock: 0.00000
Scissor: 0.92969
Unknown: 0.07031

Inferencing demo

Use Case: Controlling an LED with Gestures

An ML model, no matter how sophisticated, lacks practical value without real-life examples to demonstrate its applicability. We developed a demo application that allows users to control an LED remotely using hand gestures. As demonstrated in the code snippets below, we have configured the inferencing application to publish results to the “gestureEvent” topic.

uint8_t max_prob_index;
float max_prob_value;

for (uint8_t index = 0; index < EI_CLASSIFIER_LABEL_COUNT; index++) {
    ei_printf("  %s: ", ei_classifier_inferencing_categories[index]);
    ei_printf("%.5f\r\n", result.classification[index].value);
    if (result.classification[index].value > max_prob_value) {
        max_prob_value = result.classification[index].value;
        max_prob_index = index;
    }
}

if ((prev_max_prob_index != max_prob_index) && (millis() - prev_pub_ms) > 1000) {
    Particle.publish("gestureEvent",
    ei_classifier_inferencing_categories[max_prob_index], PRIVATE);
    prev_pub_ms = millis();
}

We have set up another Photon 2 with minimal code, which subscribes to a topic and turns the onboard RGB LED on and off based on the received messages: “paper” and “scissor”, respectively.

#include "Particle.h"
SYSTEM_MODE(AUTOMATIC);
SYSTEM_THREAD(ENABLED);
SerialLogHandler logHandler(LOG_LEVEL_INFO);

void gestureEventHandler(const char *name, const char *data) {
    String prediction(data);
    Serial.printf("Event: %s Class: %s\n", name, prediction.c_str());
    if (prediction == "Paper") {
        RGB.color(255, 0, 0);
    }
    if (prediction == "Scissor") {
        RGB.color(0, 0, 0);
    }
}

void setup() {
    Particle.subscribe("gestureEvent", gestureEventHandler, MY_DEVICES);
    RGB.control(true);
    RGB.color(0, 0, 0);
}

void loop() {}

The GIF below demonstrates the use case in action.

Published events logs can be viewed at the Cloud Services > Events menu in the Particle Console.

Conclusion

This project marks an advancement in the realm of Human-Machine Interaction (HMI), seamlessly integrating the intuitive nature of low-resolution thermal imaging with the sophisticated capabilities of machine learning. The result is an effective gesture recognition system that is robust and user-friendly, operating on a low-cost and low-powered device. The versatility of this technology allows it to be adapted for various applications beyond the initial scope, such as accessibility for individuals with physical disabilities and integration into gaming systems for gesture-based controls.