Advanced Hand gesture detection - Mudra detection

Overview

In recent years, FPGAs have become more popular based on the benefit of power efficiency and high throughput processing power compared to CPU for Machine learning tasks.
In Human History, hand gesture has been used and hand gesture recognition by computers started with glove-based  control interface[1]. While many projects successfully recognize single hand gestures, there is an opportunity we can push the boundary to recognize multiple complicated hand gestures within a timeframe such as mudra.
This project can benefit those who are disabled, by enabling them to translate the mudra into voice.

Goals

• Image recognition using FPGA: Grant the device to have ability to process image iin outdoor/indoor environment, such as where or when the device should process mudra.
• Hand gesture recognition using FPGA: Grant the device to have ability to process high correctness of hand gesture.

Current this project only include the APL hand sign for demonstration purpose.

Dataset include:

1. Dynamic Hand Gesture 14/28 dataset

2. SHREC 17 Track Dataset

3. Sign Language MNIST

4. ASL Alphabet

Reference:

1. Deploying Edge Impulse Embedded ML on the Xilinx Kria SoM FPGA

2. Classification of the American Sign Language using Pytorch

In this post, I am using edge impulse to training my dataset to fit it into KV260.

Step 1: Input data into edge impulse

Step2: Create impulse

Impulse works like a pipeline that enable you to configure input source like image size, processing block and learning block. The output feature will populate according to the data you label.

Step 3: Generate feature

Step 4: Transfer learning

You are one step closer to the end of training!
In this steps, there will have 2 epoch on going. First is training epoch. second is FINE_TUNE_EPOCHS. Due to my large data set, I must tune down the epoch to complete the training ( since it is free developer account.)

Last step: Deploy the model to KV260!

You will need to follow the instruction from Porting Edge Impulse onto the Kria KV260 Vision AI Kit for petalinux, or Ubuntu for Ubuntu 20.04.3 Desktop on Kria KV260 with Edge Impulse

Final word

Although this project is not completed as I wish, I really learnt and appreciate what edge impulse has enable developer to do amazing thing with FPGA.

That's it for this post. Thank you!