GloveTalk (Converting Sign Language to Words)

The Problem:

Communication is one of the most fundamental human needs, yet millions of people with hearing or speech impairments face daily barriers in expressing themselves and being understood by others. Traditional sign language, while an effective medium of communication within the community, is not widely understood by the general population. This creates a communication gap, often leading to social isolation, misunderstandings, and limited opportunities in education, healthcare, and employment. There is a pressing need for an affordable, accessible, and real-time solution that can bridge this gap and make interactions between sign language users and non-signers seamless.

The Solution:

Our project introduces a smart glove-based sign language translator equipped with flex sensors and an ESP32 microcontroller to bridge the communication gap. The glove captures hand and finger movements, processes them in real-time, and translates sign language into readable or audible text, enabling seamless interaction between signers and non-signers. Designed to be affordable, portable, and user-friendly, this solution empowers individuals with hearing or speech impairments to communicate more effectively and inclusively in everyday situations.

The Detailed Working with Code:

STEP 1

CONNECT THE THREE FLEX SENSORS TO THE ESP32 AND CHECK WHETHER THEY ARE WORKING OR NOT

MAKE SURE YOU CONNECT TO THE SIMILAR SETUP FOR 3 SENSORS SHOWN CONNECT TO 3 SEPARATE ANALOG TO DIGITAL CONVERTER PINS ...IN MY PROJECT I HAVE CONNECTED TO PIN 33 34 35 SEPARATELY ALSO ADD 10K ohm RESSITOR FOR A VOLATAGE DIVIDER SETUP.

STEP 2

NOW PASTE THE FOLLOWING CODE AND READ THE VALUES FOR 3 FLEX SENSORS

// Pin assignments

const int flexPin1 = 33; // Index Finger

const int flexPin2 = 34; // Middle Finger

const int flexPin3 = 35; // Ring Finger

void setup() {

Serial.begin(115200); // Initialize Serial Monitor

}

void loop() {

// Read analog values from each flex sensor

int flexValue1 = analogRead(flexPin1);

int flexValue2 = analogRead(flexPin2);

int flexValue3 = analogRead(flexPin3);

// Display the values

Serial.print("Flex 1 (Index): ");

Serial.print(flexValue1);

Serial.print(" | Flex 2 (Middle): ");

Serial.print(flexValue2);

Serial.print(" | Flex 3 (Ring): ");

Serial.println(flexValue3);

delay(500); // Wait half a second before next reading

}

STEP 3

NOW CONNECT MPU6050 AND TRY TO READ THE SENSOR VALUES AS SHOWN IN FIGURE

STEP 4

PASTE THE BASIC CODE IN ARDUINO IDE AND RUN THIS

#include <Wire.h>

#include <MPU6050.h>

MPU6050 mpu;

void setup() {

Serial.begin(115200);

Wire.begin(21, 22); // SDA, SCL

Serial.println("Initializing MPU6050...");

if (!mpu.begin()) {

Serial.println("MPU6050 not connected!");

while (1);

}

Serial.println("MPU6050 ready.");

}

void loop() {

float ax = mpu.getAccX();

float ay = mpu.getAccY();

float az = mpu.getAccZ();

float gx = mpu.getGyroX();

float gy = mpu.getGyroY();

float gz = mpu.getGyroZ();

Serial.print("Accel [X: ");

Serial.print(ax);

Serial.print(" Y: ");

Serial.print(ay);

Serial.print(" Z: ");

Serial.print(az);

Serial.print("] | Gyro [X: ");

Serial.print(gx);

Serial.print(" Y: ");

Serial.print(gy);

Serial.print(" Z: ");

Serial.print(gz);

Serial.println("]");

delay(500);

}

STEP 5

COMBINE BOTH CIRCUITS FLEX SENSOR AND MPU6050

AND PASTE THE CODE

#include <Wire.h>

#include <MPU6050.h>

// Flex Sensor Pins

#define INDEX_PIN 33

#define MIDDLE_PIN 34

#define RING_PIN 35

int threshold = 400; // Adjust this for your flex sensors

// MPU6050 Setup

MPU6050 mpu;

void setup() {

Serial.begin(115200);

Wire.begin(21, 22); // SDA, SCL

// Initialize MPU6050

Serial.println("Initializing MPU6050...");

if (!mpu.begin()) {

Serial.println("MPU6050 not connected!");

while (1);

}

Serial.println("MPU6050 connected.");

}

void loop() {

// Read Flex Sensors

int indexVal = analogRead(INDEX_PIN);

int middleVal = analogRead(MIDDLE_PIN);

int ringVal = analogRead(RING_PIN);

// Read MPU6050 Acceleration and Gyro

float ax = mpu.getAccX();

float ay = mpu.getAccY();

float az = mpu.getAccZ();

float gx = mpu.getGyroX();

float gy = mpu.getGyroY();

float gz = mpu.getGyroZ();

// Print Raw Sensor Values

Serial.print("Flex -> Index: "); Serial.print(indexVal);

Serial.print(" | Middle: "); Serial.print(middleVal);

Serial.print(" | Ring: "); Serial.println(ringVal);

Serial.print("MPU -> AccX: "); Serial.print(ax);

Serial.print(" AccY: "); Serial.print(ay);

Serial.print(" AccZ: "); Serial.print(az);

Serial.print(" | GyroX: "); Serial.print(gx);

Serial.print(" GyroY: "); Serial.print(gy);

Serial.print(" GyroZ: "); Serial.println(gz);

// Flex Conditions

bool indexBent = indexVal > threshold;

bool middleBent = middleVal > threshold;

bool ringBent = ringVal > threshold;

// MPU conditions (adjust these ranges)

bool handTiltedUp = ay > 0.8;

bool handTiltedDown = ay < -0.8;

bool handTiltedLeft = ax < -0.8;

bool handTiltedRight = ax > 0.8;

// Combined Gesture Detection

if (!indexBent && !middleBent && !ringBent && handTiltedUp) {

Serial.println("Gesture: HELLO & HAND UP");

}

else if (indexBent && !middleBent && !ringBent && handTiltedLeft) {

Serial.println("Gesture: YES & HAND LEFT");

}

else if (!indexBent && middleBent && !ringBent && handTiltedRight) {

Serial.println("Gesture: NO & HAND RIGHT");

}

else if (indexBent && middleBent && ringBent && handTiltedDown) {

Serial.println("Gesture: THANK YOU & HAND DOWN");

}

else if (!indexBent && middleBent && ringBent) {

Serial.println("Gesture: HELP");

}

else if (indexBent && !middleBent && ringBent) {

Serial.println("Gesture: STOP");

}

else if (indexBent && middleBent && !ringBent) {

Serial.println("Gesture: COME");

}

else if (!indexBent && !middleBent && ringBent) {

Serial.println("Gesture: GO");

}

Serial.println(); // readability

delay(500);

}

ESP32 The Backbone:

The ESP32 serves as the core of our sign language translator glove, handling both sensor data processing and communication. Its multiple GPIO pins allow easy integration of flex sensors to capture finger movements, while its built-in Wi-Fi and Bluetooth enable real-time translation output to external devices such as smartphones or displays. With its low power consumption and compact size, the ESP32 makes our solution both portable and efficient, perfectly suited for a wearable assistive technology.

MPU6050: Bringing Motion Intelligence to Sign Language Translation

• Accurate Motion Tracking: Combines a 3-axis accelerometer and 3-axis gyroscope to precisely detect hand orientation and movement.
• Compact & Lightweight: Small size makes it ideal for embedding in wearable devices like gloves.
• Real-Time Processing: Provides quick response to gestures, ensuring smooth translation without lag.
• Low Power Consumption: Enhances battery life, keeping the glove portable and efficient.
• Versatile Data Output: Works alongside flex sensors to capture both finger bends and overall hand gestures for better accuracy.

Working in Our Project

In our sign language translator glove, the MPU6050 tracks the orientation, tilt, and motion of the hand. While flex sensors measure finger bending to recognize static signs, the MPU6050 captures dynamic gestures such as hand rotations, movements, and positioning. The data from both sensors is processed by the ESP32, which translates the combined information into meaningful text or speech output. This dual-sensor approach increases accuracy and allows recognition of a wider range of sign language gestures.

Flex Sensors: Capturing the Language of Fingers

Flex sensors play a vital role in our sign language translator by detecting the bending of each finger with high precision. When attached to the glove, they convert finger movements into varying electrical resistance, which the ESP32 processes to interpret static gestures. Their lightweight, flexible, and cost-effective design makes them ideal for wearable applications, ensuring accurate recognition of finger positions while maintaining user comfort.

Safety First: Features Ensuring User Protection

Our glove is designed with user safety as a top priority. The electronics are insulated and securely embedded to prevent accidental shocks or short circuits. The system runs on a low-power supply, minimizing any risk of overheating. Lightweight components ensure comfort, while breathable glove material prevents irritation during prolonged use.

Built to Last: Reliability of the System

The combination of ESP32, flex sensors, and MPU6050 ensures stable and consistent performance. With accurate motion and gesture detection, the system provides reliable translations in real-time. The robust hardware design, coupled with efficient error-handling in the software, makes the glove suitable for regular, everyday use without frequent recalibration.

Quick Guide: How to Build It

• Attach flex sensors to each finger of the glove.
• Connect the flex sensors and MPU6050 to the ESP32’s GPIO pins.
• Secure all connections and power the system with a portable battery pack.
• Upload the program to the ESP32 using Arduino IDE or PlatformIO.
• Test the glove with simple signs to verify accuracy.

Under the Hood: Hardware & Software at a Glance

Hardware:

• ESP32 microcontroller
• Flex sensors (finger bending detection)
• MPU6050 (motion & orientation tracking)
• Glove, jumper wires, battery pack

Software:

• Arduino IDE for coding and uploading
• Libraries for sensor integration (Wire.h, Adafruit, MPU6050)
• Code logic for gesture recognition and translation into text/speech

Configuration Steps

• Install Dependencies: Download and install the ESP32 board package and necessary libraries in Arduino IDE.
• Connect Hardware: Ensure flex sensors and MPU6050 are correctly wired to the ESP32.
• Upload Code: Flash the program onto the ESP32 via USB.
• Calibrate Sensors: Perform initial calibration for flex sensors and MPU6050 to improve accuracy.
• Test Output: Connect to a display or smartphone via Wi-Fi/Bluetooth and verify translations.