Published December 16, 2022 © GPL3+

Sensor glove for sign language translation

A sensor glove that converts hand gestures to text and speech, which is delivered through a Bluetooth-connected Android Application.

IntermediateShowcase (no instructions)5,556

Sensor glove for sign language translation

Things used in this project

Hardware components

Arduino Nano R3

HC-05 Bluetooth Module

Adafruit Analog Accelerometer: ADXL335

Alphanumeric LCD, 16 x 2

Optional

Jumper wires (generic)

SparkFun Flex Sensors

Resistor 10k ohm

You can use between 10k and 47.5k ohm resistors. Note that every different resistor value results in different analog values you read from the voltage divider.

Solderless Breadboard Full Size

Optional but recommended

USB-A to Micro-USB Cable

Depending on the connector on your Nano, use the preferred scale for USB connection to your computer

Inertial Measurement Unit (IMU) (6 deg of freedom)

I say go for this instead of the accelerometer due to better features for position and movement sensing accuracy. This means though, you will have to find a way to incorporate it yourself.

Software apps and online services

MIT App Inventor 2

Arduino IDE

Hand tools and fabrication machines

Soldering iron (generic)

Solder Wire, Lead Free

PCB Holder, Soldering Iron

Hot glue gun (generic)

Story

INTRODUCTION

As the title suggests, this device is a sensor-based wearable that converts simple hand gestures to text and speech. I decided to make this a project in university.

METHODOLOGY

For this project, the tools and technology that will be used are an Atmega328p microprocessor, Bluetooth (Bluetooth Module), Flex sensors, Accelerometer, Power module, Resistors, and Text-to-speech software (Bluetooth terminal app). The flex sensor measures the bending of fingers according to gesture and outputs changes in resistances corresponding to the amount of bending. The Accelerometer sensor measures the linear movements of the hand on the X, Y, and Z axes and outputs different values of X, Y, and Z corresponding to the movement on these axes.

All the data from sensors are then processed by the Arduino Nano, which involves IF/ELSE, AND, and OR logical combinations of all the sensor outputs in order to match the resultant output with pre-stored values of different signs regarding the alphabets. For this, appropriate ranges are set for each alphabet and the words that can be recognized with a single hand based on the measured data obtained from repeated measurements. The HC-05 Bluetooth module is connected to Arduino NANO. The processed data are then transferred to the Bluetooth module (transmitter) in string format. The Android mobile also has an inbuilt Bluetooth capability. These two Bluetooth devices are then paired, and the string is transmitted to the Android application.

The Android application receives data via Bluetooth in bytes format and then converts them into a string. Finally, the string is converted into a voice using the text-to-speech application of Android mobile. This overall system is mounted over a normal glove for easy handling and recognizes hand gestures accurately.

Figure 1, Hardware Case Diagram

Figure 2, Software Case Diagram

Figure 3, Flowchart of sensor glove operation

Figure 4, Flowchart of Android App Opreration

The main principles behind the design of this system are change in resistance, acceleration due to gravity, and real-time systems.

The relevance of resistance is to how the flex sensor works. The more the flex sensor is bent, the greater the resistance. The conductive ink printed on the sensor acts as a resistor. When the sensor is straight, this resistance is about 25k. The conductive layer is stretched when the sensor is bent, resulting in a reduced cross-section. This reduced cross-section results in increased resistance. At a 90° angle, this resistance is about 100KΩ.

When the sensor is straightened again, the resistance returns to its original value. One can determine how much the sensor is bent by measuring the resistance. The easiest way to read the flex sensor is to connect it with a fixed value resistor (usually 10 or 47 kΩ) to create a voltage divider as shown in figure 5 To do this, one end of the sensor is connected to Power and the other to a pull-down resistor. Then the point between the fixed value pull-down resistor and the flex sensor is connected to the ADC input of an Arduino. This way, one can create a variable voltage output, which can be read by Arduino’s ADC input.

Figure 5, Voltage Divider

Bend sensing module

On the glove, 5 flex sensors are fitted for the 5 different fingers. The flex sensors change resistance depending on the amount of bend by the fingers. Pin 1 of the flex sensor is connected to the 3.3v on the Arduino Nano, and then, Pin 2 is connected to a 10k ohms resistor, which acts as a voltage divider. The point between the pull-down resistor and the flex sensor is connected to the analog pin on the Arduino. Figure 6 shows the bend sensing module.

Figure 6, Bend sensing module

Position sensing module

The accelerometer (ADXL 335) in this system is used as a tilt detector. It has an analog output that varies from 1.5 volts to 3.5 volts. ADXL335 is a three-axis analog accelerometer Integrated circuit (IC), which reads X, Y, and Z-axis acceleration as analog voltages. By measuring the amount of acceleration due to gravity, an accelerometer can figure out the angle it is tilted with respect to the earth. By sensing the amount of dynamic acceleration, the accelerometer can find out how fast and in what direction the device is moving. The basic function of this device is to detect the tilting of the hand and send relevant data according to meaningful gestures, to the microcontroller which receives the data and saves them. Figure 7 shows the interfacing of the Arduino nano with the accelerometer to produce the position-sensing module.

Figure 7, Position sensing module

Transmission module

The Bluetooth module is responsible for sending output data to the mobile app, enabling text-to-speech conversion, and comprises the HC-05 Bluetooth and the Arduino microcontroller. The microcontroller processes what phrase has been represented and sends that data to the HC-05 module to send the data to the android device which is connected to it through Bluetooth. Figure 8 shows the transmission module.

Figure 8, Transmission module

NOTE: I strongly advise soldering with care as the flex sensor pins are very delicate. You determine how you want the layout to be because mine was a bit messy and I would not advise anyone to do it my way. You may choose to use a perf board or not, just ensure that all the components are connected properly and securely to the glove.

I used a USB cord to the Arduino Nano as my power source, but you may decide to use a battery. Figure out how to use a battery yourself, it is not very difficult. I chose not to use a battery due to insufficient space on the device and inconvenience.

If you are wondering why my prototype has only 3 flex sensors, 3 go damaged during soldering, that is why I advise that you be careful when soldering. The flex sensor on the middle finger also got damaged but I did not notice until I already soldered it.

Ideally, an Inertial Measuring Unit (IMU) is of much better use in this project, but due to a few annoying factors, I could not get one in time.

If you have any other inquiries, please feel free to mention them in the comment section.

When you are done with soldering, the final product should look something like this:

Figure 9, Prototype

As for the Android app, you may choose to create one for yourself using MIT app inventor 2 or any other tool you are proficient with as an Android developer. If you want the one I created, please mention so in the comments because you will have to talk to me directly. The application looks like this:

Figure 10, Android Application

Code

Code for Operation

#include <SoftwareSerial.h>
SoftwareSerial mySerial(2, 3); // RX, TX

  const int flex1 = A0;
  const int flex2 = A1;
  const int flex3 = A2;
  const int xpin = A3;                  // x-axis of the accelerometer
  const int ypin = A4;                  // y-axis
  const int zpin = A5;                  // z-axis (only on 3-axis models)


void setup() {
  // put your setup code here, to run once:
 Serial.begin(9600);
 mySerial.begin(9600);
 delay(15000);
}

void loop() {
  // put your main code here, to run repeatedly:
  int x = analogRead(xpin);
  delay(50);

  int y = analogRead(ypin);
  delay(50);

  int z = analogRead(zpin);
  delay(50);
  int thumb;
  int pointer;
  int middle;
  int nomx;
  int nomy;
  int nomz;
  
  thumb = analogRead(flex1);
  pointer = analogRead(flex2);
  middle = analogRead(flex3);
  nomx = x-265;
  nomy = y-253;
  nomz = z-323;

 
  Serial.print(thumb);
   Serial.print("\t");
  Serial.print(pointer);
   Serial.print("\t");
  Serial.print(middle);
   Serial.print("\t");
  Serial.print(nomx);
  Serial.print("\t");
  Serial.print(nomy);
  Serial.print("\t");
  Serial.print(nomz);
  Serial.print("\n");
  Serial.println("");
  delay(500);


   //group1
  if ((thumb > 175)  && (thumb < 210) && (pointer > 210) && (pointer < 235) && (nomy > 75) && (nomy < 95))  {
    mySerial.print("LOOK FORWARD");
    }
    else if ((thumb > 175)  && (thumb < 200) && (pointer > 210) && (pointer < 235) && (nomy > 160) && (nomy < 175))  {
    mySerial.print("LOOK DOWN");
    }
    else if ((thumb > 175)  && (thumb < 200) && (pointer > 210) && (pointer < 230)&& (nomy > 20) && (nomy < 35))  {
    mySerial.print("LOOK UP");
    }

  //group2
   else if ((thumb > 160)  && (thumb < 195) &&(pointer > 160) && (pointer < 200) && (nomy > 28) && (nomy < 43))  {
    mySerial.print("OKAY");
  }
  else if ((thumb > 190)  && (thumb < 210) && (pointer > 180) && (pointer < 200) && (nomy > 160) && (nomy < 180))  {
    mySerial.print("HOLD");
    }
    else if ((thumb > 160)  && (thumb < 200) &&(pointer > 160) && (pointer < 190) && (nomy > 60) && (nomy < 75))  {
    mySerial.print("PRONTO");
  }

   //group3
    else if ((thumb > 230)  && (thumb < 250) && (pointer > 150) && (pointer < 180) && (nomy >80) && (nomy < 100))  {
    mySerial.print("CALL ME");
    }
    else if ((thumb > 230)  && (thumb < 250) && (pointer > 150) && (pointer < 180) && (nomy >105) && (nomy < 125))  {
    mySerial.print("THUMBS DOWN");
    }
    else if ((thumb > 230)  && (thumb < 250) && (pointer > 150) && (pointer < 189) && (nomy >58) && (nomy < 70))  {
    mySerial.print("I LOVE YOU");
    }
    else if ((thumb > 230)  && (thumb < 250) && (pointer > 220) && (pointer < 235) && (nomy > 80) && (nomy < 105))  {
    mySerial.print("GIVE ME");
    }
    else if ((thumb > 175)  && (thumb < 195) && (pointer > 170) && (pointer < 190) && (nomy > 90) && (nomy < 110))  {
    mySerial.print("GIVE ME SOME");
    }     
  else{
    mySerial.print("...");
  }
  }

Credits

ayooluwa98

1 project • 5 followers

Sensor glove for sign language translation

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

Bend sensing module

Position sensing module

Transmission module

Schematics

Circuit diagram

Circuit Schematic

Code

Code for Operation

Credits

ayooluwa98

Comments

Embed the widget on your own site

Sensor glove for sign language translation

Sensor glove for sign language translation

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

Bend sensing module

Position sensing module

Transmission module

Schematics

Circuit diagram

Circuit Schematic

Code

Code for Operation

Credits

ayooluwa98

Comments

Related channels and tags