Published November 30, 2023 © CC BY

Gaming with Disabilities: Face Tracking Game Controller

An attempt to develop a Gaming Controller for Persons with Disabilities, that works with movement of face/head to control movements in games

IntermediateWork in progress6 hours718

Honorable Mention Prize Gaming

Build2gether Inclusive Innovation Challenge

Lucky Draw — 3 Submissions

Build2gether Inclusive Innovation Challenge

Things used in this project

Hardware components

Useful Sensors Person Sensor

SparkFun I2C 4 wire JST connector for Person Sensor

DFRobot Beetle - The Smallest Arduino

SG90 Micro-servo motor

DIYables OLED Display 128x64 0.96 inch, I2C Interface

SparkFun LilyPad LED White (5pcs)

Seeed Studio 5mW Laser Module emitter - Red Point

Photodiode, 45 °

Seeed Studio Grove - Vibration Motor

2n2222 NPN Transistor

Resistor 1k ohm

Resistor 100 ohm

Capacitor 100 µF

Software apps and online services

Arduino IDE

Hand tools and fabrication machines

Soldering iron (generic)

Story

Introduction

Gaming can be challenging for people with disabilities. Even playing simple games can be difficult for them, as most gaming controller are not designed with their needs in mind.

In this project, I will attempt to develop a gaming controller which can be used with face/head movement, it's handsfree design, minimal input for playing simple online games/in browser games.

Demo Video

Movement Control with Face

Face tracking on OLED Display with Emoji !

Theory of Operation

In a nutshell, here is how this device works:

Find the face of a person with Person Sensor
Sense the face position in the Field of View plane over I2C
Use a microcontroller to emulate keyboard as USB HID
Send key pressing input to Host computer to control games

Final Prototype Hardware with Person Sensor

Person Sensor is a camera module with a pre programmed microcontroller with computer vision face recognition model.

Hardware Build

The prototype is build according to the following schematic

Schematic

Schematic

Prototype Hardware

This is the actual hardware prototype

1 / 2 • Hardware

Hardware Description (Bill of Materials)

Person Sensor Useful Sensor: Person sensor as the name suggests, is a person's face detection sensor using computer vision. It is preprogrammed with face detection algorithm, can track face movement in it's field of view. Person sensor does not allow access to raw image data, instead it returns information over I2C interface about face detected or not, number of faces, face position etc.

That's why it's easy to use with microcontrollers without knowing details about computer vision or face recognition AI.

White LED Lamps with Diffusers: Person sensor requires enough lighting on the user's face to work properly. That's why I added two white LEDs with diffusers. They are driven with a PWM pin to adjust the brightness.

Make sure there is enough light on the face of the person or the camera will lose face tracking

Beetle Board (Atmega32U4 mcu):This is a very small size Arduino Leonardo equivalent board with Atmega32U4 microcontroller. It is used to drive the OLED display1306, read Person Sensor data over I2C interface, drive the servo motor angle, drive the LED lamps brightness, sense the Photo Transistor beam interruption with ADC, drive the vibration motor, emulate HID USB keyboard on host PC and send press key commands based on face position in front of person sensor. Basically this mcu runs the whole device.

OLED Display: A small, low cost, I2C based OLED display for user interface with 1306 display controller. This display is used to show the face position in the field of view, if no face is detected it will also inform user about that.

Debugging while programming

Adjusting camera tilt and Lamp Brightness for better results

I found this display very useful tool for debugging the code while working with person sensor.

Dot Laser Diodes:These are tiny LED based semiconductor laser diodes that can produce monochrome beam of Red light. Laser beam is used here to shine on Photo Transistors (LDR or Diode will work too). The plan is to develop two "Zero Pressing Force" Air Buttons. Air buttons are just interruption of Laser beams with finger or any object to sense user input.

Photo Transistor: Two photo transistors are used in this device to make to Air Tap buttons (basically interrupt the beam to push switch). These photo transistors are connected to two ADC inputs of the microcontroller. When the laser beam is interrupted, then the transistors are nearly turn off causing ADC to sense Gnd potential. This change is registered as user 'button press'

Vibration Motors: I have added two vibration motors for haptic feedback, since the optical interrupt buttons (the Laser beam and Photo Transistor getting interrupted by finger or object) don't have any force feeling, haptic will give the feeling of something happening.

Servo Motor: The purpose of this servo motor is to align the camera initially with the face in center position. While the hardware is connected, I have not implemented this feature yet in the firmware.

Programming and Firmware

Download Arduino IDE

Download the latest Arduino IDE from here. IDE version 2.1 and latest are recommended for better compatibility but any version from 1.8+ or above will work.

Download Arduino

Arduino IDE

Select "Arduino Leonard" from

Tools > Board > Arduino AVR Board > Leonardo

Microcontroller Board Selection in Arduino IDE

Add Libraries to Arduino IDE

Following libraries are necessary for this project, please make sure you have included them in Arduino IDE. Details already available on GitHub links

U8G2 Library for Display Driver

Useful Sensor Library for Person Sensor

Next, compile and upload the code *attached below) into Beetle board and get ready to test the concept.

Testing

Here are some of the testing videos, yep, you guessed it ! There is a bit of lag because of 7 Hz only polling rate of the camera on face tracking !

Face movement of user as Input Control

Still, the idea is what I am trying to demo here, implementing this idea on a single board computer with at faster camera can solve the lag !

Face tracking by person sensor

Keyboard press sent by Microcontroller (emulated as USB keyboard)

References

Conclusion

This project is far from complete. I am rushing this at the last moment. The firmware is still under development. Instead of considering this a finished project, think of it as a prototype concept.

I want to make more complex gaming controller that can take other forms of input from user like making certain sounds with mouth (and detect them with sound recognition) as game controlling input. Developing gaming controller for people with disabilities is not easy, as everyone's need is different.

Let's keep exploring new ideas and try them out, something wonderful might happen and we will have the ultimate gaming controller that everyone can use

Code

// Include libraries for person sensor, OLED display, HID keyboard etc

#include <Wire.h>
#include <Arduino.h>
#include <Servo.h>
#include <U8g2lib.h>
#include <Keyboard.h>
#include "person_sensor.h"

// keyboard key short defination

#define up       KEY_UP_ARROW
#define down     KEY_DOWN_ARROW
#define left     KEY_LEFT_ARROW
#define right    KEY_RIGHT_ARROW


const int32_t SAMPLE_DELAY_MS = 50;


int x = 0;
int y = 0;
int x_offset = 15;
int y_offset = 47;
int pos = 0;
uint8_t lamp_drive_pwm =50;

// I2C OLED display object
U8G2_SSD1306_128X64_NONAME_F_HW_I2C u8g2(U8G2_R0, /* reset=*/ U8X8_PIN_NONE, /* clock=*/ 16, /* data=*/ 17);   // ESP32 Thing, HW I2C with pin remapping
// create servo object to control a servo for cam movement
Servo camSweepServo;  

void setup() 
{
  // safety delay
  delay(5000);
  camSweepServo.attach(9);  // attaches the servo on pin 9 to the servo object
  
  for (pos = 75; pos <= 105; pos += 1)
  { 
    camSweepServo.write(pos);   // tell servo to go to position in variable 'pos'
    delay(15);                  // waits 15 ms for the servo to reach the position
  }
  
  for (pos = 105; pos >= 75; pos -= 1) 
  { 
    camSweepServo.write(pos);        // tell servo to go to position in variable 'pos'
    delay(15);                       // waits 15 ms for the servo to reach the position
  }
  pos = 90;
 camSweepServo.write(pos);
 
  // You need to make sure you call Wire.begin() in setup, or the I2C access
  // below will fail.
 
  Wire.begin();
  u8g2.begin();

  // these pins drive 2 vibration motors for haptic feedback (tba)
  pinMode(10,OUTPUT);
  pinMode(A0,OUTPUT);
  digitalWrite(10,0);
  digitalWrite(A0,0);

  // this pin outputs PWM to adjust the brightness of the LED lamps
  pinMode(11,OUTPUT);

  // 5 step brightness increase
  analogWrite(11,20);
  delay(200);
  analogWrite(11,40);
  delay(200);
  analogWrite(11,60);
  delay(200);
  analogWrite(11,100);
  delay(200);
  analogWrite(11,150);
  delay(300);

}

void loop() 
{
  // readjust the brightness according to user perference if needed
  
  analogWrite(11,lamp_drive_pwm);

  // poll person sensor and put in an array

  person_sensor_results_t results = {}; 
  if (!person_sensor_read(&results)) 
  {
    delay(SAMPLE_DELAY_MS);
    return;
  }
   const person_sensor_face_t* face = &results.faces[0];

  // clear OLED display buffer 
    u8g2.clearBuffer();

  // if there is a face infront of person sensor
    
  if (results.num_faces>0)
     {

      /* 
       *  the following code will transform 256x256 coordinate into
       *  128x64 by scaling down, this is done to keep the face 
       *  into 128x64 OLED display, using circle and triangle func
       *  an emoji face will be drawn which will represent the user
       *  face, it will move as the person's head moves on the FOV
       *  plane of the person sensor's camera 
       */

        x = (face->box_left)/2; // 63
        y = (face->box_top)/4 ; //31
        x = x + x_offset;
        y = y_offset - y ;
        u8g2.drawCircle(x,y,10, U8G2_DRAW_ALL);
        u8g2.drawDisc(x-3,y-2,2, U8G2_DRAW_ALL);
        u8g2.drawDisc(x+3,y-2,2, U8G2_DRAW_ALL);
        u8g2.drawTriangle(x-4,y+4,x,y+7,x+4,y+5); 
        u8g2.setFont(u8g2_font_ncenB10_tr);
        u8g2.setCursor(0,64);
        u8g2.print("! Face Tracking !");
        u8g2.sendBuffer();
    }
    else 
    {  
        u8g2.drawCircle(63,31,10, U8G2_DRAW_ALL);
        u8g2.drawDisc(60,29,1, U8G2_DRAW_ALL);
        u8g2.drawDisc(66,29,1, U8G2_DRAW_ALL);
        u8g2.drawTriangle(59,35,63,36,67,35);
        u8g2.setFont(u8g2_font_ncenB10_tr);
        u8g2.setCursor(0,64);
        u8g2.print("!No Face Found!");
        u8g2.sendBuffer();
    }

  /*
   * now based of the face position data
   * keyboard press will be emulated and
   * arrow key press will be sent to 
   * host pc over USB
   */
   
  // let go all previouly hold keys
    Keyboard.releaseAll();
    
  // arrow key control based on face position 
   if(results.num_faces>0)
   {   
    if  (y<22)  
    {
      Keyboard.press(up);
    }
    if  (y>36)  
    {
      Keyboard.press(down);
    }
   if  (x<48)  
    {
       Keyboard.press(up);
       Keyboard.press(left);
    }

    if  (x>72)  
    {
      Keyboard.press(up);
      Keyboard.press(right);
    }
 }
 
/*
 * adjust lamp brightness using these two optical beam
 * interrupt switches
 * these can be reprogrammed to use for gaming too
 */

  if (analogRead(A1)<150)
  {
   digitalWrite(A0,1);  
   lamp_drive_pwm--;
  }
  else
  {
   digitalWrite(A0,0);  
  }
  

  if (analogRead(A2)<150)
  {
   digitalWrite(10,1);  
    lamp_drive_pwm++;
   }
  else  
  {
   digitalWrite(10,0);  
  }

  // a bit delay as person sensor can update at 7HZ only
  
  delay(SAMPLE_DELAY_MS);

} 

///////////////////////////////////////////////
////////////// end of void loop////////////////
///////////////////////////////////////////////

Credits

Shahariar

71 projects • 263 followers

"What Kills a 'Great life' is a 'Good Life', which is Living a Life Inside While Loop"

Comments

Awards

Honorable Mention Prize Gaming

Build2gether Inclusive Innovation Challenge

Lucky Draw — 3 Submissions

Build2gether Inclusive Innovation Challenge

Gaming with Disabilities: Face Tracking Game Controller