Cheza Pona

Problem Statement

In a world marked by technological advancements, individuals with disabilities, including but not limited to individuals with cerebral palsy and muscular dystrophy, face persistent challenges in accessing effective and engaging Technologies for enhancing hand-eye coordination and motor skills.

A 2018 study by the American Occupational Therapy Association found that 60% of occupational therapists reported that their patients had difficulty accessing affordable assistive technology.

Because of this, Physical Therapy such as Occupational Therapy end up being demotivating, not to mention expensive, as one has to pay for long hours with a therapist.

A 2017 study by the World Health Organization found that only 1 in 10 people with disabilities in low-income countries have access to the assistive technology they need.

Our Solution

Given the current predicament, we came up with a solution Cheza Pona.

Cheza Pona is Swahili for "Play get better".

Cheza Pona is an innovative gaming platform redefining inclusive skill development, Designed for individuals with disabilities, including cerebral palsy, muscular dystrophy, and more, Cheza Pona offers a dynamic and adaptive gaming experience.

Our Solution offers :

1.Tailored Exercises

As a gaming platform our solution to an individual needs for instance :

• Fine Motor Skills -coordinated movement of small muscles (hands, face).
• Gross Motor Skills - coordinated movement of large muscles or groups of muscles (trunk, extremities).
• Hand-eye Skills -the ability of the visual system to coordinate visual information. Received and then control or direct the hands in the accomplishment of a task.

2. Adaptability

Our solution focuses on making controls not limited or non-restrictive for people of different needs

3. Engagement

Our solution looks into making Often monotonous Physical therapy into an enjoyable and engaging experience. Through gamification elements, interactive challenges, and a user-friendly interface, individuals are motivated to participate actively in their Therapy journey

4.Affordability

We also focus in making the solution affordable through use of inexpensive hardware(but quality) and making it possible for one to partake Physical Therapy without the need of a Therapist.

Execution

To bring our solution to life we utilized Unity game engine, where we had a prototype game.

The Game

This was a prototype that is aimed at incorporating basic controls to showcase how Inertial measurement unit sensor can make the game more as a therapy tool. The game uses simple falling blocks mechanics. Here blocks of ranging shapes and sizes fall and the player evades them by moving sideways. In the meantime, the player has objects (red and black) to collect which increases their points.

ASSETS

To make the prototype we focused on using basic shapes

Player prefab – a single cube

Enemy Prefabs – different combinations of cubes of different orientation and sizes to act as obstacles

Collectibles – simple spheres colored red and Black

GAME CODE

The game is making use of 9 essential scripts.

Game Manager

This script basically manages the most essential parameters that are used by most if not all of the other scripts

• It updates the scores UI.

• Manages each panel and how they are triggered by events taking place in the game for instance Game Over.

• Manages essential events such as Game Over, Restarting Game, etc

Player Controller

• Manages player movement and in response to input

• Establishes the connection between the game and the Inertial measurement unit through serial port

• Keeps the player in screen view

This is the really important for making a more inclusive controller

using UnityEngine;
using System.IO.Ports;
using UnityEngine.Animations;

public class PlayerController : MonoBehaviour
{
    [SerializeField]
    private GameObject GameManagerObj;
    private GameManager gameManager;

    public Rigidbody2D playerRb;

    float screenWorldUnits;
    float sideSpeed = 0.2f;
    //float sideForce= 100f;
    bool gameStarted;
    float smoothing;

    public SerialPort serialPort = new SerialPort("COM12", 115200);

    void Start()
    {
        gameManager = GameManagerObj.GetComponent<GameManager>();
        float playerHalfWidth = transform.localScale.x;
        screenWorldUnits = Camera.main.aspect * Camera.main.orthographicSize + playerHalfWidth;

        try
        {
            serialPort.Open();
        }
        catch (System.Exception e)
        {
            Debug.LogError("Error opening serial port: " + e.Message);
        }
    }

    void Update()
    {
        gameStarted = gameManager.isGameStarted;

        if (gameStarted)
        {
            if (serialPort.IsOpen)
            {
                try
                {
                    string data = serialPort.ReadLine();
                    ParseArduinoData(data);
                }
                catch (System.Exception e)
                {
                    Debug.LogError("Error reading from serial port: " + e.Message);
                }
            }
        }
    }

    void ParseArduinoData(string data)
    {
        // Arduino sends data in the format "ax,ay,az"
        string[] values = data.Split(',');

        if (values.Length == 3)
        {
            float ax = float.Parse(values[0]);
            float ay = float.Parse(values[1]);
            float az = float.Parse(values[2]);

            float movement = ax * Time.deltaTime * sideSpeed;

            // Move the player left or right based on accelerometer data
           transform.Translate(-Vector3.right * movement);

            // Limit player movement within the screen bounds
            transform.position = new Vector3(Mathf.Clamp(transform.position.x, -screenWorldUnits, screenWorldUnits), transform.position.y, transform.position.z);
        }
    }

    void OnApplicationQuit()
    {
        if (serialPort.IsOpen)
            serialPort.Close();
    }

    public void SmoothMovement(float a)
    {
        float targetX = Mathf.Clamp(transform.position.x + a, -screenWorldUnits, screenWorldUnits);
        Vector3 targetPosition = new Vector3(targetX, transform.position.y, transform.position.z);

        // Use Mathf.Lerp to interpolate between the current position and the target position
        transform.position = Vector3.Lerp(transform.position, targetPosition, smoothing);


    }
}

Player collision

• Manages Player collision events and triggers an event based on whether the collision was with an enemy block or with a collectible

Move enemy

• In charge of setting up motion of the enemy blocks and the collectibles

• Destroy the obstacles and collectibles once off screen

Enemy Behavior

• In charge of randomly spawning the blocks (obstacles) and the balls (collectibles) at random positions

Other Scripts

Upgrade Spawners- to increase the rate at which the blocks and the balls spawn.

Audio Manager – manages the background and background music.

Sound class – singleton class with important variables for our music and sound effects.

GAMEDEPENDENCY GRAPH

THE CONTROLLER

For the controller we used an M5stick C, why? Because the M5Stick C is a small and compact MCU with an inbuilt 6 axis IMU sensors.

A little explanation on the IMU (Inertial measurement unit) sensor.

An inertial measurement unit (IMU) is an electronic device that measures and reports a body's specific force, angular rate, and sometimes the orientation of the body, using a combination of accelerometers, gyroscopes, and sometimes magnetometers.

Accelerometer:

• Function: Measures acceleration forces acting on the device.
• Explanation: Detects changes in velocity or direction of movement, providing information about the device's linear acceleration in three-dimensional space.

Gyroscope:

• Function: Measures the rate of rotation or angular velocity.
• Explanation: Determines how quickly the device is rotating around its three axes, helping to track changes in orientation and providing data on the device's movement.

Magnetometer:

• Function: Measures the strength and direction of the magnetic field. The M5stick C lacks this.
• Explanation: Provides information about the device's orientation relative to the Earth's magnetic field, aiding in determining the device's heading or direction.

With that understanding of the Inertial measurement unit let us now dive into how we used it in our game.

The Inertial measurement unit takes the motion data and sends it to the game which in turn acts on the data it receives to move the character.

Think of it like a magic wand inside your controller. When you tilt the controller, the IMU senses the tilt and tells the game to make your character or move accordingly

The motion data is sent in from the M5stick C to the game through serial communication , from their the player controller script in unity uses it to controller input.

#include <M5StickC.h>

float accX = 0.0F;
float accY = 0.0F;
float accZ = 0.0F;

float gyroX = 0.0F;
float gyroY = 0.0F;
float gyroZ = 0.0F;

float pitch = 0.0F;
float roll  = 0.0F;
float yaw   = 0.0F;


void setup() {
  // put your setup code here, to run once:
  M5.begin();
  M5.IMU.Init();
  M5.Lcd.setRotation(1);
  M5.Lcd.fillScreen(BLACK);
  M5.Lcd.setTextSize(1);
  M5.Lcd.setCursor(40, 0);
  M5.Lcd.println("IMU DATA");
  M5.Lcd.setCursor(0, 10);
  M5.Lcd.println("  X       Y       Z");
  M5.Lcd.setCursor(0, 50);
  M5.Lcd.println("  Pitch   Roll    Yaw");

  Serial.begin(115200);//Serial baud rate
}

float temp = 0;

void loop() {
  // put your main code here, to run repeatedly:
  M5.IMU.getGyroData(&gyroX,&gyroY,&gyroZ);
  M5.IMU.getAccelData(&accX,&accY,&accZ);
  M5.IMU.getAhrsData(&pitch,&roll,&yaw);
  M5.IMU.getTempData(&temp);
  
  M5.Lcd.setCursor(0, 20);
  M5.Lcd.printf("%6.2f  %6.2f  %6.2f      ", gyroX, gyroY, gyroZ);
  M5.Lcd.setCursor(140, 20);
  M5.Lcd.print("o/s");
  M5.Lcd.setCursor(0, 30);
  M5.Lcd.printf(" %5.2f   %5.2f   %5.2f   ", accX, accY, accZ);
  M5.Lcd.setCursor(140, 30);
  M5.Lcd.print("G");
  M5.Lcd.setCursor(0, 60);
  M5.Lcd.printf(" %5.2f   %5.2f   %5.2f   ", pitch, roll, yaw);

  M5.Lcd.setCursor(0, 70);
  M5.Lcd.printf("Temperature : %.2f C", temp);



  //data sent over serial
 // Serial.print("gyroX:");
  Serial.print(gyroX);
  Serial.print(",");
  //Serial.print("gyroY:");
  Serial.print(gyroY);
  Serial.print(",");
  //Serial.print("gyroZ:");
  Serial.println(gyroZ);
  
  delay(100);
}

Note :

The M5stick C uses a baud rate of 115200 and higher.

Ensure you setup the the input port of your in the player controller script

public SerialPort serialPort = new SerialPort("COM12", 115200);

Mounting the controller

Now to use the controller one can mount it on their hands or any any of their limbs. For our case we mounted it on our friend's left arm, we used tape but M5stack have a watch harness for the M5stick C that works better better. Once you have mounted the controller connect it to your PC with a USB C cable.

GAME-PLAY

When the game starts the score is set to 0 and random blocks start to appear moving past the block. Also circular collectibles appear at random positions also following suits. Main objective is having the player collects as many circular collectibles as they avoid the falling blocks. Each time the player collides with a collectible it is destroyed and a point is added. If in the player collides with the enemy blocks then the game stops and it is game over.

DEMO VIDEO

Conclusion

The above game has been just an example of how Cheza Pona can gamify the Physical Therapy for people with disabilities. This game mainly test for hand eye coordination, we believe with more investment, this will revolutionize therapy that can sometime be painful and tedious.Join us on the journey to revolutionize inclusive skill development and therapy for individuals with disabilities and play a vital role in creating a more accessible and engaging future.

Future Improvements

1. Integration of Virtual Reality or Augmented reality.

2.Addition of more games to prevent this from being monotonous.

3. Integration of data analytics for AI recommendations and help.

We have attached the Repository containing all the code used for this project.

Special Thanks to all the sponsors and organizers of the Build2gether Inclusive Innovation Challenge.