SensiGlove- adding sensory perception to any prosthetic arm

Introduction

In the USA alone, nearly 2 million people live with limb loss. While prosthetic arm technologies for more advanced motor control are quite numerous, there is a significant lack of a low-cost method for replacing sensory feedback. Prosthetic arms that do include some sort of sensory feedback are often extremely expensive (as high as $100, 000), and are not accessible to the vast majority of the population.

Fortunately, we've developed a prototype that presents a pretty nice solution to this issue. Our device, SensiGlove, enables people with prosthetic arms to feel touch again through haptic feedback.

When developing our device, we wanted to focus on making it both marketable and practical. Our intention was to create a project submission that could actually be taken to market by Neosensory (of course with some improvements). We in particular wanted to focus on improving upon characteristics that make the vast majority of solutions available on the market fail as accessible and effective products. By structuring the design of our device around key characteristics, we were able to develop an innovative, effective, and practical solution to this problem.

Let's take at the look at the characteristics of our design which we think make it a winning product.

1)Compatible with virtually ANY prosthetic arm on the market

Here's a very important point that I'd like you to consider. There are a ton of people out there who have already invested in a prosthetic arm. Even the less advanced ones are by no means cheap. If you wanted to give these people the ability to sense touch again through their prosthetic arm in a practical, affordable, and accessible manner, which of these two options would you pick?

• OptionA- Build them an entirely new custom 3D printed prosthetic arm with all kinds of sensors from scratch that gives them the ability to sense touch (hint: very expensive and impractical)
• Option B- Build a low-cost accessory that is simply worn on top of the glove, empowering them with sensory perception. (hint: this doesn't require them to spend thousands of dollars on a completely new prosthetic arm).

Option B is by far the better choice. It is clear that making an entirely new prosthetic arm with built in sensation through the Buzz is NOT the answer. A lot more people would be willing to buy a product like SensiGlove if it was compatible with the prosthetic arm that they have ALREADY invested (potentially tens of thousands of dollars) in. Therefore, we designed our solution to be a simple glove interface worn on top of a prosthetic arm, thus allowing it to be universally compatible with virtually any existing prosthetic arm on the market. This feature is the key to our overall marketing strategy with this product.

2)Low-cost

If our solution is to be both practical and accessible, we would have to keep costs low. With the exception of the Neosensory Buzz, our glove can be built for about 28 USD. Even if we were to build an ultra low-cost prosthetic arm, total costs would undoubtedly be at least 10 times this value. Thus, our solution is the superior option.

3)Low-power

If we wanted our device to be usable throughout the day, power consumption would have to be kept very low. Our device can be powered for a whole day of use by a 3.7V 150 mAh battery about half the size of your pinky finger.

With these key features in mind, we developed SensiGlove, a practical, cost-effective, and marketable solution to sensory feedback for prosthetic arms.

Now that you know what SensiGlove is, why it's needed, and what makes it superior to existing solutions on the market, let's take a look at how it works.

How it Works

SensiGlove consists of three main functional components.

• Conductive Plates
• Adafruit nrF52840 Microcontroller
• Neosensory Buzz

There are conductive plates embedded on the thumb, fingers, and palm. These plates are connected (via wires) to a series of GPIO pins (on an Adafruit nrF52840 microcontroller) that contain built in capacitive touch sensors. Therefore, when these conductive plates are touched, the GPIO pins are also "touched" and the microcontroller is able to read the corresponding values.

Through some data collection, we found the capacitive touch sensor values that corresponded to the sensors touching human skin. We then programmed the microcontroller to take input from four different "groups" of these conductive plates. Each of these four groups are linked to one of four corresponding linear resonance actuators on the Buzz through the SensiGlove software. These groups are as follows.

• Group 1- Thumb →LRA 1 on Buzz
• Group 2- Index and Middle Fingers → LRA 2 on Buzz
• Group 3- Ring and Pinky Fingers → LRA 3 on Buzz
• Group 4- Palm → LRA 4 on Buzz

The reason we split the conductive plates into four groups based on their proximity to each other was so that we could have each group stimulate a corresponding linear resonance actuator (LRA) on the Neosensory Buzz. The Buzz itself is a haptic feedback device that allows humans to feel input data from a microcontroller (communicated over BLE) as vibrations on the skin.

Tying this all together, when a conductive plate on the glove is touched to human skin, the corresponding LRA on the Buzz will vibrate. If multiple "groups" of conductive plates come in contact with skin, multiple LRAs (each corresponding to their defined group) on the Buzz will vibrate. But there's another layer of complexity here. The capacitive touch reading ranges can also be modified to ranges corresponding to other common materials (besides human skin) to enable the user to make distinctions between touching different materials through differing haptic feedback mechanisms.

But we're not done there either. Through the use of Edge Impulse's amazing platform for machine learning on microcontrollers, we've developed a neural network that allows the user to even perceive different touch patterns through haptic feedback. Through our neural network, the user can differentiate the feeling of tapping something from touching something at each of the four conductive plate groups.

Let's take a more in-depth look at the hardware and software components used to build this project.

Hardware Overview

At its core, SensiGlove consists of an Adafruit nRF52840 based Circuit Playground Bluefruit microcontroller. This was sewn into a black glove.

Now I know what you're thinking.

Isn't Adafruit's Circuit Playground board designed to teach kids how to code? How can you give your device a cool name like "SensiGlove" when your microcontroller is literally called the "Circuit Playground"?

That's what we though at first too. But, unknown to most, the Circuit Playground Bluefruit is actually by far the best microcontroller that's fully supported by the Neosensory Arduino SDK. Why you may ask? This board not only has a huge variety of built-in sensors, but has GPIO pins with built in capacitive touch sensing, 10 embedded Neopixels for a flashy look, has its own library to make coding quicker, can be sewn on to cloth (like our glove), has built-in mechanisms to prevent us frying the circuit board (unlike the Feather), and is 25% less expensive than most Feather boards (which also happen to have less features than the Circuit Playground). These device characteristics help keep costs low, and add a huge variety of functionality to our project. If all of these amazing features don't convince you to switch over to the Dark Side and pick up one of these boards, we don't know what will.

Next off, we have the capacitive touch plates embedded onto the fingers of the glove, along with the palm. These get their capacitive touch capabilities by being electrically connected (via wires) to the GPIO pins of the microcontroller, which have built-in capacitive touch sensors. These are made of flexible aluminum strips to allow bendability.

The Neosensory Buzz is the star of our show, and is the irreplaceable haptic feedback device that enables the SensiGlove with the power to allow its prosthetic-arm wielding bearer to detect human touch.

The device itself is powered by a tiny 3.7V 150 mAh LiPo battery. This battery has lasted us through days of testing on a single charge, and has the perfect size for use in a project like this.

Software Overview

All of our code was developed and deployed on Visual Studio Code. The Arduino IDE is also a great option, but coding collaboratively is far easier with VSCode's VSLiveShare extension.

Besides our development environment, a key few features of our device were dependent on some libraries. Capacitive touch data (from the GPIO pins) was read through a library for the Adafruit Circuit Playground Bluefruit. Communication between the Neosensory Buzz and the microcontroller occurred over BLE through the use of the Neosensory Arduino SDK. All of the code for our project can be found at the linked GitHub repository at the end of this document.

Edge Impulse Experimental Features

In addition to the features discussed earlier, we wanted to use Edge Impulse to add some extra functionality to our project by leveraging the power of machine learning.

We first decided to run a quick script on our microcontroller in order to collect capacitive touch sensor data from the four conductive plate groups using the Edge Impulse CLI. This would be used to train our model. First, we ran and uploaded the following script to our microcontroller.

```
#include <Adafruit_CircuitPlayground.h>

void setup() {
  Serial.begin(9600);

  CircuitPlayground.begin();
} 

void loop() {
  int group1 = CircuitPlayground.readCap(6);
  delay(10);

  int group2 = CircuitPlayground.readCap(9);
  delay(10);
  
  int group3 = CircuitPlayground.readCap(0);
  delay(10);
  
  int group4 = CircuitPlayground.readCap(2);
  delay(10);

  Serial.println(group1);
  Serial.println(group2);
  Serial.println(group3);
  Serial.println(group4);

}
```

After uploading this code to our microcontroller, we installed the CLI through command prompt using the following command.

npm install -g edge-impulse-cli

We then used Edge Impulse's Data Forwarder tool to collect data (to train our model) from our microcontroller over serial.

After setting up a means to collect data, we can start using Edge Impulse to train our model. Here's an example of the various classes of data in our model.

Next, we used this data to train an impulse. This was also a really simple and straightforward process. The processing and learning blocks we utilized are as follows.

After training our model, we got the following results. Our model performed with around a 94% percent accuracy. Overall, not too bad.

The model can be deployed on the SensiGlove device by exporting it as an Arduino library.

Demonstration and Testing

We made a demonstration video to show that SensiGlove is indeed fully functional. Please check it out!

Extension and Roadmap

We think that SensiGlove is the prototype for something big. However, there are definitely key areas that should be improved upon before taking this project to market. The reason we couldn't address these in our prototype is primarily due to budget and access to more professional construction resources. Let's discuss them below.

1) Integration of force-sensitive resistors

High quality versions of these things are pretty expensive, and sadly, we are college students. And while we have a lot of time to think critically and develop cool projects, we don't have always have the budget to add these kinds of things to our project. However, we think that integrating force-sensitive resistors in the SensiGlove can add yet another layer of functionality to this project. It can be used to help adjust prosthetic arm grip strength in arms with more advanced motor functions. We could also use it as an input sensor and leverage Edge Impulse to create a more advanced sensory perception system. Luckily, these FSRs can easily be integrated into the design of the SensiGlove without any problems.

2) A more professional and aesthetic design

Once again, our design is a prototype. We think that SensiGlove has a somewhat cool appearance, but for an actual market-ready product, it would be much better to have all the wires and circuitry on the inside of the glove. A more flexible conductive material could be used for the conductive plates. Additionally, it might be better to have the conductive aluminum strips on the finger tips.

Overall, the design can be cleaned up to make things more aesthetic. We do think that the Circuit Playground Neopixels give it a futuristic look though, but of course this increases power consumption.

Concluding Remarks

Thank you for checking out SensiGlove! We had a lot of fun creating this project, and we think it can bring a lot of positive impact to the world. Please feel free to message us if you have any questions or suggestions!

Also a big thank you to Neosensory and Edge Impulse for organizing this contest.