BuzzNPoses: Haptics for Cheap Prosthetics

INTRODUCTION

Like learning to use chopsticks or other useful tools, we can learn to use prosthetic limbs to expand and augment the natural range of available actions, allowing us to act remotely and/or with unnatural implements, such as robotic arms used in the handling of toxic chemicals. Besides augmentation, prosthetics are used in the tragic circumstance of limb loss. Losing a hand, for example, severs the crucial feedback from the world and the amputee has to adapt to use some other tool instead. Most common prosthetics are simple and do not provide adequate feedback to the user. Without proper feedback, learning can be inefficient and incomplete, therefore integrating sensory feedback with artificial limbs is one of the main goals in prosthetic development.

Considerable efforts have already been made to tackle sensory restoration, for example, in the development of artificial sensitive skin. (doi/10.1089/soro.2019.0013) While technologically stunning, such solutions are out of reach of almost everyone, spare for the few with access to bleeding-edge scientific institutions. Simple and accessible technology needs to be developed, in order to have a considerable impact on the population in need of prosthetics.

Hands have one of the most developed sensory representations in the brain, compared to other body parts. We naturally expect our hands to respond to the lightest of touches and be precise when gesturing. Some sensations are low-dimensional and change gradually (i.e., do not require high resolution), for instance, thermoception can be represented by a single variable and encoded, for example, as a static vibration of variable intensity on a person’s skin. On the other hand, the proprioceptive and tactile space of a human hand is exponentially larger: each of the five fingers has three joints that can move independently and feel pressure with high resolution (to read Braille for instance). Recent developments in supervised learning methods open an exciting opportunity to capture the essential features of our hands' senses and transmit them to a human via another channel, like vibration patterns on the user’s skin. This allows for an accessible solution to sensory restoration and augmentation.

In this work, we test haptic encoding and bio-decoding to create a functional prosthetic of a human hand that robustly reports both its static and dynamic states through a haptic device generating vibration patterns on the user’s arm. We believe this development will extend the accessible set of actions for prosthetic users, ranging from amputees to workers with hazardous materials.

We use the feedback from the actuators and pressure sensors installed in the prosthetic hand to provide simple continuous proprioception. In contrast to the fiction-esque neural interfaces that can only be installed in few advanced medical facilities, our solution is simple and accessible - apart from motors and circuitry, it can be obtained through basic 3D printing and can be assembled manually. It is also inexpensive: its price is estimated lower than that of Neosensory Buzz. By stringing together our prosthetic hand and the Buzz, we created a simple device providing a complete cycle of interaction with the world, from acting to receiving feedback.

Our solution features continuous feedback: positions of all five fingers can be transmitted to the Buzz providing the sense of proprioception. The actuators allow control precision of single-digit degrees. In addition, we used force-sensitive resistors, placed on the surface of the prosthetic arm, to extend the available spectrum of sensory inputs. We explored two approaches to find the optimal mapping, a naïve mapping and an auto-encoding neural network. The range of available patterns of the Buzz gives us an opportunity to encode both the state of the hand with all its five fingers and the dynamic response from our movements in a compressed signal over the four motors of the Buzz. We envision that the learning capacity of the human brain will be sufficient to disentangle the encodings of static and dynamic responses and learn to associate the haptic feedback from the Buzz with the missing senses.

The story of our project goes all the way from assembling the 3D-printed robotic arm, soldering motors to the PCB, coding PID controllers on the STM32 processor using embedded C/C++, setting up communications on the ESP32 chip armed with Arduino/ESP32 libraries, and developing Android applications.

Chapter 0. THE HAND

The hardware part of our story turned out to be the biggest snag for us, taking most of our time we spent on the project. Simply speaking, we need 5 motors and a chip to receive commands and control these motors. Not more, not less. We used the hand from The Sparthan Prosthetic hand project (https://project-sparthan.github.io/). However, resurrecting and modifying old hardware was a long journey:

1. Soldering new motors to the new Sparthan board with extensions.

In the beginning, we, software guys, had zero experience in soldering. Had to pick it up from scratch!

2. Programing PID controllers on the STM32 Chip

The code to control each motor by regulating the applied voltage according to the encoder ticks is presented in ProjectSparthan/ repo. It was hard and painful to tune those PIDs, especially given that magnetic rotary encoders tend to malfunction all the time (ouch!).

3. Adding FSR sensors

Another obstacle we encountered was the connection of force sensitive resistors (FSRs), that were supposed to augment our robotic hand with pressure and touch sensing. We planned to add sensors to each fingers, but 3 of them broke down (we just burned the plume with soldering iron, ok la). We were left with two functional FSRs. That's why such a questionable placement! Yet another challenge was waiting for us, when we realized that sensors should be connected to specific pins on the PCB, because one-out-of-two analog-digital converter (ADC) of the ESP32 was busy with Wi-Fi. We honestly thought that connected sensors are malfunctioning before discovering that we ought to use ADC2.

But! Eventually the robotic hand worked, as you can see from the video below.

Chapter I. CLASSIFIER FOR GESTURE RECOGNITION

To create a prototype of the naturally controlled prosthetic hand, we need to train and deploy deep learning classifier for gesture recognition. Hand gestures and gestural intentions (of amputees) are characterized by electrical activity inside hand muscles. The technique called Surface Electromyography (sEMG) is used to measure this activity. In our project we use a commercial sEMG bracelet, called MyoBand.

1. Defining gesture set

As presented in CNB/datacore_constants.py/BUZZ_SET, we first define the set of recognizable gestures.

"00000",  # palm
"11111",  # fist
"01111",  # thumb
"10111",  # point
"00110",  # rock

Here, “0” corresponds to a relaxed finger, “1” to a contracted one. Indexing goes from thumb (0) to pinky (4), left hand, palm up.

2. Collecting training data

To collect the training dataset, launch Myo Connect App. The procedure for data collection is defined in CNB/main_window.py. Launch parameters are “-d EMG”. 30 seconds per gesture is usually enough. Slight variations of hand position during the recording are recommended to capture the ever-changing EMG signal.

3. Training the model

The training procedure is described in keras_models.py/train_models_main

We use a simple CNN over a configurable time window, as shown in Figure I.2.

Good choices for the window size are 50 and 100 samples (corresponding approximately to 0.5 and 0.25s given 200 Hz sampling rate of the Myo). Sounds laggy, doesn’t it? In real-time the actual lag of the classification is defined by classifier invocation interval over the accumulated sEMG data, so worries are unnecessary, the interval can be as small as you want, and as the computational capabilities of your devices allow.

4. Deploying the model

The main procedure in keras_models.py/train_models_main creates and validates the .tflite file. Next step is to import the model via Android studio. To do so, right click on the res folder “New > Other > TensorFlow Lite Model”, follow nice ML-models import wizard. Then, rebuild the project, double click on the model file, copy Sample Code to NAA/CLFWrapper.java, where inference is done. In our case, average inference time on Huawei P10 (2017) is below 60ms.

Chapter II. COMMUNICATION

The developed system features a lot of communications between components, as shown in Figure II.1.

0. From myo

We use myonnaise library (https://github.com/cortinico/myonnaise) to connect to Myo on Android smartphone via BLE. Thanks Nicola!

Myo sends 8 16-bits values at a time, 200 samples per second. Also, to vibrate the Myo the way we want, we had to look into the official Myo BLE API (https://github.com/thalmiclabs/myo-bluetooth/blob/master/myohw.h).

The procedure of sending custom vibration is implemented in NAA/MyoWrapper.java/vibrate.

1. From Android to ESP

A simple HTTP-server hosted on the Android on 5000 port serves the configuration of the most recently recognized (or selected via touch controls) gesture. A simple HTTP-client on the ESP32 polls the server.

Both, the Android phone and the ESP board, shall be connected to one WiFi network. IP-address of the phone shall be preconfigured in ESP32 code, as well as WiFi SID and password at BuzzNPoses/BuzzNPoses.ino, as shown below:

const char* SSID = "SPARTHAN";  
const char* PASS = "66668888";
const char* CONN = "http://192.168.137.243:5000/";

2. From ESP to STM and back, and back again

ESP and STM parts of the Sparthan PCB communicate via hardware serial, as defined in BuzzNPoses/uart_messaging.ino. There are two threads running on the ESP: one polls the app and transmits them via UART to STM, another relentlessly attempts to read back from the STM and pressure sensors, and runs a simple HTTP server to pass the data to the phone. Similarly, on the STM side, one thread is listening to the incoming command from the ESP over the hardware serial, another thread is controlling the motors, constantly keeping ESP updated on the current motors' position. Sounds like overkill for such a simple task. And it is indeed! But we are not looking for easy solutions, aha.

3. The app wants to know

Yet another thread on the phone reads positional and pressure data from STM. Ideally, the Android app listens to the incoming connections and start polling the server (on the ESP32) on the address of the first connected client (supposedly the ESP32). Somehow, it doesn't work (client address is null) on some WiFI networks, perhaps because of the security settings, so eventually all IP-addresses need to be configured manually.

4. One App to rule them all

Like a spider in the middle of the web, the app orchestrates all of the data streams within the network. It connects to the Myo Band and the Buzz via BLE. After receiving data from ESP32, it commands the Buzz and the Myo (in case of Naïve haptic translation) to vibrate.

App features several widgets, visualizing (1) incoming sEMG data from Myo, (2) recognized hand position, (3) vibrating motors of the Buzz. The UI of the app enables choosing preferable haptic profile and controlling finger positions of the robotic arm for debugging and showcasing.

Chapter III. BUZZINDER

Nobody did gamification better than tinder. So we took the idea of swiping and used it to train ourselves to recognize buzzes corresponding to gestures.

Chapter IV. THE BIOENCODER

We explored three ways of encoding the buzz signals.

First, a naive encoding where the angle of each finger corresponds to vibration motors on the Buzz and Myo (5 vibromotors in total). The more bent finger is, the more intense the vibration of the corresponding motor.

Second, is pattern translation. The idea is to identify several key states and assign distinctive patterns to them, similar to the training examples in the original NeoBuzz App. We define patterns in JSON files that are dynamically loaded and parsed by the app:

{
    "type_id": "type_3",
    "interval": 300,
    "config": [
        {
            "gesture": [1.0, 1.0, 1.0, 1.0, 1.0],
            "pattern": [
               [0, 255, 0, 255],
               [128, 128, 128, 128],
               [255, 0, 255, 0]
             ]
        }
    ...
    ]
}

These patterns were optimized to be as distinguishable from one another as possible since the skin touch resolution is low, making the close vibrating motors hard to resolve.

Third, the more advanced buzz encoding was based on a deep autoencoder neural network that was trained on 200 ms windows of continuous signal from all five motor positions as well as pressure readings. The bottleneck of the autoencoder contained 4 neurons that were relayed to individual buzz motors. The network displayed adequate performance on the test dataset, meaning that the original signal was recoverable from the encoded signals with good fidelity.

Future perspective

Though we created a complete framework encompassing gesture recognition, robotic hand motion and buzz-encoding of the hand poses, more work needs to be done before it can be useful outside of the testing environment.

The chassis of our hand only permit five bendable fingers, which allow only simple gestures and grasping. This is far from the range of motion accessible to a real human hand. To approach the natural range of motion, more actuated joints need to be added. This is a serious engineering problem that should seek to balance technical sophistication with the low cost of production and ease of maintenance.

The encoding framework needs to be tested on a cohort of real users to evaluate its performance as well as user experience and ease of learning. The set of inputs should to be increased beyond the currently enabled five fingers and two pressure sensors to allow applications to more complicated devices. We envisage that this point will be the most fruitful way forward to a final marketable product that could be easily adapted to various real-world devices and will be of great use in remote manipulation applications such as remote surgery and operation of instruments from heavy machinery to microfabrication.

Epilogue

This work was performed by Kirill Shatilov, who did most of the post-hoc soldering, developped the apps, programmed the ESP32 controller and did the funky animations for the video. Kirill Konovalov, who programmed the STM32 controller, tuned the motor performance, edited the video, and mastered the audio. Last but not least, Alex Wong designed and assembled the Spartan circuit and was generally involved in all hardware troubleshooting.