e-Ink RPM Letterboard (v4.0)

What, Why and How

This is another iteration in an attempt to help my son and other non-verbal autistic individuals in their struggles to communicate. Contrary to what they may seem on the outside, just know that they are intellectually intact. People with autism may know exactly what they want to say, but may be unable to say it. They may know how to play a game, but sit motionless, or simply rock back and forth. As much as someone with autism might want to, it may seem impossible to turn a thought in his head into speech from his lips, or to convince her hand to pick up a pencil—in other words, to break free from the gap between intention and action. It's an aspect of neurology called “praxis".

To this avail, Soma Mukhopadhyay created a method called RPM to open a window into their minds, that, oversimplified, teaches the individual to, slowly and painstakingly, point at letters to spell his/her thoughts and sayings as well as expose them to information and interaction that they would otherwise never have.

I've built several letterboards mimicking the simple laminated version used in RPM but providing sensory feedback (visual and audio) in an attempt to reinforce the mindful intents at communication. Past attempts avoided any screen-based option because to some, the light emitted from an iPad becomes a disruption in an already noisy sensory system. These versions have evolved in functionality and complexity as I learned electronics and associated abilities.

This latest iteration uses an e-ink Android tablet, the Onyx Boox Note Air 2 Plus E-reader that has the screen size for a visually acceptable experience. Being e-Ink, it does not emit light, nor does it evoke memories of YouTube or Internet interactions of an iPad, presumably limiting the disruptions produced by a LED display.

In my first attempt I tried Android Studio, thinking that such an easy app would be easy to develop using that IDE & framework. Wrong. My experience with Android Studio was like trying to teach ballet to Frankenstein's monster (and consider that I am fairly well versed in C++ and Python). So, I switched back to hardware and researched e-ink screens with a capacitive overlay and some microprocessor to program the whole thing in C++ (Arduino, to be precise) as I've done in the past. That entailed a whole new series of electronics complexities.

And then, I discovered MIT App Inventor.

Not only is it incredibly easy to learn and use, its community forum is the standard bearer for extraordinary support. As good as other forums are, like Arduino, KiCad or others, I always got a precise and effective answer in a few hours from people with patience and willingness to help. Just awesome! (Of course, as in anything in life, how you ask defines what you get).

Hardware

I settled on the Note Air 2 e-Reader because:

• Screen: E Ink Carta, 10.3", touch (inductive + capacitive), resolution 1404*1872 dots, 227 ppi, 16 shades of grey
• Processor, RAM & Memory: 8-core @ 2 GHz; 4GB; 64GB
• OS & TTS: Android 11; Speech Services by Google
• Other: G-Sensor (to detect screen orientation)
• plus the many positive reviews.

Software

Program Flow

The program has two modes of the letterboard: QWERTY in landscape and ABC in portrait with an output line on top with what's being typed and the corresponding predictive text (if the function has been enabled).

Audio feedback options include two types of clicking sound (or none) with each touch, TTS of letters and TTS of words. Visual feedback options are dark buttons with light letters or inverse, flash feedback when touching a button and predictive text. All these can be turned on/off to experiment if it is helping (reinforcing) or disrupting the RPM session.

For the predictive text function, I picked the first 40k words (weighted in frequency of usage) from the Real Academia Española's Corpus de Referencia del Español Actual (CREA) and using Excel, I curated the list, adding a search vs display values (to account for accents in Spanish) and 8 columns with 1 to 8 character fragments of each word to facilitate the search functions. This data is saved as CSV and copied to the "Download" folder of the Android tablet. (The CSV file is included in the GitHub link with the code, below).

Finally, some configuration tuning was necessary in the Boox's system, being a e-ink tablet it does have certain nuances on screen & refresh management.

The app is designed using Responsive Design Guidelines, but it is only tested in the Boox. In theory it should work in other screen sizes. Consider also tips on screen management from FAQ Section: Screens.

Finally, a must read is the Technical and Other Guides reference to fully understand the live debugging concept and other useful stuff in AI2.

Screen1

The .Initialize method sets the permissions to read from external storage (as defined by Android) using this guide. There are other alternatives to access files in Android. The method then opens the corresponding keyboard layout based on the device´s current orientation using the accelerometer (Screen1.Width and Screen1.Height have a 250ms delay after orientation has changed).

To avoid issues with this delay, I move between screens using the result parameter to identify what screen closed as opposed to testing for width & height to determine where to go next. I'm using Screen1 as a screen "manager" as suggested by PuraVida's recommended method of switching screens.

If you'll be using the AI Companion, consider (from that same link) some limitations:

• The close screen block triggers the Initialize event instead of the OtherScreenClosed event.
• The close screen with value block triggers both the Initialize and OtherScreenClosed events instead of only the OtherScreenClosed event.
• The close application block does not work, a message 'Closing forms is not currently supported during development' will be displayed instead.

So I use the companion to start developing and to design & test UI. After some time, I test only using compiled version (and continue using the AI Companion for UI changes). As an alternative to the AI Companion, there are several emulators you can use to test on screen w/o an Android device).

KeyboardABC and KeyboardQWERTY

Both screens have the same logic with minor differences from the different layout

Logic starts by reading the csv file with the words for predictive text and loading that into a table (a "list of lists", in AI2 parlance) then loading the feedback configuration parameters. Good tips on manipulating lists.

The non-letter buttons (space bar, backspace & clear all) have their own code blocks

The letter buttons and the predictive text buttons are handled by the AnyButton.Click block which has two sections one for predictive text and one for an individual letter.

The predictive text function searches for relevant words with each letter being typed. To speed this up, I found it necessary to avoid using any text conversion at all during the search block.

And it makes sense: each search entails 40k comparisons so adding 2 x 40k text conversions to each comparison slows things a lot!. This is resolved by using the searchText and fragment column local variables.

MIT AI2 uses functional programming: "...a programming paradigm where programs are constructed by composing pure functions, avoiding shared state, mutable data, and side effects...".

I added a page flip sound effect when changing screen orientation (and thus keyboard layout) as a way to ensure the 250ms elapse and all screen orientation variables are updated. Just in case.

The full source code (.aia) and compiled package (.apk) are included down below.

Enclosure

The 3D printed enclosure for the Boox allows the therapist / provider to hold the tablet comfortably in an adequate position in both landscape (ABC Layout) and portrait (QWERTY Layout) modes. Some creative junctions had to be created given the print size limitations of my 3D printer.

What's Next

Already in the development pipeline, is adding two functions to the predictive text module:

• Modify the frequency weight of each word as it is used, so that more common words of a specific user have a greater probability of appearing in the predictive text selection.
• Add new words as they are used and that are not found in the existing 40k list but curated by the therapist / provider after the RPM session is done to avoid misspellings.