Assistive Communicator & Smart Night Light

NOTE: because of the #cloudgames2022 submission deadline, I did not have time to add all the functionality I designed. Pending functions will be marked as TBR (To Be Released) as will the "dress-up" of some visuals. I'll continue to revisit this post throughout the endless battle between my day job and this passion of mine.

#CLOUDGAMES2022

Many thanks to Arduino for putting me in these cloudgames2022. Through the Opla IoT Kit and the always-on Arduino community, I learned about audio reproduction, digital-to-analog conversion, integration to cloud services, PIR sensors, color in light elements, TFT screens, the UNIX timestamp and more. In this write-up, I share my research & findings (and a lot of links) on these and on other topics I've learned in the past year (since I started down this microelectronics rabbit hole) in the hopes of providing others with the same gift: the opportunity to learn while producing something useful.

BACKGROUND

This project is another effort in using technology to provide alternative communication options for autistic individuals (see my Capacitive Touch & Bluetooth Letterboard). Recognizing the challenges they face to initiate an interaction (‘Autistic Inertia, ’ An Often Debilitating Difficulty Acting on Their Intentions), I'm using the Opla IoT Kit as a pre-defined message facilitator (text & voice) for remote interactions.

The pre-defined messages provide a repeatable experience to the autistic individual via multi-sensory (i.e. voice, text and RGB LED sequences) to attempt expedite its familiarization, repetition and thus, its use. The messages sent to the Opla carrier can be targeted at specific events or moments during the day (i.e. "Good night, son", "I'm proud of your effort in school"," I love you", etc.) and those sent to the dashboard as the usual or repetitive messages from the subject using the Opla carrier (i.e. "I want sushi", "I am happy", "I need help", etc.).

To further improve on the familiarization to the Opla carrier and help the autistic subject in making it his/her own, some apps are added, like a Smart Night Light.

And as with most things related to autism, it's trial and error...

ENHANCEMENTS TO THE OPLA IOT KIT

The Opla IoT Kit is based on the MKR IoT Carrier (cheat sheet) and the Arduino IoT Cloud (cheat sheet) and has many features that are available out-of-the-box. But as with every microelectronics project, each use case can/will require some enhancements, so I added two new features:

1. Targeted notifications on a mobile phone. The Arduino IoT Remote App (iOS/Android) does not produce notifications when some variable changes in the IoT Cloud dashboard, making you miss important thresholds or, in this case, messages.

I started looking into IFTTT's SMS integrations with ClickSend to receive SMS messages in my mobile. Although easy and functional, the problem lies in that outside the US & Canada (I'm in Mexico), there's a per message cost involved.

I then tested the integration of the IoT Cloud with Telegram, quite easy to implement, but since it implied an additional app, I discarded it for something simpler.

Finally, I realized that the IFTTT Mobile App (iOS) does produce notifications, so tying the Arduino dashboard with an IFTTT applet via an IoT Cloud Webhook was the way to go.

This approach has a caveat: the webhook on the IoT Cloud Dashboard is called with any change on any variable. No filtering or selection is possible, i.e. if the Thing has 10 variables, the webhook will be called 10 times on startup. Solution #1 was to get an IFTTT Pro subscription and add a filter with JavaScript in the IFTTT applet. Solution #2 was to use a targeted GET HTTP statement to the IFTTT server inside the MKR1010's code (just make sure to use a secure https connection). I wanted to offload this to the cloud but Option 2 is way simpler.

So, to create your IFTTT applet, start with a Maker Webhook and under Documentation(which will appear only after you create an IFTTT account, for free), you'll get your key and JSON details to add to your code. Make sure to visit their Connect API documentation for complete details.

2. Playback of pre-recorded messages in the carrier. The IoT Cloud dashboard provides for sending open or pre-defined text messages to the carrier. I added this audio function to play back pre-recorded messages stored in the SD Card (library) when specific triggers occur.

I connected a Class D mono amplifier to DAC0 to send WAV data. But because DAC0 is also used as A0 in one of the touch sensors of the carrier, a click sound was generated with every loop of the carrier code when checking for touch events in the sensor. Disabling touch on A0 would limit the carrier's functionality, so using the amplifier's Shutdown pin tied to digital pin 14 of the MKR1010 board (I'm not using the carriers' relay assigned to D14) allows for turning the amp on only when playing back a message.

I found several audio libraries with varying limitations for this use case:

• TMRpcm is the best but only for AVR MCUs (i.e. UNO, MEGA, Nano Classic, etc.)

• ArduinoSound is very good but only for I2S devices (i.e. I2S 3W Amplifier)

• AudioZero works with my analog amp and SAMD MCU's (i.e. MKR Family, Nano 33 IoT, etc.) but only with a 8kHz sampling rate, producing poor voice sound.

• I settled on D34G's awesome SamdAudioSD (targeted for SAMD-based MCU's) albeit with some modifications: I commented out all references to the digital Pot functions in my local copy of SamdAudioSD.cpp since I'm not using a digital potentiometer and those sections created compile errors.

Sending audio from DAC0 is limited to 8-bit mono PCM WAV formats (uncompressed & analog). The SamdAudioSD library allows for a 44.1 kHz sampling rate in WAV files, which provides an acceptable voice quality. The pre-recorded messages are easily produced, enhanced and converted using Audacity, an amazing open source audio editor & recorder. You can also add all sorts of sounds and special effects from Freesound.

The play back of pre-recorded messages also allows to produce feedback or confirmation of events locally, i.e. when presence is detected via movement or touch or when a message is sent out from the carrier. Below is the subroutine for playback.

Other limitations of the IoT Cloud Dashboard fort his Use Case

• The "Messenger" type widgets (which handle Strings) do not recognize escape sequences (i.e. \n) nor do they wrap long strings. Consequence: long strings will be truncated.

• If the string value received (i.e. "Presence detected") is the same as the immediately previous value, even if at a different time interval, no instance of its reception is created. Workaround: a simple time stamp is added to the string sent to the widget to make it unique.

MESSAGING FUNCTIONS

Messaging from Carrier to Dashboard

• Detect presence by grabbing or touching the carrier, then send a remote notification to the receptor's mobile and to the IoT Cloud dashboard that the individual may be ready to receive a message.

• Select via the 5 touch sensors one of the pre-defined messages. When touched, play pre-recorded voice of the selected text as confirmation of message being sent. (TBR)

Messaging from Dashboard to Carrier

• From the IoT Cloud Dashboard (either PC or Mobile), send any of 4 predefined messages by pressing the corresponding button. The message is received as text on screen with buzzer & LED notification and play back of its assigned pre-recorded message.

• An open-text message can also be sent to the carrier albeit without an assigned pre-recorded message other than a simple notification.

• Each message has a distinct GIF image or characteristic Opla UI displayed on screen. (TBR)

SMART NIGHT LIGHT

The Smart Night Light service considers a separate dashboard to manage the schedule at which the service should be turned on and the type of light to use as night light (color and brightness). The night light is automatically turned on within the schedule and if the detected room luminance is below a defined threshold. This also helps is saving battery usage during day time or when the room's light is turned on.

Additionally, this service will use a PIR motion sensor with a filter to determine if movement detected should be considered as relevant, based on a number of detections within a defined time frame. This will help in eliminating false positives like when the subject turns in bed (as opposed when the subject is awake). When this happens, the service will change the night light to green: there are some studies suggesting how light color affects sleep.

Consider also some other notes in using a PIR sensor. There are some nuances regarding sensitivity and blocking time that should not be overlooked.

To select the night light color & brightness, I first tried to use a CloudColor variable associated to a Color widget (simple & elegant), but these elements use the HSV (hue-saturation-value) color scheme, whereas the LED Strip on the carrier uses RGB (red-green-blue; well, actually BGR. See line 128 of MKRIoTCarrier.h). The DotStar Library for the carrier's LED strip has an HSV-to-RGB conversion function, but because it's aimed at RGB (not BGR), I could not make it work in time for this submission. So I used a separate variable & slider widget for each RGB color.

Additional night light functions (TBR)

• Increase lumen output with detected movement.

• Different color output with different time of night to signal if it's time to sleep or time to wake up using the RTCZero or RTC libraries and a simple subroutine to convert GMT (obtained from the IoT Cloud) to local time (see updateLocalTime() in code).

• After a defined threshold of movement is detected, play back a pre-defined message (i.e. "go back to sleep", etc.) and send notification to cloud and/or mobile.

WHAT´S NEXT

Additional features to be added as the project progresses:

• Remote update of the WAV files on the carrier's SD card. Preliminary Option 1 is to receive the instruction from the dashboard which then triggers a GET HTTP to a data server like Firebase or Firestore holding the audio files. Option 2 is to run a webserver on the MKR1010 with the webpage on the SD card and with a public URL service like ngrok to access the client's filesystem, select a file, and transfer it to the SD card.

• OTA system updates. This should not present an issue since it is offered by Arduino's IoT Cloud service.

• Battery monitoring. Read ADC_Battery on the MKR1010 (perhaps via BatterySense) and report via dashboard and on the carrier's TFT screen when recharge is needed.

• Animated GIFs. This would enhance the carriers palatability to the subject on several of the functions. Best option so far is to use Larry Bank's AnimatedGIF and his own image conversion tool. The real issue will be total available memory on the MKR 1010.

• 3 Breaths App. Can be invoked after certain movement threshold from the Smart Night Light service is passed, or selected by the subject intentionally (or from too much shaking the carrier) or as message sent from dashboard user. Balloon image in TFT screen inflates and deflates (see Animated GIFs) as subject breathes 3 times into carrier, detected with humidity sensor. Breathing sequence is assisted with pre recorded messages (i.e. "breath in", "breath out slowly", etc.)