MicroPython to AWS-IOT

Background

The objective is to show a method to exchange state information between a device running micropython and AWS-IoT. The use case is an IoT application where the device sleeps in low-power mode most of the time and upon wake-up retrieves commands/state from the cloud/server. A motorized window treatment that uses a low-power + WiFi processor to control a motor and position a shade is an example. It wakes up and connects to a server every few minutes; it retrieves the desired window treatment position from the server. If the desired position is different than the current position, the processor controls the motor to move the shade to the desired position.

AWS-IoT services provide a stateful entity called 'thingShadow' which maintains a desired and reported state data dictionary. A device can retrieve the state from AWS-IoT when it wakes up. Clients can change the desired state using Amazon's API. Other Amazon services (Cognito, Lamda Functions, data stores) can be employed to add features like authentication, authorization and logging.

The thingShadow state is persisted in the Amazon cloud. It benefits from the Amazon infrastructure - it is highly available and is preserved through software updates. It has metadata for versioning and timestamping, so message duplication can be handled to avoid performing commands twice.

AWS-IOT can be accessed using either MQTT or REST. This example uses REST to do a simple GET and POST. REST was used instead of MQTT because the connection is lost when the device is sleeping.

For authorization, the AWS REST API requires an authorization header which includes a computed signature; the signature is a hash of keys obtained from AWS, plus timestamp and URI. The repository, aws-signature-iot-python provides this code. It was coded to run on micropython using minimal resources. The boto SDK was to large to run on an ESP.

Overview

The device behavior can be controlled by editing the 'desired' values in the AWS-IoT thingShadow state:

{  
"desired": {  
"sleep": 30,  
"signal": 2,  
"test": "none",  
"test_param": 1000  
},  
"reported": {  
"sleep": 30,  
"signal": 2,  
"test": "none",  
"test_param": 1000,  
"status": "done: beeped 2 times."  
}  
}

The desired state is what a client can edit; the device retrieves it on wake-up. The reported state is what the device publishes to acknowledge the desired state has been reached.

This example has 4 configuration variables:

• sleep: how long the device goes to sleep for; range: 5 to 600 seconds
• signal: the number of times to turn on/off an indicator, such as a LED or a beeper
• test: the name of a test to run. Tests can be added to test a devices capabilities.
• test_param: some tests need an additional parameter, such as duration.

The device can report additional state information, such as environmental conditions, battery state or other operational parameters. AWS DB or logging services can be used to store state change histories using the thingShadow metadata.

The control flow is as follows:

Software Overview

microPython was used instead of C++ because of its portability and ease of use. It allows running/debugging the non-hardware specific code on a development machine running OSX, Linux, etc.

There are 4 modules:

• aws_thing_loop: instantiates a thing, gets the shadow state from AWS-IOT, sets the thing's shadow state, posts the reported state to AWS-IOT and then sleeps.
• base_thing: provides the infrastructure for restoring current state, setting the shadow state, dispatching functions based on differences between current state & desired state and generating the reported state.
• a child thing: inherits from the base_thing. Infrastructure functions include restore/persisting state, providing a device ID and providing a timestamp. Device-specific functions can be added; these are called based on the thing's desired state compared to current state.
• awsiot_sign: generates the authorization header required for the AWS IOT REST API

Two example 'thing' child classes are provided in aws-iot-GET-POST-loop:

• signal_thing_unix: Adds the 'signal' function. For UNIX, the signal is sending the beep character to the terminal. The thing will signal (beep) based on the value of the desired state. For example, after waking up, the thing will beep 2 times when 'signal' is changed from some value to 2. This class uses a file to persist/restore current state. I did not find an easy way of getting the host address from micropython when running on UNIX, so the default ID returned is 'id-1'
• signal_thing_esp8266: Similar to signal_thing_unix: Adds the 'signal' function. However, instead of beeping, the ESP module has an LED that flashes to signal. RTC memory (non-volatile) is used to persist/restore current state. NTP is used to obtain the timestamp because the on-board RTC is not accurate in the long term. ESP deep sleep mode is used when sleep is called.

AWS-IOT Preparation

• Obtain an AWS account in order to manage AWS-IoT and AWS-IAM resources.
• Determine the name of your thing. If running the UNIX version, the name is 'id-1'. If running an ESP, the name will be in the format of ESP-chip_id. To get the chip_id of your ESP, use the esptool:

esptool.py --port /dev/<your_port> chip_id

• Using the AWS IoT Management console, select 'Registry' and create an AWS-IoT 'thing' having the name as described above, e.g. 'id-1' or 'ESP-08675309'. Refer to the AWS console documentation for how to create it. Edit the shadow state to have the 4 behavior control variables:

"desired": {  
"sleep": 30,  
"signal": 2,  
"test": "none",  
"test_param": 1000  
}

• Using the AWS IAM Management Console, add a user - the user name does not matter. Give the user only Programmatic Access, select 'Attach existing policies directly', search for iotdata and check the AWSIoTDataAccess policy, click next, and then finish adding the user.
• After the user is added, grab the access & secret keys and create a file named 'aws_credentials.txt' as shown below. If possible, store the secret key in secure storage instead of in a file. The file will be stored in the same directory as aws_thing_loop is started.

{"akey": "XXXXXXXXXXXXXXXXXXXX ", "skey": "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"}

• Create a file 'aws_iot_cfg.txt' with the region and endpoint prefix obtained from AWS for the device, as show below.

{"region": "us-east-1", "endpt_prefix": "ZZZZZZZZZZZZZ"}

• The endpoint_prefix for the is obtained from the AWS IOT Management Console - click the 'Settings' icon to display the Custom Endpoint, which has the format as shown below. And change the region to your region. The file will be stored in the same directory as aws_thing_loop is started.

XXXXXXXXXXXX.iot.us-east-1.amazonaws.com

Device Preparation for a Unix device

• Install micropython
• clone/download the two repositories: (1) aws-signature-iot-python and (2) aws-iot-GET-POST-loop
• Install the required python modules:

micropython -m upip install micropython-binascii   
micropython -m upip install micropython-errno   
micropython -m upip install micropython-ffilib   
micropython -m upip install micropython-hashlib   
micropython -m upip install micropython-struct   
export MY_GIT_PATH=<directory of cloned repositories>  
cd ~/.micropython/lib  
ln -s $MY_GIT_PATH/aws-signature-iot-python/hmac_ltd.py  
ln -s $MY_GIT_PATH/aws-signature-iot-python/awsiot_sign.py  
wget https://raw.githubusercontent.com/manningt/micropython-lib/urequest-with-content-length/urequests/urequests.py -O trequests.py

• cd to the aws-iot-GET-POST-LOOP directory. Move or copy the 'aws_credentials.txt' and 'aws_iot_cfg.txt' files to this directory.
• At this point, the aws_thing_loop can be run as follows:

$ micropython main.py

Device Preparation for an ESP8266 or ESP32

• Install micropython
• Due to ESP8266 RAM limitation, the python modules for this project need to be included in the flashed firmware in frozen bytecode
• Therefore the ESP micropython firmware needs to be built on your computer. Here is how to do it on OSX
• After successfully building and flashing the ESP micropython firmware, then add the following links in the ports/espXX/modules directory:

export MY_GIT_PATH=<directory of cloned repositories>   
export ESP_MODULES=$MY_GIT_PATH/micropython/ports/esp8266/modules   
ln -s $MY_GIT_PATH/aws-iot-GET-POST-loop/base_thing.py $ESP_MODULES/base_thing.py   
ln -s $MY_GIT_PATH/aws-iot-GET-POST-loop/signal_thing_esp8266.py $ESP_MODULES/signal_thing_esp8266.py   
ln -s $MY_GIT_PATH/aws-iot-GET-POST-loop/aws_thing_loop.py $ESP_MODULES/aws_thing_loop.py
ln -s $MY_GIT_PATH/aws-iot-GET-POST-loop/thing_accessor_http_sigv4.py $ESP_MODULES/thing_accessor_http_sigv4.py
ln -s $MY_GIT_PATH/aws-iot-GET-POST-loop/thing_accessor_mqtt_cert.py $ESP_MODULES/thing_accessor_mqtt_cert.py
ln -s $MY_GIT_PATH/aws-signature-iot-python/awsiot_sign.py $ESP_MODULES/awsiot_sign.py   
ln -s $MY_GIT_PATH/aws-signature-iot-python/hmac_ltd.py $ESP_MODULES/hmac_ltd.py   
ln -s $MY_GIT_PATH/micropython-lib/urequests/urequests.py $ESP_MODULES/trequests.py

• Build and flash the firmware with the frozen modules.
• Connect the ESP to your WiFi network
• Copy the 'aws_credentials.txt' and 'aws_iot_cfg.txt' files to the ESP's file system. Use WebREPL or ampy or rshell to transfer files to the ESP,
• To use the ESP8266 deep sleep feature, GPIO16 needs to be connected to RESET using either a 1K resistor or schottky diode (1N5817).
• At this point, the aws_thing_loop can be run manually:

>>> import aws_thing_loop  
>>> aws_thing_loop.main()

• You should see something like the following on the RS232 port. Note it may take a few attempts to connect to the NTP server.

connected with <your WiFi network>   
dhcp client start...   
ip:10.0.1.15,mask:255.255.255.0,gw:10.0.1.1   
Exception in get NTP: [Errno 110] ETIMEDOUT   
Free mem before POST: 21568   
Main took: 4375 msec. ---  Free mem before exit: 10736   
Going to sleep for 15 seconds.

• To have the aws_thing_loop run automatically after reset or deep sleep, copy the file 'main.py' from the aws-iot-GET-POST-loop repository to the ESP's flash file system. Micropython runs main.py after running boot.py after coming out of reset.

Conclusion

The target for this software is the ESP family of devices (8266, ESP32) because of their very low price and micropython is readily available/supported. However, all the modules except for the signal_things are hardware agnostic, which allows for debug/experimentation on a unix-like PC (linux or OSX). The intention is the child thing adds the device specific functionality; the signal_thing_esp8266 code can be used as a starting point.

The code provides a test infrastructure to easily add tests, as demonstrated by the _test_child code in the signal_thing_unix module. For example, tests for device-specific environmental sensors can be added.

The signal_thing_unix code should run on a linux platform, but I did not test it.

There are 2 additional functions that need to be added to make this infrastructure production ready:

• A method of securely storing the AWS keys so they can't be reverse engineered. The ATECC508 can be used.
• Over-the air upgrades. ESP provides this capability, if you sign up with their services. Downloading firmware stored in S3 is being considered as an alternative to have a server-less architecture.