Menopause Ultimate tracking device - Stay connected

About.

The project consist of a low cost device that will help women to track symptoms of menopause. The device is connected to the internet and a simple menu allow the user to navigate and report a symptom whenever it is experienced. The data is transmitted to the cloud where it is safety stored and make it available for doctors and physicians as well as the user. The platform can send reminders to the user as a follow up after certain symptom have been reported or a given AI algorithm dictate an action.

The Low cost Solution.

The low cost solution chosen involves a small hardware device that can be found for around 30$.

Thou is low cost, the same functionality can be implemented using an smart phone device application.

There was one good reason to choose a hardware platform such as the M5Stack, besides the fact that I am more familiar with developing on this embedded devices than in android apps for the purpose of showing a working demo. There are many ways that a cell phone can be turn into negative aspects of life such as the constant obsession to check it out for messages for example.

The Architecture.

The system can be better explained with the help of a diagram.

The user with the aim of the device will report symptoms from a list of common and not so common symptoms. The device will upload the reported values to the cloud using MQTT protocol, the way to do this is by publish a message to a topic. The device can also subscribe to a given topic to receive notifications.

The cloud implementation has to be secure as this aspect is very important for medical applications that handle patient information. The cloud platform chosen is Amazon AWS IoT as it offers several mechanism to help in transfer securely and efficiently the MQTT messages.

The MQTT messages are received in the Amazon IoT core and with the help of lambda functions, the messages are forwarded to other MQTT endpoint, for instance there is an endpoint to which the Reporter device publish messages, when does messages are received by AWS, the lambda function forward it to a subscribe endpoint that a client can subscribe. In this case the client is a cloud server running in an amazon EC2 instance. This way is possible to have a serverless mqtt broker and have different securities for the devices and for the EC2 server instance.

The amazon EC2 instance is running Ubuntu OS in which the Web Server is running. For the purpose of the demo the Web Server handles the communications with the MQTT IoT core of AWS, this might be done separately if the traffic demands it.

The Web Server is implemented using Flask. Flask is built using python language and we can hook up the amazon IoT SDK for python to allow creation of a client that can subscribe and publish messages to communicate with the symptom reporter device. The data has to be persisted, for this several options and thou decisions can be made in terms of what technology to use. For a production scenario, the decision made here is probably not the best or maybe it is, but for now this consideration is postponed in order to build a working demonstration.

The data received from the MQTT endpoint is saved to a InfluxDB database. InfluxDB is a time series database which means is more efficient at recording large number of values over time. This is good choice to store symptoms data because those are values that always will have a timestamp associated. An SQL database will be more suited to store patient data that will not change with time such as name, social security, age, etc. For this we can use the built in database of Flask.

Last but not least, to display certain symptoms in graphs, a dashboard framework called grafana is used. This solution allow us to directly connect the dashboard to our time series database using native InfluxDB plugin. The dashboard can be inserted in our web server pages to provide a rich experience to users that look thru the data in search of menopause evidence.

The system can be extended to provide AI capabilities in the cloud by connecting an AI engine node to our database and look for different events, trends or indications of possible menopause symptoms and help in provide a response to the user that help in her process.

The Low Cost Hardware.

I was looking for an off the shelf device that can provide different interfaces to build around a complete solution, one that can even be expanded. As commented before, the demo at this moment only provides the connectivity and with a basic button and lcd interface it provides the capability to report symptoms to the menopause cloud solution. It came out the M5Stack device, thou different flavors exist, the Gray version is around 30$ and provides plenty of other interface for expansion in the future such as external sensors.

The device demo is programmed in Arduino. The WiFiClientSecure is an ESP32 secure class for a client connection, this class is used by mqtt library to connect to Amazon AWS using the certificates and security that is required, i.e.

WiFiClientSecure net = WiFiClientSecure();
MQTTClient client = MQTTClient(512);

// Configure WiFiClientSecure to use the AWS IoT device credentials
net.setCACert(AWS_CERT_CA);
net.setCertificate(AWS_CERT_CRT);
net.setPrivateKey(AWS_CERT_PRIVATE);
// Connect to the MQTT broker on the AWS endpoint we defined earlier
client.begin(AWS_IOT_ENDPOINT, 8883, net);

An excellent library for making effortless menu interfaces onM5Stack is found here. The M5ex library is very good documented and easy to integrate once you get used to. The library also offers a complete solution for M5Satck by leveraging other tasks such as WiFi connectivity and button events. A few concepts have been learned to allow integration with MQTT library.

The MQTT has a function that need to be called eriodically, this has to go to an addEvent function such as

ez.addEvent(mqtt_loop);

This in turn call he function that return the next period to wait before calling in again, in our case, calling client.loop() and waiting 50 ms is enough to keep running the mqtt client.

uint16_t mqtt_loop(){
client.loop();
return 50;
}

The menu and device communication protocol.

The menu is based on a simple selection done using three keys that help to select and go up or down the different options until the final action is complete such as reporting a specific symptom characteristic.

Below is a schema that shows how it works the selection for period bleeding characteristic.

Some of the menu are for now just place holders for the functions to collect the data and send to the cloud. For demonstration purposes the Period symptom menu and all of it's sub's menu are fully functional. So I am going to focus on some of the key features here, some of this might or not apply to other menu/sub menu but in general it will give you an idea of what the application will looks like.

As you can see in above diagram, the buttons allow you to move from one menu to the next until you select the desired option. The state of the application will dictate if the flow is allow or not. Let's for example assume the user haven't report any Period symptom such as the application is in initial state.

As you can see in Period submenu, you can report start date, which means the date in which the user got the period in a given month. If the user go and try to report an end date without having specified the start date, the system will show up a dialog informing the user about the error and how to correct it.

This is done in the code with the help of a global variable

bool startperiod = false;

which is set to false by default. For simplicity we are not updating the status with the server, which is a further step in case the device experience a power cycle, but in both cases it might be good practice to implement a storage mechanism and an algorithm to keep track of this on the device without the use of connectivity, this way symptoms can still be reported and later on uploaded when connection is reestablished. At the moment the implementation only consider dates to be the value for the RTC which is updated from an NTP server, but a more flexible design might allow users to enter different values such as to report a symptom from a previous day.

Depending on the given sub menu, different actions can be taken to assure correct report of a symptom, for instance a user can go straight to report bleeding type under the Period symptom menu without first specifying the start date, the software will set both values. In order to do that, the values are grouped by symptoms almost the same as you might find in a sub menu. Let's take a look at how the Period information is transmitted.

MQTT protocol don't restrict per se the packet length but brokers normally limit it's size, in order to effectively transmit data, JSON file format is used. A pretty print of how a full packet to send Period data will look like is shown below.

{
  "userid": "user123",
  "time": 1351824120,
  "symptom": {
    "period": {
      "startdate": 1351824120,
      "enddate": 0,
      "bleeding": "heavy",
      "crampspain": 55,
      "dayrate": 5
    }
  }
}

Under the period field, one or more different annotations of symptoms can exist. The Server will decode accordingly. There are a few things that are free to be change either in the server side or in the device side. For example, the day rate is a measure from 1 to 10 of how do you feel on that given day, this is a subjective measure but certainly something doctors normally ask, and so for those days the women is having the Period is a good indication of her overall experience. A day has 24 hours, now imagine you got so far a very good day, so you are happy to not have to much pain or headaches or any other consequence of having the Period, then by afternoon you decide to set your day rate as 3 in the 1 to 10 scale. After dinner you become feeling tired and by the time you are trying to have some sleep you are getting really bad cramps and guess you get as well some insomnia, sounds bad right? well you decide to track at midnight 8 on the 1 to 10 scale for the date. If your application is limited to get only the first value you are actually rejecting data which is not very good idea.

In the above example we can see a lot of opportunities to improve or change in order to provide better diagnostics.

The scale menu is also implemented and for now you are allowed to send new values every time you wish. Here is how the menu looks like.

All the sub menus that require this functionality can use a common function as shown below.

bool level_1to10_menu(ezMenu* callingMenu) {
    Serial.print( callingMenu->pickCaption() );
    ez.header.show("Record Level");
    ez.canvas.font(&FreeSans12pt7b);
    ez.canvas.lmargin(20);
    ez.canvas.println("");
    ez.canvas.println("Record the Level of:");
    ez.canvas.println(callingMenu->pickCaption());
    ez.buttons.show("Up # Down # Done");
    uint8_t value = setNumberInScreen();
    time_t currentTime = now();
    StaticJsonDocument<512> doc;
    doc["userid"] = "user123";
    doc["time"] = currentTime;
    if(!startperiod){
        startperiod = true;
        doc["symptom"]["period"]["startdate"] = currentTime;
    }
    if(callingMenu->pickCaption() == "cramps pain"){
        doc["symptom"]["period"]["crampspain"] = value;
    }else if(callingMenu->pickCaption() == "day rate"){
        doc["symptom"]["period"]["dayrate"] = value;
    }else{
        return true;
    }
    char jsonBuffer[512];
    serializeJson(doc, jsonBuffer);                           // print to client
    client.publish(AWS_IOT_PUBLISH_TOPIC, jsonBuffer);
    return false;
}

The Cloud side of things.

The use of Amazon AWS IoT was essential for testing and demonstration and a good idea to focus on a production solution. The following section will highlight key aspects of the configuration and point to some information on the world wide web that can be follow to understand the different steps done.

First of all, a serverless MQTT is essential, for this we use Amazon IoT core and configure a device as shown in this guide. The key aspect is to create the device thing in Amazon AWS and download the certificates, which has to be open with text editor and place the content in the secrets.h file.

Then you define a couple of topics to which your device will publish and subscribe, in my case I stay with the ones originally used by the author of that guide for simplicity, that is.

#define AWS_IOT_PUBLISH_TOPIC   "esp32/pub"
#define AWS_IOT_SUBSCRIBE_TOPIC "esp32/sub"

I change the thing name to

#define THINGNAME "M5StackGray"

Connection is done by

// Configure WiFiClientSecure to use the AWS IoT device credentials
    net.setCACert(AWS_CERT_CA);
    net.setCertificate(AWS_CERT_CRT);
    net.setPrivateKey(AWS_CERT_PRIVATE);
    // Connect to the MQTT broker on the AWS endpoint we defined earlier
    client.begin(AWS_IOT_ENDPOINT, 8883, net);
    // Create a message handler
    client.onMessage(messageHandler);
    Serial.println("Connecting to AWS IOT");
    while (!client.connect(THINGNAME)) {
        Serial.print(".");
        delay(100);
    }

I print over serial the status of the connection but it is also under the status menu.

The next step will be to send any of the Period menu symptoms or events and confirm you can see the data in AWS under test tab. For this you have to subscribe to the esp32/pub topic.

You should also send fromAWS to the esp32/sub topic and confirm the M5Stack can receive. There is a function called messageHandler that at the moment only prints the message on the screen, but is a place holder for a message parser function.

void messageHandler(String &topic, String &payload) {
    Serial.println("incoming: " + topic + " - " + payload);
    StaticJsonDocument<200> doc;
    deserializeJson(doc, payload);
    const char* message = doc["message"];
    Serial.println(message);
    M5.Lcd.setCursor(3, 55);
    M5.Lcd.setTextColor(YELLOW);
    M5.Lcd.setTextSize(2);
    M5.Lcd.println(payload);
}

The EC2 configuration on Amazon. This is needed for seeting up the different server, we need the InfluxDB, Grafana and Flask. For reasons of time, the demo will cover only the Flask server. The remaining work is to save the data in the InfluxDB and configure grafana to access it using a plugin.

Flask is based on python. A skeleton to build on top the project can be found here. Download and install as the instructions. It's nice to have as you will already have a working skeleton with database and follders to store the views for example.

When using the EC2 instance to run your servers, you have to open different ports to access them. For testing I open them only to my IP address.

I was accessing the EC2 from two different machines, so you probably need one set of ports. SSH on port 22 is a must. 5000 is the port that uses Flask, 3000 is grafana and 50050 to 50100 are ports used by WingPro python IDE to remotely debug the Flask server.

Now you need to install the python sdk for Amazon AWS. I have omitted to mention a few steps while installing Flask and the SDK. First, you better use a virtual environment for your Flask deployment, instructions can be found here.

Next, you need to create another amazon "thing" for your EC2, in this case we need to be able to publish to esp32/sub topic and subscribe to esp32/pub topic. For this you need to change the rule policy as shown below.

Last but not least, generate new certificate and download them as you will need to store in a folder under your EC2 instance and point the python SDK to those files as we will see in a minute.

Under the __init__.py file in the app directory, create the MQTT client instance by declaring

myMQTTClient = AWSIoTMQTTClient("EC2_manager")

and then doing the following in the create_app function.

# Instantiate app.
    app = Flask(__name__)
    
    myMQTTClient.configureEndpoint("a2n6lks10bxnsf-ats.iot.eu-west-1.amazonaws.com", 8883)
    myMQTTClient.configureCredentials("/home/ubuntu/Devel/AWS/rootCA.pem", "/home/ubuntu/Devel/AWS/3ba5efb1e5-private.pem.key", "/home/ubuntu/Devel/AWS/3ba5efb1e5-certificate.pem.crt")
    myMQTTClient.configureOfflinePublishQueueing(-1)  # Infinite offline Publish queueing
    myMQTTClient.configureDrainingFrequency(2)  # Draining: 2 Hz
    myMQTTClient.configureConnectDisconnectTimeout(10)  # 10 sec
    myMQTTClient.configureMQTTOperationTimeout(5)  # 5 sec
    
    myMQTTClient.connect()

After that you can reference the instance in the views.py by doing

from app import myMQTTClient

for testing you can send a hello text to the M5Stack whenever there is a hit on the index page been serve, such as

@main_blueprint.route('/')
def index():
    print ('hello', flush=True)
    myMQTTClient.publishAsync("esp32/sub", "{\"text\":\"Hello world!\"}", 0)
    return render_template('index.html')

for receiving the messages you might want to create an on_message function under the wsgi.py file such as

def on_message(client, userdata, message):
    print("Received a new message: ")
    print(message.payload)
    print("from topic: ")
    print(message.topic)
    print("--------------\n\n" , flush=True)

then you can subscribe to the topic on the main function

if __name__ == '__main__':
    myMQTTClient.subscribe('esp32/pub', 1, on_message)
    
    if 'WINGDB_ACTIVE' in os.environ:
        app.debug = False
    app.run(host='0.0.0.0', use_reloader=True)

of course you need an import on this file as well

from app import create_app, db, models, forms, myMQTTClient

And that is pretty much for receiving messages.

If you see the video, in the end you can see the command as it gets into the Flask server, the JSON contains the date, which I am setting in arduino using now() funtion ezTime library. Notice the number value of 1589324842, if you go to epoch converter you will see that I do not have more time to continue adding to this contest project. So hopefully you like the project.

Final words.

Thou the project is not finish to the point of seeing a final web server and fancy app shining with the latest patient data, we got into a point that proof the concept and open the field for a large opportunities to help keep track of menopause symptoms in a cool and easy way that probably can make an impact in women treatment of menopause.