How to take care of your grandpas.

The Team

The team is composed of Fabio Di Spazio, Stefano Milani, and Marco Ferraro. We are three passionate Computer Science Engineering students from La Sapienza University in Rome. The project was build as a proof of concept of an Internet of Things application under the help of Professor Ioannis Chatzigiannakis.

Introduction

The idea of the project is to provide a way to remotely monitor the physical state of an elder relative for either medical reasons or to quickly intervene in case of an emergency.

Smartwatches provide all the needed sensors and are extremely easy to use, which is a crucial requirement.

Setup

We went for the Amazfit Bip because of the long battery life, but most smartwatches will work.

Since we need the raw data from the sensors, the Gadgetbridge App is used to avoid the closed source limitations of the smartwatch producers.

Data is then processed by Apache Edgent which takes care of information extraction and computation locally, to optimize the amount of work done by the cloud.

We send the retrieved information to the IBM Watson IoT Platform which takes care of storing data and visualization.

Technical Details

• Amazfit Bip & Gadgetbridge

The first thing to do when you power up for the first time your Amazfit Bip is to connect it via Bluetooth to Gadgetbridge. So after you have installed the app, invoke the discovery activity manually via the "+" button. It will ask you for some personal info that are needed for proper steps calculation on the band. Then it will automatically attempt to discover and pair your Amazfit Bip. When your Bip notifies the pairing process, tap it to confirm the pairing. After that, you can start tracking your health values launching an activity. Gadgetbridge stores raw data in its database. Raw data has to be read as "data as sent by the device". The firmware supports 3 different modes to measure the heart rate:

• a single measurement
• automatic measurement during sleep
• continuous measurement used in the live activity tracking screen

Gadgetbridge allow to automatically export its database once an hour, so due to the fact that we want to keep track of the patient at least 2 or 3 times an hour, we modified a little bit the code of Gadgetbridge. The first thing to do, is to modify the variable exportPeriod by deleting a a multiplication by 60 in PeriodicExport.java a class that you fan find at app/src/main/java/nodomain/freeyourgadget/gadgetbridge/database.

int exportPeriod = autoExportInterval * 60 * 1000;

Then in order to change how this option is displayed in the settings, just modify an other line in strings.xml in app/src/main/res/values.

<string name="pref_summary_auto_export_interval">Export every %d minutes</string>

Now all our activities are saved in the database, more precisely in the table MI_BAND_ACTIVITY_SAMPLE where we can find these fields:

• TIMESTAMP: The recording time was taken in Unix time, rounded to the nearest minute.
• DEVICE_ID: The unique identifier of the device in which the recording was taken, referencing table DEVICE.
• USER_ID: The unique identifier of the user, referencing table USER.
• RAW_INTENSITY
• STEPS: The number of steps taken.
• RAW_KIND
• HEART_RATE: The BPM recorded by the device.

RAW_INTENSITY and RAW_KIND are used together to distinguish for example in which stage of the sleep we are and so on.

• IBM Watson IoT Platform

After creating an account on the IBM cloud service and having selected the Lite (free) version, you'll get access to the IBM Watson IoT Platform Dashboard:

On the left bar, you have access to the main features provided by the platform. You can now go ahead and generate an API key for your organization from the "Apps" tab in the menu. This will provide the connection credentials for communication. "Standard Application Role" is the most general.

At this point it's possible to create a device for your application in the "Devices" tab. Much like with the API key, just select "Add Device" and follow along providing all the information you need about your device. The informations about the device are used to create a file called device.cfg :

org = <your-organization-id>

type = <your-device-type>

id = <yor-device-id>

auth-method = <your-authentication-mode> (usually token)

auth-token = <your-auth-token>

The provided information will later be used to pair Apache Edgent to this specific device and store your transmitted data.

To visualize the data received to the newly created device, one needs to create a new Board from the "Boards" tab and add a Card to the Board to display the selected kind of data stored on the device. When creating a card, select your desired Device and connect a new Data Set. Select the triggering events and the appropriate parameters for a correct tracking, the process should be very intuitive.

Apache Edgent

Apache Edgent permits to retrieve data, filter and send them to the cloud. The complete code is available on GitHub here.

• Setup Apache Edgent

It is possible to download the entire code of Apache Edgent samples here (follow only the download the Edgent Samples step, since we are gonna use an IDE like Eclipse (that's the one I used! ) or Netbeans. We import the directory as new maven project and we obtain several example projects. To start coding a new application we can import only the template directory.

• Let's coding

Using Gadgetbridge we obtain a database of an activity, in particular we use the table MI_BAND_ACTIVITY_SAMPLE to retrieve the steps and the heart rate during the activity.

We define a SmartWatchSensor class that implements the Supplier interface. The interface contains a method get that supply a value, each call to this function may return a different value. The constructor open the file provided by Gadgetbridge and using JDBC we obtain the entire MI_BAND_ACTIVITY_SAMPLE table.

public class SmartWatchSensor implements Supplier<Triple
                                            <String , Integer , Integer>> {
    
    ResultSet res = null;
  
    Triple<String , Integer , Integer> pair = null;
    String timestamp;
    int steps , heart_rate;
    Connection con;
  
    public SmartWatchSensor() {
        try {
            Class.forName("org.sqlite.JDBC");
            Connection con = DriverManager.getConnection(
                                         "path/to/Gadgetbridge.db","", "");
  
            Statement statement = con.createStatement();
            res = statement.executeQuery("select datetime(TIMESTAMP, 
                                                'unixepoch') ,"+ " steps , 
                                 heart_rate from MI_BAND_ACTIVITY_SAMPLE");
  
        } catch (ClassNotFoundException | SQLException e) {
            e.printStackTrace();
            System.exit(1);
        }
    }
    
    @Override
    public Triple<String, Integer , Integer> get() {
        try {
            if(res.next()) {
            timestamp = res.getString(1);
            steps = res.getInt(2);
            heart_rate = res.getInt(3);
            pair = new MutableTriple<String , Integer , Integer>
                                          (timestamp , steps ,heart_rate);
            }
        } catch (SQLException e) {
            e.printStackTrace();
        }
        return pair;
    }
  
}

Instead of "path/to/Gadgetbride.db" you have to insert the path to your file.

The App class contains the main method. Here's the entire code, then we analyse the code line by line.

public class App {
  
    public static void main(String[] args) {
        DirectProvider dp = new DirectProvider();
        DirectTopology topology = dp.newTopology();
  
        String DeviceCfg = "path/to/device.cfg";
        IotpDevice device = new IotpDevice(topology, new File(DeviceCfg));
  
        SmartWatchSensor sensor = new SmartWatchSensor();
        
        TStream<Triple<String, Integer , Integer>> readings = 
                    device.topology().poll(sensor, 1, TimeUnit.SECONDS);
  
        TStream<Triple<String, Integer , Integer>> StepsTriple = 
                    readings.filter(reading -> reading.getMiddle() > 0 );
        TStream<Triple<String, Integer , Integer>> HeartTriple = 
                readings.filter(reading -> reading.getRight() != 0 &&
                                reading.getRight() != 255 &&
                                reading.getRight() != -1 );
  
        
        TStream<JsonObject> stepsJSON = StepsTriple.map(reading ->{
            JsonObject j = new JsonObject();
            j.addProperty("name", "SmartWatchSensor");
            j.addProperty("Date", reading.getLeft());
            j.addProperty("Steps", reading.getMiddle());
            return j;
        });
  
        TStream<JsonObject> heartRateJSON = HeartTriple.map(reading ->{
            JsonObject j = new JsonObject();
            j.addProperty("name", "SmartWatchSensor");
            j.addProperty("Date", reading.getLeft());
            j.addProperty("Heart rate", reading.getRight());
            return j;
        });     
  
        stepsJSON.print();
        heartRateJSON.print();
  
        device.events(stepsJSON, "sensors", QoS.FIRE_AND_FORGET);
        device.events(heartRateJSON, "sensors", QoS.FIRE_AND_FORGET);
  
        dp.submit(topology);
    }
  
}   

Like above instead of "path/to/device.cfg" you have to put the path of the device file created before.

At first we initialize the provider and the topology.

DirectProvider dp = new DirectProvider();
DirectTopology topology = dp.newTopology();

A direct provider runs a submitted topology as a job in threads int he current virtual machine. A job (execution of a topology) continues to execute while any of its elements have remaining work, such as any of the topology 's source streams are capable of generating tuples.

String DeviceCfg = "path/to/device.cfg";
IotpDevice device = new IotpDevice(topology, new File(DeviceCfg));

The IotpDevice is a device connector to IBM Watson platform. This class implements the generic device model IotDevice (interface). This connector is a wrapper over the Watson IoT Platform API, this gives complete control over the construction and configuration of the underlying connection WIoTP.

SmartWatchSensor sensor = new SmartWatchSensor();
TStream<Triple<String, Integer , Integer>> readings = 
                        device.topology().poll(sensor, 1, TimeUnit.SECONDS);

We initialize an instance of SmartWatchSensor. The poll method declare a new source stream that calls sensor.get() periodically. In this case the get method is called every second.

TStream<Triple<String, Integer , Integer>> StepsTriple = 
                    readings.filter(reading -> reading.getMiddle() > 0 );
  
TStream<Triple<String, Integer , Integer>> HeartTriple = 
                readings.filter(reading -> reading.getRight() != 0 &&
                                reading.getRight() != 255 &&
                                reading.getRight() != -1 );

With the filter method we can divide the useful data in two stream one for the heart rate and one for the steps.

TStream<JsonObject> stepsJSON = StepsTriple.map(reading ->{
    JsonObject j = new JsonObject();
    j.addProperty("name", "SmartWatchSensor");
    j.addProperty("Date", reading.getLeft());
    j.addProperty("Steps", reading.getMiddle());
     return j;
});
  
TStream<JsonObject> heartRateJSON = HeartTriple.map(reading ->{
    JsonObject j = new JsonObject();
    j.addProperty("name", "SmartWatchSensor");
    j.addProperty("Date", reading.getLeft());
    j.addProperty("Heart rate", reading.getRight());
    return j;
}); 

Here we create two stream of JSON object. The elements of these streams can be sent to the cloud.

stepsJSON.print();
heartRateJSON.print();
  
device.events(stepsJSON, "sensors", QoS.FIRE_AND_FORGET);
device.events(heartRateJSON, "sensors", QoS.FIRE_AND_FORGET);
  
dp.submit(topology);

The print() method print on the console the elements of the JSON streams.

The events() method publish a stream element with event identifier "sensor" and a fixed Quality of Service level. The QoS is stetted to FIRE_AND_FORGET so the message containing the event arrives at the message hub either once or not at all (at most once).

The submit()method submit an executable, so start the execution of the topology.

• Modifying Maven pom.xml file

Finally we modify the pom.xml file to add the required dependencies.

We uncomment the edgent-analytics-math-3, edgent-connectors-file, edgent-connectors-iot, edgent-connectors-iotp dependencies.

<!-- declare Edgent Analytics dependencies 
-->
<dependency>
    <groupId>${edgent.runtime.groupId}</groupId>
    <artifactId>edgent-analytics-math3</artifactId>
    <version>${edgent.runtime.version}</version>
</dependency> 
  
[...]
  
<!-- declare Edgent Connector dependencies
-->
<dependency>
    <groupId>${edgent.runtime.groupId}</groupId>
    <artifactId>edgent-connectors-file</artifactId>
    <version>${edgent.runtime.version}</version>
</dependency>
  
<dependency>
    <groupId>${edgent.runtime.groupId}</groupId>
    <artifactId>edgent-connectors-iot</artifactId>
    <version>${edgent.runtime.version}</version>
</dependency>
  
<dependency>
    <groupId>${edgent.runtime.groupId}</groupId>
    <artifactId>edgent-connectors-iotp</artifactId>
    <version>${edgent.runtime.version}</version>
</dependency>

We also add another dependencies to use JDBC with sqlite.

<!-- SQLite dependency 
-->
<dependency>
<groupId>org.xerial</groupId>
<artifactId>sqlite-jdbc</artifactId>
<version>3.16.1</version>
</dependency> 

Optionally we can add a plugin that permit to build a running jar with the main class in it.

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>2.4</version>
        <configuration>
            <archive>
                <manifest>
                    <mainClass> <path to main class> </mainClass>
                </manifest>
            </archive>
        </configuration>
    </plugin>
  
[...]
  
</plugins>

Instead of <path to main class> you have to put your main class path (default of the template sample is com.mycompany.app.TemplateApp ).

Finally we can build a runnable jar. As I told before I used Eclipse IDE, so right click of project directory, go to "Run As --> Maven Build..." . You should obtain that window:

At this point we compile the configuration like this:

It's REALLY REALLY important to set the JRE to Java SE 8, so we click on JRE option in the same window as before. If your default JRE is NOT Java SE 8 we select the right one in Alternate JRE .

Finally we can Run. If everything goes correctly you should have this message on the console:

Now we have the runnable jar in target directory of our project.

And we are gonna try it! Open the command line interface and type it:

cd path/to/your/project/directory

cd target

java -jar my-app-1.0-SNAPSHOT-uber.jar

We can see the connection phase between the application and IBM Watson IoT Platform, and then the data that we are sending to the platform. In the IBM Watson IoT Platform dashboard we can see the event:

Improvements

Although connecting the different technologies proved successful, numerous improvements can be made.

Amazfit Bip only provides data when an "activity" is taking place, which in non-intuitive for the user and therefore not very useful. The monitoring should be happening continuously without any user effort.

On the visualization side, IBM Watson IoT Platform doesn't allow for the live data to be exported directly, which means that the monitoring can only happen through the platform. Ideally, one would want the data to be live streamed to an user friendly app on the phone. To allow this to happen, another service must be used, IBM Cloudant, which is a persistent database to store the history of shared data, from which one can export the JSON files into CSV files to be read by different applications.

Find us on LinkedIn:

Fabio Di Spazio

Stefano Milani

Marco Ferraro

Look our presentation:

Proof of concept presentation