LoRa-E5 <> The Things Network <> MQTT

The objective of this project is to monitor and control LoRa-E5 based IoT devices using The Things Network MQTT Integration on a LoRaWAN.

The LoRa-E5 module has an embedded MCU alongside its LoRa modem, however I am most comfortable with ESP32 so I am using the LoRa-E5 just as a LoRa modem with an ESP32 Dev Module. Of the several LoRa-E5 boards, when you only need LoRa-E5 as a modem, Grove LoRa E5 module is the recommended option, thus the same module is used in this project.

Initially, I just wanted to exchange messages between two LoRa-E5 modules without the need of a LoRaWAN so I used the TEST mode and made it work successfully. Checkout this project if you only want to exchange messages between LoRa modules without having to setup a gateway (this is referred as the TEST mode)!

Continuing on to meet the objective of this project, I divided it into the following milestones

• Setting up The Things Network gateway (LoRaWAN)
• Setting up LoRa-E5 device on The Things Network
• Upload the firmware on ESP32 to communicate with TTN
• MQTT Integration
• Monitor and control your LoRa devices from a Python program

Setting up The Things Network gateway

In order to send the data packets from my ESP32 to The Things Network, I need a gateway supporting the same frequency as that of my LoRa-E5 module. This is where I used my Raspberry Pi + RAK2245 Pi HAT.

If you have any other LoRaWAN gateway of same frequency range as that of your LoRa-E5 module and are able to connect to The Things Network through that, it is well and good!

To setup my Pi+RAK2245 LoRaWAN gateway, I simply had to follow the official guide from RAK Wireless and it worked smoothly.

Once the setup is complete, you shall see the Gateway page as follows

Setting up LoRa-E5 device on The Things Network

While configuring the LoRa-E5 device on The Things Network, the following information from the device is required

• DevEUI
• AppEUI

All the LoRa-E5 modules have this information already. We can either choose to use what is already provisioned or generate new EUIs and configure the LoRa-E5 module with it. I chose to use the EUIs that are pre-provisioned on the modules.

To fetch the DevEUI and AppEUI, upload the Serial passthrough firmware on ESP32 Dev Module with the connections as shown in the schematics. Note that the LoRa-E5 module works on 9600 baud so configure your Serial passthrough firmware accordingly.

Once the ESP32 Dev Module is flashed, open the Serial Monitor / CoolTerm and send the following command

AT+ID

This will return something similar to the following

+ID: DevAddr, 25:1B:2E:11
+ID: DevEui, 2C:F7:F1:21:42:60:90:CE
+ID: AppEui, 80:00:00:00:00:00:00:06

Note this information maybe somewhere in a notepad. We will need it in the next steps.

Lets now configure The Things Network (TTN) so we can send and receive data between ESP32+LoRa-E5 and TTN with the following steps:

• On the Things Stack console, select a region of your choice and go to Applications and create a new application

• Now you have a Things Stack application created. Once you are in the applications console (as shown above), click on + Add end device
• I wasn't able to find LoRa-E5 in the LoRaWAN Device Repository so I provided the information manually in the Manually section
• Select the frequency plan of your LoRa-E5 module, the LoRaWAN version (depends on your gateway), Regional Parameters version. I found this info here
• Now, enter the DevEUI and AppEUI you noted in the earlier step
• Generate a new AppKey and make a note of it somewhere.

Your Things Stack console configuration is pretty much completed to exchange messages to and from the ESP32+LoRa-E5 device

Upload the firmware on ESP32 and communicate with TTN

This firmware when configured with the AppKey (noted above) and uploaded to ESP32, connects to a LoRaWAN (TTN) gateway in its vicinity and sends the following JSON object once in every 5 seconds

{
    "state":  {
                "temp": 27.4,
                "humi": 89
              }
}

At present, the firmware sends constant values of temp and humi to TTN, however you can modify the firmware to suit your requirement of integrating any sensor and have those values sent instead.

Make sure you replace the AppKey you noted from the earlier step in line 101 of the firmware code. I see the following in my Serial port once the code is uploaded successfully

The JSON gets converted into byte array (i.e. 7B 22 73 74...) and is sent to TTN. You could see this data on the TTN console by selecting the End device's Live Data tab

If you like to view this data in byte array as JSON, go to Payload formatters tab and select Formatter type as Javascript and change the formatter parameter to the following

function decodeUplink(input) {
  return {
    data: JSON.parse(String.fromCharCode(...input.bytes)),
    warnings: [],
    errors: []
  };
}

Save it

Go back to Live data tab, select an entry of type 'Forward Uplink Data Message' and you should now be able to see the sent LoRa data in JSON

Just like how the firmware allows you to send data from the ESP32 to TTN, it also allows you to receive data in ESP32 from TTN.

To send data from TTN to ESP32, go to Messaging tab and select the Downlink tab. I'm sending the following JSON from TTN to the ESP32

{
    "name": "Sufian",
    "id": "sufiankaki"
}

TTN console has options to send data as JSON as well as byte array. To send JSON data, we need to modify the Payload formatter as we did earlier (for Downlink). Change the existing script to

function encodeDownlink(input) {
  return {
    bytes: JSON.stringify(input),
    fPort: 1,
    warnings: [],
    errors: []
  };
}

function decodeDownlink(input) {
  return {
    data: {
      bytes: input.bytes
    },
    warnings: [],
    errors: []
  }
}

Once done, save it and go to the Messaging tab and select Downlink. Select JSON and enter the data in the Payload text box and hit Schedule Downlink

On your Coolterm/Serial Monitor, you should see a message being received now

Now that you know how to send (Uplink) and receive (Downlink) data from a LoRa module to TTN, modify the firmwave and develop your project accordingly

MQTT Integration

Although monitoring and controlling the LoRa devices from TTN is simple enough, I prefer having it controlled from my custom built applications. Once such method to do is by using the MQTT Integration offered by TTN. This allows interacting with the LoRa devices using MQTT clients

One of the simplest option to get an MQTT client is using the Mosquitto (CLI) installed in your computer. Alternatively, you can try any MQTT Explorer to interact with a GUI.

To enable MQTT integration for your LoRa End device, select the Integrations option on the menu (left) in your TTN End device console and select MQTT

You will see the host name, port number and username already. Click on Generate new API key so you have the password as well

In addition to the 4 fields you see on the MQTT integration page (hostname, port, username and password), you will also need the topic to subscribe or publish to TTN, depending on whether you want to receive the messages from LoRa device or send messages to LoRa device respectively.

To receive messages sent from LoRa device to TTN, the topic you need to subscribe to is v3/<TTN App ID>/devices/<TTN App - End Device ID>/up. Replace the <TTN App ID> and <TTN App - End Device ID> with respective values. Thus, your CLI command to receive messages from a LoRa device will be

mosquitto_sub -h eu1.cloud.thethings.network -p 1883 -u test-app-868@ttn -P NNSXS.IG5UN5CKVNA7IWCQBLS5XXCEGRKNKBEDIJTFUUY.BZSNJY6HJ2KRF36XOPLXZF45AGFZSR2AYTT4L3HF45BBSGH4RG6Q -t 'v3/test-app-868@ttn/devices/eui-70b3d57ed004d84c/up'

Once subscribed, you will receive message (JSON) like the following

This contains all the information pertaining to the end device, gateway, metadata, etc pertaining to the uplink message from LoRa device to TTN.

To send data from MQTT client to LoRa device, the topic you need to publish message is to v3/<TTN App ID>/devices/<TTN App - End Device ID>/down/push. Also, the message needs to be in the following format

{
    "downlinks":
        [
        {  "f_port": 1, 
            "frm_payload":"ewogICAgIm5hbWUiOiAiU3VmaWFuIiwKICAgICJpZCI6ICJzdWZpYW5rYWtpIgp9", 
            "priority":"NORMAL"
        }
        ]
}

The message you need to send should be base64 encoded. Say you want to send the following message

{
    "name": "Sufian",
    "id": "sufiankaki"
}

you first need to base64 encode it. I used this online tool to convert my message. For the above message, I got ewogICAgIm5hbWUiOiAiU3VmaWFuIiwKICAgICJpZCI6ICJzdWZpYW5rYWtpIgp9 as my base64 encoded string

Thus my MQTT publish message looks like

mosquitto_pub -h eu1.cloud.thethings.network -p 1883 -u test-app-868@ttn -P NNSXS.IG5UN5CKVNA7IWCQBLS5XXCEGRKNKBEDIJTFUUY.BZSNJY6HJ2KRF36XOPLXZF45AGFZSR2AYTT4L3HF45BBSGH4RG6Q -t 'v3/test-app-868@ttn/devices/eui-70b3d57ed004d84c/down/push' -m "{\"downlinks\":[{\"f_port\": 1, \"frm_payload\":\"ewogICAgIm5hbWUiOiAiU3VmaWFuIiwKICAgICJpZCI6ICJzdWZpYW5rYWtpIgp9\", \"priority\":\"NORMAL\"}]}"

Make sure you escape the double quotes within the message.

For more info on MQTT integration, check the official docs. My explanation above is for Eclipse Mosquitto.

Monitor and control your LoRa devices from a Python program

Using the MQTT integration, the LoRa devices could be monitored and controlled from a python program as well. I'm going to use Eclipse Paho library to send and receive messages from my Python program to LoRa devices

Program to receive and print messages from LoRa device

import paho.mqtt.client as mqtt
import json

def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc))
    client.subscribe("v3/test-app-868@ttn/devices/eui-70b3d57ed004d84c/up")

def on_message(client, userdata, msg):
    messg = json.loads(msg.payload)
    print(messg['uplink_message']['decoded_payload'])

client = mqtt.Client()
client.username_pw_set("test-app-868@ttn", "NNSXS.IG5UN5CKVNA7IWCQBLS5XXCEGRKNKBEDIJT..")
client.on_connect = on_connect
client.on_message = on_message

client.connect("eu1.cloud.thethings.network", 1883, 60)
client.loop_forever()

Program to send messages to LoRa device

import paho.mqtt.client as mqtt
import json
import base64
from time import sleep

def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc))
    client.subscribe("v3/test-app-868@ttn/devices/eui-70b3d57ed004d84c/up")

def on_message(client, userdata, msg):
    messg = json.loads(msg.payload)
    print('Received: ' + str(messg['uplink_message']['decoded_payload']))

client = mqtt.Client()
client.username_pw_set("test-app-868@ttn", "NNSXS.IG5UN5CKVNA7IWCQBLS5XXCEGRKNKBEDIJT...")
client.on_connect = on_connect
client.on_message = on_message

client.connect("eu1.cloud.thethings.network", 1883, 60)
client.loop_start()

try:
    while True:
        topic = "v3/test-app-868@ttn/devices/eui-70b3d57ed004d84c/down/push"
        js = {'name': 'Sufian', 'id': 'sufiankaki'}
        str_js = json.dumps(js)

        x = {}
        x["f_port"] = 1
        x["frm_payload"] = base64.b64encode(str.encode(str_js)).decode()
        x["priority"] = "NORMAL"
        
        message = {}
        message["downlinks"] = []
        message["downlinks"].append(x)

        client.publish(topic, json.dumps(message))
        print('Published: ' + json.dumps(message))
        sleep(5)
except KeyboardInterrupt:
    client.loop_stop()
    print("Stopped the client!")

Thanks for going through this project, please do like and share if you found it useful. Also if you like me to help you with your LoRaWAN implementation, feel free to reach out :)