Hardware components | ||||||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
IoT is about connection. Little things connect, and share, and form a gigantic system, just as we humans form our society. But connection doesn’t mean anything if there is no persistence of it — if the components forget about the connection as soon as it vanishes. Can you make a friend if you forget his name as soon as you hear it? Memories make that happen. Memories make connections meaningful. Memories make friends, groups, and societies. Society is itself a living organism. And what evolves a society? Its history.
Similarly in IoT, things need a persistent storage space, a datastore, to put and recall their memories. On Grandeur, datastore is a NoSQL database which leverages serverless and therefore lies at client-side removing any need of a backend to manage a database. This means you can query it directly from your apps and your hardware devices. Imagine it as an infinite storage between your apps and your devices — ensuring high availability and scalability without you ever worrying about them. Millions of your devices could be logging tons of variables each and you can just treat it as a chunk of Big Data, analyze it, pass it through AI models, and learn from it. The sacred journey of making informed decisions begins from persisting the data. Datastore is the base. Combining it with graphing and other features we are rolling out soon, this will give birth to the core of IoT cloud analytics. 📊
This article gives a quick glimpse into the datastore API which wraps the functionality of making basic queries like insert, delete, and search or executing complex query pipelines. This tutorial is part of a whole introductory series and we urge you to checkout our previous one too where we explained why critical IoT applications cannot / SHOULD NOT exist without event-drive.
Basic Datastore Concepts:Before we get our hands dirty, there are some key concepts in datastore which will help you understand how datastore works so you can easily implement your complex use cases.
Datastore is a NOSQL document-style, serverless database. The smallest chunk of information in datastore is stored in a field which is a key-value pair written as {key: value}. But fields cannot exist openly. They are always contained in a document which is equivalent to an SQL record. Here's example of a document:
{
"name":"Tobias",
"nickname":"four",
"s/o":"Marcus",
"gender":"male",
"faction":"dauntless"
}name, nickname, s/o, gender, faction are all fields of this document.
In datastore, documents are classified into collections. A collection is a set of similar, related documents. You can imagine it as a flexible form of an SQL table, without the enforcement of a schema.
A collection being a set means it cannot contain the same document twice. And to make each document different, even when all its fields are same, datastore makes use of auto-generated IDs. So when you insert a document in a collection, datastore generates a unique string and adds it to the document as documentID which makes the inserted document different than any other in the collection.
Enough for today, let's play.
Step 1: Getting StartedIn the previous article, we were sending state update (in data) to the internet after regular intervals. The device, on the other end, had set a listener on data to receive those updates.
In this tutorial, we'll get a step out of our comfort zone. We'll not only update the state variable but also tag each update with a timestamp and log it in a datastore collection. We'll later get the complete state update history from the datastore collection and plot it vs. time to show the update pattern. This is an excellent use case to see during which period has your appliance been ON and OFF.
We'll use the same project and user-device pair and extend the same index.html and main.js files that we used the last time.
Here's our new index.html:
<!-- @file index.html -->
<!DOCTYPE html>
<html>
<!-- Head block of our page-->
<head>
<!-- Title of our page-->
<title>Data Storage with Grandeur</title>
<!-- Link to the SDK CDN -->
<script src="https://unpkg.com/grandeur-js"></script>
<!-- Link to chart.js CDN -->
<script src="https://cdn.jsdelivr.net/npm/chart.js@2.9.3/dist/Chart.bundle.js"></script>
<!-- Link to style sheet -->
<link rel="stylesheet" href="./styles.css">
</head>
<!-- Body block of our page-->
<body>
<!-- Heading of the page -->
<h1>Data Storage with Grandeur</h1>
<!-- This tag displays current status of the app flow -->
<p id="status">Starting...</p>
<!--
This div contains a button and a paragraph tag to
toggle and display the device state
-->
<div id="state-container" style="display: none;">
<button id="toggle-state">Toggle State</button>
<p id="state">State is: false</p>
</div>
<!--
This div contains a table, a graph and a button
to refresh the table and graph data
-->
<div id="graph-container" style="display: none;">
<button id="refresh-graph">Refresh Graph</button>
<canvas id="graph" width="400" height="400"></canvas>
</div>
<!-- Link to the main script file -->
<script src="./main.js"></script>
</body>
</html>And main.js:
/**
* @file main.js - Handles the working of our web app.
*/
const apiKey = "YOUR-API-KEY";
const secret = "YOUR-SECRET";
/** Initializing the SDK and getting reference to our project */
var project = grandeur.init(apiKey, secret);
/** Function to log in the user */
async function login() {
/** Hard-coding user's credentials here. In a production-ready app,
* this will be replaced by a login screen.
*/
var email = "test@grandeur.tech";
var password = "test:80";
/** This sets the status to "logging in" */
document.getElementById("status").innerText = "Logging in";
/** Getting reference to Auth class */
var auth = project.auth();
try {
/** Logging in the user */
var res = await auth.login(email, password);
/** Checking response code */
switch(res.code) {
case "AUTH-ACCOUNT-LOGGEDIN":
case "AUTH-ACCOUNT-ALREADY-LOGGEDIN":
/** If the user's logged in,
* This sets the status to "User Authenticated".
*/
document.getElementById("status").innerText = "User Authenticated";
break;
default:
/** If login did not succeed
* This sets the status to "Authentication Failed".
*/
document.getElementById("status").innerText = "Authentication Failed";
}
}
catch(err) {
/** If an error occurred.
* An error usually occurs only if we are not
* connected to the internet.
*/
document.getElementById("status").innerText = "Network Error";
}
}
/** Calling login here runs it when the app loads */
login();
/** State of our device */
var deviceState = false;
/** Our device's ID */
var deviceID = "YOUR-DEVICE-ID";
/** Mode of graph */
var mode = "r";
/** Configuration for chart */
const chartConfig = {
type: "line",
data: {
labels: [],
datasets: [{
borderColor: '#000000',
steppedLine: true,
data: []
}]
},
options: {
legend: {display: false},
responsive: true,
maintainAspectRatio: false,
scales: {
xAxes: [{
type: "time",
scaleLabel: {labelString: "Time"},
}],
yAxes: [{
ticks:{min: 0, max: 1, stepSize: 1},
scaleLabel: {labelString: "State"}
}]
}
}
};
async function toggleState() {
/** This function toggles the state value and publishes it to the cloud */
/** Toggling the state */
deviceState = !deviceState;
/** Getting current timestamp */
var timeStamp = Date.now();
/** This gets reference to device class */
var device = project.devices().device(deviceID);
/** Forming the packet */
var packet = {
state: deviceState,
changedAt: timeStamp
}
/** This publishes the state update to the cloud */
await device.data().set("", packet);
/** Logging in history collection of our datastore */
await project.datastore().collection("history").insert([packet]);
}
/** Get data from datastore */
async function getData() {
/** Variable to store documents */
var documents = [];
/** Search datastore and get packets */
var history = project.datastore().collection("history");
var res = await history.search();
/** Push documents to array */
documents = res.documents;
/** If the number of documents received
* are less than total. Then query again
*/
var page = 2;
while(documents.length < res.nDocuments) {
res = await history.search({}, undefined, page);
/** Push documents */
documents = [...documents, ...res.documents];
/** Increase page number */
page += 1;
}
/** Loop over the documents */
documents.forEach(doc => {
/** Push data to chart */
chartConfig.data.labels.push(doc.changedAt);
chartConfig.data.datasets[0].data.push(doc.state);
});
}
/** Change graph mode */
async function changeMode(m) {
/** Set mode */
mode = m;
/** Update label */
document.getElementById("graph-mode").innerHTML = mode == "r"? "Realtime" : "History";
/** Clear the graph data and labels */
chartConfig.data.labels = [];
chartConfig.data.datasets[0].data = [];
/** If we are in "history" mode */
if (mode == "h") {
/** Variable to store documents */
await getData();
}
/** Update graph */
window.graph.update();
}
/** After successful user authentication,
* the SDK establishes a real-time connection with the
* cloud. And that change in connection status is propagated
* through the onConnection function here.
*/
project.onConnection((status) => {
/** This callback function is called when the
* connection status changes
*/
switch(status) {
case "CONNECTED":
/** If the app is connected with the cloud,
* this sets the status to "Connected".
*/
document.getElementById("status").innerText = "Connected";
/** Add event listener on device data */
project.devices().device(deviceID).data().on("", (path, update) => {
/** If the mode of the graph is realtime */
if (mode == "r") {
/** Push to graph */
chartConfig.data.labels.push(update.changedAt);
chartConfig.data.datasets[0].data.push(update.state);
/** Update graph */
window.graph.update();
}
});
/* Set mode and label */
mode = "r";
document.getElementById("graph-mode").innerHTML = mode == "r"? "Realtime": "History";
break;
default:
/** If the app is disconnected from the cloud,
* this sets the status to "Disconnected".
*/
document.getElementById("status").innerText = "Disconnected";
}
});When the web app loads it tries to log into the account of the user whose credentials we hard-coded in main.js. After logging in, the SDK establishes the real-time duplex connection with the cloud. Variable updates and datastore queries (insert/delete/search/pipeline), all happen through this real-time connection. After establishing the connection, the app displays "Connected" and gives you a Toggle State button with a text displaying current value of state variable. The app listens for data update as well, so that we know when the state is updated and we can push the realtime feed to our graph.
When you click the Toggle State button, the state value is toggled and the updated value of state along with the current timestamp is pushed to the cloud. The update triggers our data update handler which pushes the timestamped state update to our history collection in datastore. The graph in app can work in two modes 1) realtime, or 2) history. Realtime mode adds the update to the graph as soon as it occurs. In history mode, the app gets the complete state update history from datastore and plots it on graph.
From datastore, we receive an array of objects where each object contains state (true/false) and the timestamp of when that state update occurred. We extract all states and timestamps in separate arrays and plot states vs. timestamps in our history graph.
Our device program stays the same as we designed in the previous article.
/** Including the SDK and WiFi header */
#include <Grandeur.h>
#include <ESP8266WiFi.h>
/** Configurations */
String deviceID = "YOUR-DEVICE-ID";
String apiKey = "YOUR-APIKEY";
String token = "YOUR-DEVICE-TOKEN";
/** WiFi credentials */
String ssid = "WIFI-SSID";
String password = "WIFI-PASSWORD";
/** Variable to hold project and device objects */
Project project;
Device device;
/** Function to check device's connection status */
void onConnection(bool status) {
switch(status) {
case CONNECTED:
Serial.println("Device is connected to the cloud.");
return;
case DISCONNECTED:
Serial.println("Device is disconnected from the cloud.");
return;
}
}
/** Function to handle update in device state */
void updateHandler(const char* path, Var update) {
/** Print state */
Serial.printf("Updated state is %d\n", (bool) update["state"]);
}
/** Function to connect to WiFi */
void connectWiFi() {
/** Set mode to station */
WiFi.mode(WIFI_STA);
/** Connect using the ssid and password */
WiFi.begin(ssid, password);
/** Block till WiFi connected */
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
/** Connected to WiFi so print message */
Serial.println("");
Serial.println("WiFi connected");
/** and IP address */
Serial.println(WiFi.localIP());
}
void setup() {
/** Begin the serial */
Serial.begin(9600);
/** Connect to WiFi */
connectWiFi();
/** Initializes the global object "grandeur" with your configurations. */
project = grandeur.init(apiKey, token);
/** Get reference to device */
device = project.device(deviceID);
/** Sets connection state update handler */
project.onConnection(onConnection);
/** Add event handler on data update */
device.data().on("", updateHandler);
}
void loop() {
/** Synchronizes the SDK with the cloud */
if(WiFi.status() == WL_CONNECTED)
project.loop();
}Here we are finally. Let's test what we have build. For this purpose we hook an ESP8266 to Arduino IDE and we burn the code. Then we open serial monitor to view live activity. But we gotta run our web app to send the data. For that we run a local server in our app folder. But there's a security measure. You cannot send data from your app to Grandeur unless you whitelist your domain (the address that is shown in the browser's URL bar when you open your app) from the Console. This prevents all the unidentified sources from accessing your project's data in case you lose your secret.
Go into the directory of your web app and run the following command in your terminal:
python3 -m http.server 8000 # Serves your web app over a local http serverIf you are using VSCode, you can also use the Live Server extension to start the local server. It also supports hot reload.
When we press Toggle State button, the state is toggled (0 to 1 or 1 to 0) and its updated value is pushed in device data on Grandeur. The device, having put a listener on data, receives the update. The app also listens to the update on data and displays a realtime feed of the data on graph. We can also switch to history mode which shows the complete update trail of state variable.
Here's a bonus step, let's add bit of styling in our app to make it look professional.
/* Custom Font */
@import url('https://fonts.googleapis.com/css2?family=Roboto&display=swap');
/* Apply Font to body*/
body {
font-family: 'Roboto', sans-serif;
margin: 0;
top: 0;
left: 0;
}
/* Apply styling to button */
button {
width: 150px;
height: 40px;
background-color: rgb(36, 107, 206);
color: white;
outline: none;
border: none;
border-radius: 5px;
transition: background-color 100ms ease-in-out;
cursor: pointer;
margin-left: 20px
}
/* Button hover */
button:hover {
background-color: rgb(36, 107, 220);
}
/* Apply styling to h1 */
h1 {
min-height: 25px;
width: 100%;
background-color: rgb(36, 107, 206);
color: white;
padding: 20px;
font-size: 20px;
margin: 0;
box-sizing: border-box;
}
/* Apply styling to p */
p {
margin-left: 20px;
color: grey
}
/* graph */
.graph {
padding: 20px;
box-sizing: border-box;
width: 100%;
height: 400px
}Here are our app and hardware in action:
In this article we highlighted a very basic use case where we stored timed logs on the Cloud to display our device's ON/OFF routine in our web app. This tutorial is part of an introductory series where we highlight a bare-minimum set of tools you'd need while developing your smart product.
We know about this stack because we've been there building our own IoT product (we'll talk about it some day). So you can trust us when we say the IoT dev stack is huge and you are better off using a managed back-end than building one of your own. You can learn it by trying to build your own obviously but at what cost? Time is the most important thing in business; if you don't launch at the right time, you may not launch altogether. While working on our IoT product, we realized this pain and decided to help people like our own selves build fast, performance-efficient, and stable IoT products (apps and hardware programs) while keeping development costs to minimum. We built Grandeur – a managed backend that includes crucial dev tools like devices management, data storage, user accounts, drag and drop visualization and control dashboard builder, OTA updates, and more, and supports opensource hardware. We built this to abstract away the complications of a distributed system. Datastore for example – which we used in this project – is a serverless database which means you don't have to worry about the headaches of deploying your own database cluster, managing it, making sure of its high availability and high throughput by employing techniques like database replication and database sharding, scaling your database when needed, and keeping it secure.
In the coming tutorial, we’ll go a step ahead of local development and see how you can deploy your web apps and host them on Grandeur with a single command.
You can go through the docs or checkout our youtube channel to learn to use the datastore in complement with other features and build pompous apps in a snap. Ask your questions and give us your feedback at hi@grandeur.dev. Join us on discord for a direct communication with us.
Muhammad Abdullah
Comments