Indoor-outdoor environmental classification using tinyML

Nowadays, wireless technologies penetrate all aspects of our lives. Internet of Things (IoT) is the pillars of Smart City concept, which has become a buzzword. Devices used in the IoT are often low-powered with limited computational resources. One of the most viable solutions for Smart City wireless connectivity seems to be the use of long-range, low-power and low-throughput networks like NB-IoT and LTE-M.

As you probably know, a proof of concept (POC) is a demonstration to verify that certain concepts or theories have the potential for real-world application. As a follow up to our first IoT proof of concept, we wanted to see if we could now build an IoT application using the Nordic Thingy:91 development board in conjunction with the Edge Impulse platform. It is well-know fact that hardware developers are using single board computers(SBCs) and development boards as proof-of-concept before going on to build custom hardware for real-world application. This project is intended for those who are planning to scale further the POC.

We have seen several indoor and outdoor classification systems which can be built using Arduino, Raspberry Pi, ESP8266/ESP32 and many other development boards. This work presents an indoor and outdoor classification system which uses temperature, humidity, air quality, air pressure, color(red, blue, green) and light measurements jointly with machine learning algorithm to predict whether the sensing environment is indoors or outdoors. Environmental monitoring has become a primary concern for individuals and businesses because of its impact on personal lifestyle and industrial production. GPS(or more correctly, GNSS) devices are subject to errors caused by satellite or receiver issues, atmospheric and ionospheric disturbances, multi-path signal reflection, and signal loss or blocking. So, here we will be using a new approach called TinyML, an emerging technique with high potential for a low-power environments. Our focus in this project was to understand which are the limits of the indoor and outdoor classification system and how much complexity can the ML models have with it.

Nordic Semiconductors jointly with Electromaker.ioand Edge Impulse was kind of enough to send me a Nordic Thingy:91 development board to participate in the Make it Smart with Nordic Thingy:91 competition.

Nordic Thingy:91 Cellular Development Board

So, as you already understood, the device used in this project is the Nordic Thingy:91 Cellular Development Board built by Nordic Semiconductors. You can use Nordic Thingy:91to monitor every aspect of your indoor and outdoor environment. It allows to measure temperature, pressure, humidity and air quality sensing, as well as a light sensor capable of recognizing R, G, B and IR components. There’s also a low-power accelerometer. The device offers a UART port for programming and debugging as well as a USB-mini port to charge it.

Here are the list of typical applications:

• Indoor/Outdoor air quality
• Home automation and control
• Internet of things
• Weather forecast
• GPS mapping
• Indoor/Outdoor navigation

It comes with an Asset Tracker is an application provided by Nordic Semiconductors. It collects sensor and GPS data from the sensors on the Thingy:91 and sends it to a cloud application by connection to either the NB-IoT or LTE-M network. Check here NB-IoT and LTE-M network coverage and connection quality prior to usage of the Asset Tracker application.

The Nordic Thingy:91 is a very useful development tool for IoT which is marketed for people who want to dive into IoT product prototypes and demos without building hardware or writing firmware from scratch.

Watch this video for an overview of The Nordic Thingy:91 development board.

Data flow diagram

That being said, as it in deep indoors, it is difficult to receive the visible satellite signal because the reinforced concrete and the wall structure block the GPS satellite signal. Outdoor positioning systems based on the Global Navigation Satellite System have several shortcomings that have deemed their use for indoor positioning impractical. The only way to overcome this situation is to use Machine learning algorithm. It really shines when it comes to complex tasks like that.

The data acquisition works based on below flow:

Deployment of the model into target edge device is not covered in this project.

Getting Started with Edge Impulse platform

Edge impulseis a cloud-based platform. Using the Edge Impulse platform, users can train their AI, ML models without deep knowledge of programming or AI, ML concepts. Quantization is the most common form of optimization used when deploying deep learning models to embedded devices. Edge Impulse uses TensorFlow’s Model Optimization Toolkit to quantize models, reducing their weights’ precision from float32 to int8 with minimal impact on accuracy.

• To use Edge Impulse you first have to install nRF Connect and its dependencies on your computer. Download nRF Connect for Desktop for your operating system. nRF Connect is a software built by the Norwegian company Nordic Semiconductor. The nRF Connect app is available for Linux, Windows and macOS
• Download the latest Edge Impulse firmware. We're going to unzip this file.
• Open nRF Connect for Desktop and open the Programmer application.

• You have to tick the EnableMCUboot field to program the Nordic Thingy:91 modem through USB. MCUboot is a secure bootloader for 32-bit MCUs.
• Connect your Nordic Thingy:91 board to your computer using Micro USB cable.
• Turn off the Nordic Thingy:91 using ON/OFF(SW1) selector button.
• The Nordic Thingy:91 needs to be put into serial recovery mode before flashing. Hold the button (SW3) while turning on the Nordic Thingy:91 board.
• Click Add File > Browse and find the unzipped firmware you downloaded previously. You are looking for a file called firmware.hex.
• Click Select device button and choose your Nordic Thingy:91 board.

• Select write, and the program will be uploaded to the Nordic Thingy:91 board.

Well done! You have successfully managed to update the modem firmware and we have the example program loaded, we are ready to connect to Edge Impulse!

Connecting to Edge Impulse platform

• To train a machine learning model with Edge Impulse, create an Edge Impulse account, verify your account and then start a new project.
• Create Edge Impulse project
• Once logged in, open the command prompt and run the below command:

edge-impulse-daemon

You should get a response that looks like the one below.

edge-impulse-daemon
Edge Impulse serial daemon v1.14.6
? What is your user name or e-mail address (edgeimpulse.com)? 
? What is your password? [input is hidden]
? Which device do you want to connect to? /dev/tty.usbmodem2101 (Nordic Semiconductor)
[SER] Connecting to /dev/tty.usbmodem2101
[SER] Serial is connected, trying to read config...
[SER] Retrieved configuration
[SER] Device is running AT command version 1.3.0
Setting upload host in device... OK
Configuring remote management settings... OK
Configuring API key in device... OK
Configuring HMAC key in device... OK
[SER] Device is not connected to remote management API, will use daemon
[WS ] Connecting to wss://remote-mgmt.edgeimpulse.com
[WS ] Connected to wss://remote-mgmt.edgeimpulse.com
? What name do you want to give this device? nordic
[WS ] Device "nordic" is now connected to project "your_project-project-1"

The Nordic Thingy:91 connects quickly to the Edge Impulse server and properly sends data.

That's all! Your device is now connected to Edge Impulse.

Go to Data Acquisition of the Edge Impulse to build your machine learning model.

Collect sensor data by clicking Connect using WebUSB button.Data were collected by placing the Nordic Thingy:91 development board in a static location. These locations fell into two categories: indoor and outdoor. A total of 40 data sets were collected which varied in location(indoor/outdoor) and weather condition.

• Perform Train/Test split data. Train/Test is a method to measure the accuracy of your model. 80% for training, and 20% for testing. Edge impulse have functionality of automatically split data between training and testing sets. Go to Dashboard and click train/test split button.
• Train the model by pressing the Start training button. This process might take around 1-5 mins, depends on your data set sizes. You can add additional dense layer. Dense layer is the regular deeply connected neural network layer. It is most common and frequently used layer.

You should not get the 100% accuracy from your training dataset. If it does, it could mean that your model is overfitting. The solution here to reduce this is either by adding more data or reducing the learning rate. The accuracy of your ML model is directly correlated to the quality of data used to train it.

You can add more photos by following the Data Acquisition steps and then selecting Retrain model in Edge Impulse.

• Retrain the model using data collected.

• Good accuracy in machine learning is subjective. But in my opinion, anything greater than 70% is a great model performance. There is always a trade-off between accuracy and speed. It is a model that offers >90% accuracy and developed in an open format so that it can be easily deployed to any target edge device.
• After the model’s training is completed, the next step is the run the Model testing in the Edge Impulse platform. Click Classify all button.

• Note that it probably won’t be very accurate we used a very small training set!
• With the model trained and tested, we can use it to make classification predictions! Go to Live Classification section of the Edge Impulse.
• Connect your board by clicking Connect using WebUSB button

The built-in led will start to blink. Turns an LED on for one second, then off for one second, repeatedly.

On execution, you should see something like this:

Classification for outdoor environment. If everything works fine, you will receive result such as:

To receive most accurate test result for your model, it is recommended not to use sensor data from same environment with the similar weather condition.

Our results reveal that a TinyML-based device such as a Nordic Thingy:91 development board can be used to collect environmental data while consuming very low power compared to a high-end device. In NB-IoT and LTE-M coverage areas, you can expand this project by adding low-power remote functionality using this link.

Conclusion

Today I tried to give you an overview of how easy it is to train a deep learning model for environmental classification of the indoor-outdoor system with Nordic Thingy:91 development board using Edge Impulse platform. Because of low-power, low-speed and low-cost connectivity with wide-range coverage to IoT applications it is progressively gaining popularity in industrial and research communities. Sometimes trying to run things locally and spend lots of money buying hardware and optimize your PC, it's just not worth it. Especially when there are so many ready-to-use solutions out there like Edge Impulse platform. Methods used in artificial intelligence such as machine learning algorithms like TinyML may be more effective in objectively determining the indoor-outdoor location of the users.

Indoor-outdoor classification is an interesting field of study in computer vision as it may be utilized in many applications, such as, self-conduct, surveillance, robots, etc. Finally, Nordic Thingy:91 is the best development board for proof-of-concept prototyping and for low-cost and DIY automation projects.

References

• Make it smart with Nordic Thingy:91 Design Contest
• Nordic Thingy:91 Getting Started // Cellular IoT
• Connecting the Thingy:91 Using NB-IoT to the nRF Connect Cloud
• Create a cellular IoT project for a Smart Home or Smart City application
• Nordic Semi Thingy:91
• Environmental Monitoring with Nordic Thingy:52
• Step By Step on Nordic Connect SDK for Development
• Connecting constrained devices to the cloud using Zephyr
• How TensorFlow helps Edge Impulse make ML accessible to embedded engineers