SAJAC PROJECT : Smart Jacket for Caving

Problem

Caves, which are quiet places and rarely passed by humans, often have extraordinary beauty. The beauty that is in it is very different from other natural landscapes. Caves often still keep a very pure natural beauty because of the lack of human activity in them. Caving is one of the exploration options to see this natural beauty.

Caving has become a hobby that has become increasingly popular in recent decades. Caving is sometimes done just for the pleasure of doing the activity or for physical exercise, still the beginning of exploration, or physics and biology also play an essential role. Caving has become even safer because of modern clothing and equipment.

Are all the equipment carried by explorers able to protect them 100% from the dangers? Certainly not. Many other dangers can threaten the safety of explorers in the cave. What are those things? Extreme temperatures, wet and humid air, low air pressure, poor air quality, and the potential for toxic gases. Moreover, the lack of a communication system sometimes worsened matters as explorers could not contact the outside world.

Solution

Related to this problem, I will develop the Sajac Project to help people who have a hobby of caving. Sajac Project is a smart monitoring system aims to observe environmental conditions in the caves. This system will provide information about how the environment around the users. This machine is equipped with machine learning to determine whether the environmental conditions and user friendly. At each checkpoint inside the cave, a transmitter will be connected directly to the guards outside. If the user gets a danger notification from his jacket, user can immediately ask for help at the nearest checkpoint using the transmitter, and the guards can immediately provide help as soon as possible.

How it Works?

This system uses Nicla Sense ME to condition determination. This tools can at one measure environmental quality around the user and send the result to Sajac Application in User’s smartphone. If the cave condition is not good enough to explore, there will be a notification alert in Nicla Sense ME or Users’s smartphone. This calculation is carried out using a model that previously applied on edge impulse platform with a high level of accuracy and small error rate.

At each checkpoint, there will be a transmitter that will directly inform the cave environmental conditions to the post guard. Considering there is no internet connection in the cave, this system will use LoRa communication system to transmit data from the checkpoint. This system is developed using XIAO ESP32C3 that is equipped with Grove Wio E5 from Seeed Studio. When the users arrives at the checkpoint, he only need to press “send” button after connecting to the transmitter at the post. The user also can determine whether he will return by him self or be picked up by the guards if the condition is impossible for him to continue his exploration.

The guards at the main post will receive data transmitted from the cave by using LoRa. At the front post, there is Wio Terminal Equipped with Grove Wio E5 to receive data from the transmitter inside the cave. Wio Terminal is effective to use in place with minimum power sources because it only needs 5 volt of power from powerbank.

Step by Step

• 1. Setup the Nicla Sense ME in Arduino IDE
• 2. Testing Sensor using Sense ME Dashboard
• 3. Testing Transmit Sensors Data using BLE
• 4. Gui Design using MIT App Inventor
• 5. Receive Data from Nicla Sense ME using Android
• 6. Transmit Data from Android to XIAOESP32C3
• 7. Send Notification from XIAOESP32C3 to Wio Terminal
• 8. Show notifications to the Guards using Wio Terminal
• 9. Collect data
• 10. Data Training using Edge Impulse
• 11. Model Implementation
• 12. Testing the Device

1. Setup the Nicla Sense ME in Arduino IDE

The Nicla Sense ME is a tiny, low-power tool that sets a new standard for intelligent sensing solutions. With the simplicity of integration and scalability of the Arduino ecosystem, the board combines four state-of-the-art sensors from Bosch Sensortec:

• BHI260AP motion sensor system with integrated AI
• BMM150 magnetometer
• BMP390 pressure sensor
• BME688 4-in-1 gas sensor with AI and integrated high-linearity, as well as high-accuracy pressure, humidity and temperature sensors.

Nicla Sense ME can be programmed easily with the Arduino IDE. Before you can use it, you must do some setup. Setup process consist of installing the board and some important libraries on Nicla Sense ME on the Arduino IDE. You can easily follow the tutorial in the video below.

Installed Boards:

Installed Library:

• Arduino_BHY2
• Arduino_BHY2Host
• ArduinoBLE
• EdgeML-Arduino
• Arduino_ConnectionHandler
• ArduinoIoTCloud

Important note: I am using Arduino Mbed OS Nicla Boards version 3.5.1 and it makes me unable to charge the Nicla boards. You need to use the latest version which ignores ntc inside the boards.

2. Testing Sensor using Sense ME Dashboard

Nicla Sense ME can provide users with 65 variable sensor readings. The 65 variables are included in 6 different classes:

• Sensor: This class handles all the other sensors which have a single value to be read, like temperature, gas, pressure, etc. And also the event sensors, like the step detector. This generic sensor class provides the sensor data through the value property that returns a float.
• SensorOrientation: This class handles sensors with the Euler format, used for example with orientation. It allows you to read the pitch, roll and heading property.
• SensorXYZ: This class handles sensors with the XYZ format, like the accelerometer and the gyroscope. It contains x y and z values
• SensorQuaternion: Can be used to handle sensors with the quaternion format, can be used to calculate rotation vector, game rotation vector and geomagnetic rotation vector. You can access the x, y, z and w property using this class.
• SensorActivity: Use this class to handle sensors with the activity format. The activity is encoded as ID and can be retrieved from the value property. Use getActivity to get a human readable version of the activity e.g. "Walking activity started".
• SensorBSEC: BSEC stands for Bosch Sensortec Environmental Cluster, basically you can access the air quality (IAQ) level.

For more information, you can see the official Arduino Nicla Sense ME Cheat Sheet documentation.

Nicla Sense ME can give you information about the environment such as pressure, temperature and gas readings. Sometimes, you may have to place the sensor in a hard-to-reach area due to certain environmental requirements. Therefore, it will be much convenient and helpful to access the data wirelessly. To demonstrate this, Arduino prepared a simple sketch and hosted a dashboard so you can try it yourself. You can download the code here or at the code attachment below.

You only need to upload it without changing anything, then you can enter Arduino Nicla Sense ME - Web BLE Test. This section is expected to help you better understand the sensors contained in Nicla Sense ME better.

3. Testing Transmit Sensors Data using BLE

In this project, a system will be created that can detect environmental conditions around the user. There are several selected parameters to be used:

• Temperature
• Humidity
• Air Pressure
• Indoor Air Quality
• VOC gas

Some of these sensors can be accessed easily by following the instructions on the Arduino Nicla Sense Cheatsheet. To try it, let's display it on the serial monitor using the following code, which you can access on this Github link or in the attachment.

Now we can read the value of the sensor that will be used. Furthermore, the data that has been read will be sent using BLE to be read by other devices. BLE stands for Bluetooth Low Energy (Bluetooth LE, marketed as Bluetooth Smart). Bluetooth Low Energy (BLE), sometimes referred to as “Bluetooth Smart”, is a lightweight subset of classic Bluetooth. and was introduced as part of the Bluetooth 4.0 core specification. We will use our smartphone as a receiver and Nicla Sense ME as a transmitter to transmit data. Therefore, we need an application to display data from Nicla Sense ME. The application that will be used is LightBlue which can read data from the transmitter via BLE.

Data that has been read from the sensor on Nicla Sense ME will be sent using BLE. We can modify the Nicla Sense Dashboard program in the previous step for this project. It is necessary to make a few modifications to the program, which will significantly affect the data displayed in the application. The change is to change the variable data format sent to "Char/String". This is due to simplifying the process of displaying data in the next step. You can try the program on the GitHub link or on the attachment. (additionally, you can compare this program with the Nicla Sense Dashboard program to see the difference).

Data will be received in Hex format. Now let's see what we get in the LightBlue app. 😀

4. Gui Design using MIT App Inventor

Now we can send the data from Nicla Sense ME using BLE. The next process is to create a simple platform that can help users read Nicla Sense ME's readings. The application that is made must be able to operate without the internet because the place where the user is located is in a cave.

Application development will be carried out using the MIT App Inventor. MIT App Inventor is an open-source application originally developed by Google, and currently managed by the Massachusetts Institute of Technology. App Inventor allows new users to program computers to create software applications for the Android operating system. MIT App Inventor was chosen because of its ease to use, especially for those who are just learning how to make software.

MIT App Inventor has complete features, starting with the components needed to design and create the program. Another interesting thing is that MIT App Inventor can be programmed with scratch/block so it will be very helpful in the application development process. You can start learning how to use it on this YouTube playlist.

And this is the GUI design of the Sajac Application.

BLE components are not actually available by default in the MIT App Inventor application, but you can add them to your project. You can download it for free from the official MIT App Inventor repository at this link. Then you just need to add it directly to your project. Don't worry, I also give a video how to add it directly to your project. Please follow the video below.

Important note: Please use the latest version of the extension. I'm using the previous version and it doesn't work well on the app that I'm building.

5. Receive Data from Nicla Sense ME using Android

The next step is the process of generating the code. In the previous video, there is also a way how to connect the application that you are developing to BLE. You can follow step-by-step from the explanation in the video.

As we know, there is a UUID service on Nicla Sense for sending data, and there is a Characteristic UUID for each section of data sent. We can use this to send each data in a different Characteristic UUID. Nicla Sense ME will send 7 variables, namely user ID, temperature, humidity, air pressure, gas parameters, air quality, and one additional variable to display environmental conditions from the results of edge impulse machine learning. Each data will be sent on a different UUID, so it is necessary to make different readings for each data.

The way to read is quite simple, the first step is registering each UUID characteristic, and then when there is data received it will be displayed in the application. You can follow the image below to read data from a multi-characteristic UUID. (source)

Now you can read the data sent from Nicla Sense ME in your smartphone without using the internet, but using BLE.

6. Transmit Data from Android to XIAOESP32C3

The next stage is the process of making the Sajac Transmitter. Sajac Transmitter will be made using XIAOESP32C3. Seeed Studio XIAO ESP32C3 has equipped a highly-integrated ESP32-C3 chip, built around a 32-bit RISC-V chip processor with a four-stage pipeline that operates at up to 160 MHz. The board equips highly-integrated ESP32-C3 SoC. The chip has been installed with a complete 2.4GHz Wi-Fi subsystem which means it supports Station mode, SoftAP mode, SoftAP & Station mode, and promiscuous mode for multiple Wi-Fi applications. It works under an ultra-low power state, also supporting features of Bluetooth 5 and Bluetooth mesh.

I added an expansion board to the XIAOESP32C3 to make it easier to display data sent from the user's smartphone. The expansion board enables building prototypes and projects in an easy and quick way. With its rich peripherals, you could explore the infinite possibilities of Seeed Studio XIAO series. Circuit python is also well supported by this board.

The programming steps for the XIAOESP32C3 are almost the same as those for Nicla Sense ME. The difference is that if on Nicla Sense ME we receive data on our smartphone, on the XIAOESP32C3 we will send data from our smartphone. The data to be sent is in the form of the user's status in the cave.

There are three status options that can be selected by the user. The first status is "Safe", the user can choose this option if the results from the Sajac application show a safe index and the user can proceed to the next checkpoint. The second status is "Bad", the user can select this option if the results of the Sajac application show a dangerous index, but the user can return on his own. The third status is "SOS", the user can choose this option if the results of the Sajac application show a dangerous index, and the user needs help from guards to return.

Let's start by programming the XIAOESP32C3 as a BLE center. We need to make initial settings on the Arduino IDE so that it can be used to program the XIAOESP32C3. You can view the complete documentation on the official Seeed Studio website. After you complete the initial settings, then we will enter the programming section.

If you already understand basic programming on the XIAOESP32C3, then you can then try the basic program to receive data sent by other devices found on this link or in the project attachment. You can carry out simple communication using the LightBlue application as in the previous experiment. Good luck!

If you can already communicate using the LightBlue app, let's continue with the code in the MIT App Inventor to send commands from a smartphone. In general, the code to be generated is still the same as on the Sajac Sensor page, the code to be generated includes scan, connect, disconnect, help, and home functions. The difference lies in the code that will be used if the Sajac Application is connected to the XIAOESP32C3 as the Sajac Transmitter. We're going to create three buttons to send the conditions from the user in Create as we planned above.

Three buttons are selected following the status of Nicla Sense ME and user conditions.

• Safe : If Nicla Sense ME indicates that the environment is in good condition and the user is ready to proceed to the next post.
• Bad : If Nicla Sense ME indicates that the surroundings are in bad condition, then the user will postpone the expedition and return immediately on their own.
• Danger/SOS : If Nicla Sense ME indicates that the surroundings are in bad condition, but the user cannot return and needs assistance from Guards who are on standby.

OK, you've done your best.

Next, we will modify the display on the OLED on the XIAOESP32C3 to display different text if the input given is different. You can see how to display text in the documentation on the official Seeed Studio website or you can try the simple code that I provide in this link and project attachment.

We need to add the function as shown above in the XIAOESP32C3 program. There are three different functions to be used. These three functions will display different results on OLED. For now we can just program here, but we will modify the program again in the next step. The results of our program can be seen in the image below.

7. Send Notification from XIAOESP32C3 to Wio Terminal

We can already send data from the smartphone to the transmitter. In this section we will transmit data from the Sajac Transmitter to the Sajac Guards. Sajac Guards will use the Wio Terminal. Wio Terminal is an ATSAMD51-based microcontroller with both Bluetooth and Wi-Fi Wireless connectivity powered by Realtek RTL8720DN, compatible with Arduino and MicroPython. Currently, wireless connectivity is only supported by Arduino. More than just a single embedded functional module, Wio Terminal itself is integrated with a 2.4” LCD Screen, onboard IMU(LIS3DHTR), microphone, buzzer, microSD card slot, light sensor, infrared emitter(IR 940nm).On top of that, it also has two multifunctional Grove ports for Grove Ecosystem and 40 Raspberry pi compatible pin GPIO for more add-ons. All of these practical add-ons are carefully housed in a compact enclosure, making it an efficient and product-ready product. Equipped with Raspberry Pi 40-pin GPIO, it can be mounted to a Raspberry Pi as a slave device.

Based on tri's project, Seeed Studio Grove LoRa-E5 uses ATTX serial communication to communicate with each other. This can be used to send sensor readings by XIAO Series to Wio Terminal. This data will be received by Wio Terminal and will be displayed on the tft screen.

• For XIAOESP32C3

#include <Arduino.h>
#include <SoftwareSerial.h>
#include <Wire.h>
SoftwareSerial e5(1, 2); 
int data1=0;
int data2=0;
int data3=0; 
int data4=0; 

static char recv_buf[512];
static bool is_exist = false;

static int at_send_check_response(char *p_ack, int timeout_ms, char *p_cmd, ...)
{
    int ch = 0;
    int index = 0;
    int startMillis = 0;
    va_list args;
    memset(recv_buf, 0, sizeof(recv_buf));
    va_start(args, p_cmd);
    e5.printf(p_cmd, args);
    Serial.printf(p_cmd, args);
    va_end(args);
    delay(200);
    startMillis = millis();
 
    if (p_ack == NULL)
    {
        return 0;
    }
 
    do
    {
        while (e5.available() > 0)
        {
            ch = e5.read();
            recv_buf[index++] = ch;
            Serial.print((char)ch);
            delay(2);
        }
 
        if (strstr(recv_buf, p_ack) != NULL)
        {
            return 1;
        }
 
    } while (millis() - startMillis < timeout_ms);
    return 0;
}

static int node_send(uint32_t timeout)
{
    static uint16_t count = 0;
    int ret = 0;
    char data[32];
    char cmd[128];
    
    
    
    memset(data, 0, sizeof(data));
    sprintf(data, "%04X,%04X,%04X,%04X", data1, data2, data3,data4);
    sprintf(cmd, "AT+TEST=TXLRPKT,\"5345454544%s\"\r\n", data);
    ret = at_send_check_response("TX DONE", 2000, cmd);
    if (ret == 1)
    {
 
        //send Airquality
        Serial.print("Sent successfully!\r\n");
        
    }
    else
    {
        Serial.print("Send failed!\r\n");
    }
    data1++;
    data2=data2+5;
    data3=data3+10;
    data4=data4+20;
    return ret;
}
 

void setup(void)
{
 

    
    
    Serial.begin(115200);
    // while (!Serial);
 
    e5.begin(9600);
    
    uint16_t error;
    char errorMessage[256];

    

    
    
    Serial.print("ping pong communication!\r\n");
 
 
    if (at_send_check_response("+AT: OK", 100, "AT\r\n"))
    {
        is_exist = true;
        at_send_check_response("+MODE: TEST", 1000, "AT+MODE=TEST\r\n");
        at_send_check_response("+TEST: RFCFG", 1000, "AT+TEST=RFCFG,866,SF12,125,12,15,14,ON,OFF,OFF\r\n");
        delay(200);

    }
    else
    {
        is_exist = false;
        Serial.print("No E5 module found.\r\n");

    }
}
 
void loop(void)
{
    if (is_exist)
    {

        //node_send_then_recv(2000);
        node_send(2000);
        delay(3000);
        
    }
}

There are some basic changes that must be made. XIAOESP32C3 has no compatibility with the Software Serial library. We can replace it by using Serial Hardware as stated in the documentation on the Seeed Website.

Finally we will get a program that is able to send flag to the Wio Terminal using LoRa. The flag to be used is an integer number. Number 1 will represent a safe condition, number 2 will represent a Bad condition, and number 3 will represent an SOS condition. We will also send some data such as user ID and checkpoint ID. You can check the full code on this link or in the attachment.

8. Show notifications to the Guards using Wio Terminal

Okay, now we will go to the last part of the system that we will create before we create a machine learning model using Edge Impulse. In this section we will create a system that will be used by guards outside the cave to find out the condition of users inside the cave. We will call this system Sajac Guards. Sajac Guards will use the Wio Terminal from Seeed Studio.

The Wio Terminal is a SAMD51-based microcontroller with Wireless Connectivity powered by Realtek RTL8720DN that’s compatible with Arduino and MicroPython. Currently, wireless connectivity is only supported by Arduino. It runs at 120MHz (Boost up to 200MHz), 4MB External Flash and 192KB RAM. It supports both Bluetooth and Wi-Fi providing backbone for IoT projects. The Wio Terminal itself is equipped with a 2.4” LCD Screen, onboard IMU(LIS3DHTR), Microphone, Buzzer, microSD card slot, Light sensor, and Infrared Emitter(IR 940nm).

Wio Terminal will be used to read data sent by XIAOESP32C3 as a transmitter in the cave, and will notify guards. The data that will be received is in the form of user ID, Checkpoint ID where the user is located, and the status chosen by the user. You can try it using the code on this link or in the attachment

The data sent will give a different response to the Wio Terminal. If the user selects "Safe" the status data will be displayed in green, if the user selects "Bad" then the status will be displayed in orange, and if the user selects "SOS" the status will be displayed in red and accompanied by a sound from the buzzer as a notification. You can see in the illustration in the image below.

Important note: you can download the Wio Terminal bracket in my previous project documentation at this link. You can print it using a 3d printer.

9. Collect Data

All systems are ready to use with proper. It's time for us to get into making machine learning models. This process will begin with collecting data from the sensor. The data will be divided into two types, the first is data for the safe category and the second is for the bad category. The first category data is taken with proper environmental conditions for exploration. The second category data is taken with unfavorable environmental conditions, for example low temperature, high humidity, low air pressure, and the presence of gases that can interfere with the respiratory system.

First we have to prepare some applications that need to be installed to retrieve data. Data will be retrieved by data forwarder method. Based on the documentation from edge impulse, we need to install the following applications:

• Node.js v12 or higher
• Arduino IDE
• The Edge Impulse CLI. Install by opening command prompt or terminal and run (windows):

npm install -g edge-impulse-cli

There are some notes that must be considered in the installation process:

• install the addition tools when installing NodeJS (not selected by default).
• The Arduino IDE version must be at least 1.15.
• more details on this link. (link)

After all applications are installed, the next process is to add a program for the Serial monitor which will later be used to send data using a data forwarder. The code for sending data to the edge impulse with the data forwarder is quite simple, we can use the serial print command on the sensor data. You can download the code at this link or in the attachment.

if (millis() - lastCheck >= 200) {
    lastCheck = millis();
    Serial.print(int(temperature.value()));
    Serial.print(",");
    Serial.print(int(humidity.value()));
    Serial.print(",");
    Serial.print(int(pressure.value()));
    Serial.print(",");
    Serial.print(int(gas.value()));
    Serial.print(",");
    Serial.println(int(bsec.iaq()));

The next step is to create a new project on the edge impulse platform. Once you have created a new project, open the node.js command prompt and open the edge impulse data forwarder using the command:

edge-impulse-data-forwarder

Fill in the data according to the request, and VOILA! Your device is connected to Edge Impulse and is ready to retrieve data.

Due to unfavourable circumstances and conditions for the data collection process directly in caves in Indonesia (due to the rainy season Dec 2022 until Mar 2023, and prone to landslides), the data collection process will be carried out using supporting media as closely as possible to the actual conditions.

This project will use several different types of conditions for the data collection process. The first category is the "safe" category, data in this category will be taken under normal conditions. The second category is "bad", data in this category will be taken in unfavourable environmental conditions, for example, cold temperatures, high humidity, low air pressure, and the presence of gases that can interfere with breathing.

10. Data Training using Edge Impulse

The first process of creating a machine learning model is Impulse Design. This section begins with creating an impulse that will be used. We must select the processing block and learning block to be used.

In this project, the flatten processing block was chosen. Flatten block will take several features in the form of statistical parameters such as Average, Min, Max, and other functions of raw data within time window. In this project four features were selected to simplify the process.

In the learning block section, the NN Classifier is selected to carry out the classification process. Neural networks, also known as artificial neural networks (ANNs) or simulated neural networks (SNNs), are a subset of machine learning and are at the heart of deep learning algorithms. Their name and structure are inspired by the human brain, mimicking the way that biological neurons signal to one another. This process aims to create a machine learning model from features that have been extracted from raw data in the previous process.

Edge Impulse features a user-friendly modeling process, where we can create models without the need to write code in a particular programming language. We simply set the input layer that we will use. Next, we simply press the "Start training" button and our machine learning model will be generated automatically.

11. Model Implementation

Edge impulse is equipped with features to convert machine learning models for various devices and programming languages. On the Deployment tab, you can select the type of device or programming language to be used. In this project, the machine learning model will be implemented using the Arduino IDE.

The zip file will be downloaded on your PC, you now just need to load it into the Arduino IDE by selecting Sketch > Include library > add.ZIP library. Then you can use the library that contains machine learning models. Let's try it first to make sure that our machine learning model works. Open the example file in your library on File > Examples > Sajac_inferencing > static_buffer > static_buffer. Enter the total number of features to be used, in this project there are 25. Then enter the following code into the raw_feature_get_data function. You can download the code used at this link or in the attachment. Upload the code into Nicla Sense ME and this is what we will get.

So, we just need to make a few changes in the code we are going to use and combine it with the code we created in step 5 to replace the status parameter that was previously dummy. You can download the code at this link or in the attachment. Let's see the final result. And....

So, I found this discussion and I download and run the code using platform.io

And VOILA!

last but not least, Perfect!

12. Testing the Device

Closing

After all, by using SAJAC, the users can find out how the condition around himself is. If it is not good enough, the users could somehow cancel or postpore his exploration. Even if he could not get out of the cave by himself, he can ask for help through SAJAC to then can be rescued by the guards.

The developers hope that Sajac can be an option to help cavers reduce the risk of hazards that can arise at any time. In the future, the developer hopes that Sajac will develop with more enhanced features and a better system. There are many things that can be explored to make users safer, for example, to detect user activity (running, walking, not moving) which will help Guards to monitor the user's condition directly.

Of course, the Sajac Project is far from perfect, and many shortcomings need to be corrected before it is ready to be explored further. Therefore, the developer hopes for a lot of suggestions and support from other developers to improve this project.

We are free to explore whatever we want, whether food, music, or hobbies, but we still need to pay attention to our safety. For all people who have a hobby of cave exploration, please use equipment that complies with existing standards because "Safety is number one".

SEE YOU ON THE NEXT PROJECT! :)