Face Detection using Conexio Stratus & Useful Person Sensor

Face recognition technology is important for the internet of things (IoT) because it allows devices to identify and authenticate individuals based on their facial features. This can have a variety of applications, including security and access control, personalization of devices, and even enhancing the user experience. For instance, a smart home security system could use face recognition to grant access to authorized individuals, while a smart device could use the technology to recognize and greet the user by name. In addition, face recognition can be used to improve the accuracy of biometric authentication methods, such as unlocking a phone or accessing sensitive information. Overall, face recognition technology has the potential to improve the security and functionality of IoT devices, making them more convenient and user-friendly.

This tutorial is a stepping stone toward how one can use IoT and machine learning to realize one of the applications mentioned above. For this tutorial, we have specifically chosen the Useful Sensors person sensor. The Person Sensor has built-in facial recognition and determines how many people there are, as well as their relative position.

Overall, this tutorial demonstrates how to:

• Integrate Useful Sensors person sensor to the Conexio Stratus dev kit.
• Port drivers for the person sensor for nRF Connect SDK + Zephyr RTOS.
• How to use I2C driver API in Zephyr RTOS.
• Detect and output faces recognized by the person sensor to the terminal.
• Outlines how to send person detection data to the cloud via cellular connectivity of Stratus dev kit.

Why Cellular And Why the Conexio Stratus kit?

Wi-Fi service might not be the most reliable and even not an option in some places for users. For instance, if you want to monitor and deploy a sensor device in the middle of the farm, where there is no WiFi, but you have good cellular coverage. This is where cellular connectivity comes in handy and which is why I decided to go with the cellular-based Conexio Stratus dev kit. Although there are many other IoT platforms that are cellular-based, however, following are some of the reasons why Conexio Stratus stands out from the crowd.

• The Conexio Stratus dev kit allows us to easily prototype an IoT solution as it comes with a prepaid 500MB of data with 10 years of global cellular service. No contract is required.
• The dev kit supports LTE-M and NB-IoT protocols and has integrated GPS as well, eliminating the need to buy and integrate an external GPS module.
• It also has onboard environmental sensors, specifically, Sensirion Sht4x temperature and humidity sensor and LIS2DH accelerometer from ST micro.
• Last but not least, the kit supports energy harvesting and battery recharging from solar. This is one of the missing features that is currently not found in many IoT devices.

In addition to the Stratus device, I also used a Stratus shield which supports Sparkful QWIIC connector, Mikro bus for Mikroe click boards, and Grove I2C connector for connecting external sensors and actuators. Using this shield, we have connected the Useful person sensor to the Stratus kit. This is mainly to demonstrate how easy it is to interface and connect a third-party sensor module to the Stratus device without soldering. It can’t get easier than this 🙂.

No external MCU is Required. Isn’t that Cool!

Unlike other cellular-based IoT devices where you need to have a dedicated MCU to control other peripherals such as GPS, external sensors, etc, Conexio Stratus eliminates the need for an external MCU. It simply combines the main MCU, cell, and GPS module in a tiny form factor enabled by the nRF9160 SiP. nRF9160 incorporates an Arm Cortex-M33 application processor solely for applications, a full LTE modem, RF Front End (RFFE), and a power management system, making it the most compact, complete, and energy-efficient cellular IoT solution on the market. Moreover, eliminating the need for an external MCU, cellular, or GPS module, cuts down the device integration cost and time.

So let’s dive into setting up the required software and hardware configurations.

Software Integration

Note: This tutorial assumes that one has already installed the nRF Connect SDK v2.1.1 and the prerequisites for the Conexio Stratus device. See our previous posts as to how to complete those.

At the time of writing this tutorial, there was no Useful Sensor I2C driver implementation for Zephyr RTOS, therefore, we have ported one from the sample Arduino driver provided in the Useful Sensor git repo. The application sample shows how to use the I2C driver API in ZephyrRTOS to fetch face data from a useful person sensor.

This sample application performs a few different tasks:

• Setups up and initializes the Person sensor.
• Performs a read action on the I2C address of the sensor and fetches the current face detection information.
• Display this data on the terminal window.

Here is the full sample application which can be found in the conexio-stratus-firmware repo on GitHub.

/*--------------------------------------------------------------------------*/
/*
 * Copyright (c) 2022 Conexio Technologies, Inc
 *
 * SPDX-License-Identifier: Apache-2.0
 */   
/*--------------------------------------------------------------------------*/
#include <zephyr.h>
#include <stdio.h>
#include <zephyr/device.h>
#include <zephyr/drivers/i2c.h>
#include <zephyr/drivers/sensor.h>
#include "person_sensor.h"
#include <logging/log.h>
/*--------------------------------------------------------------------------*/
/* Register the logging module */
LOG_MODULE_REGISTER(app, CONFIG_USEFULSENSORS_LOG_LEVEL);
/*--------------------------------------------------------------------------*/
const struct device *i2c_device;
/*--------------------------------------------------------------------------*/
static int rc;
static int i;
/*--------------------------------------------------------------------------*/
// How long to wait between sensor sampling. The sensor can be read as
// frequently as you like, but the results only change at about 5FPS, so
// waiting for >200ms is reasonable. Here, we are waiting for 1 sec. 
#define SAMPLING_INTERVAL_MS   (K_MSEC(1000))
/*--------------------------------------------------------------------------*/
/*
 * Brief : Fetch the latest results from the sensor
 * Param ; Pointer to the results
 * Return : false if the read didn't succeed.
 */
bool person_sensor_read(person_sensor_results_t* results) {
    rc = i2c_read(i2c_device, (uint8_t*)(results), 
          sizeof(person_sensor_results_t), PERSON_SENSOR_I2C_ADDRESS);
    if (rc < 0) {
         LOG_ERR("i2c_read error: %d", rc);
         return false;
	} 
    else {
        return true;
    }
}
/*--------------------------------------------------------------------------*/
/*
 * Brief : Writes the value to the sensor register
 * Param1 : Register address
 * Param2 : values to be written
 * Return : void
 */
void person_sensor_write_reg(uint8_t reg, uint8_t value) {
    uint8_t write_bytes[2] = {reg, value};

    rc = i2c_write(i2c_device, write_bytes, sizeof(write_bytes), 
         PERSON_SENSOR_I2C_ADDRESS);
    if (rc < 0) {
            LOG_ERR("i2c_write error: %d", rc);
	}
}
/*--------------------------------------------------------------------------*/
/*
 * Brief : Initialize the I2C bus
 * Param : I2C device 
 * Return : -EINVAL if failed, else 0 if Successful
 */
static int i2c_init(void) {
    LOG_INF("Setting up i2c");

    i2c_device = DEVICE_DT_GET(DT_NODELABEL(i2c1));
    
    if (!device_is_ready(i2c_device)) {
		LOG_ERR("I2C: Device is not ready");
		return -EINVAL;
	}

    int ret = i2c_configure(i2c_device, I2C_SPEED_SET(I2C_SPEED_FAST));
    if (ret < 0 )
    {
        LOG_ERR("I2C: Configuration failed");
        return ret;
    }

    return 0;
}
/*--------------------------------------------------------------------------*/
void main(void) 
{
    LOG_INF("Usefulsensors person sensor sample started"); 
    person_sensor_results_t results = {};

    i2c_init();
    
    while (1)
    {
        // Perform a read action on the I2C address of the sensor to get the
        // current face information detected.
        if (!person_sensor_read(&results)) 
        {
            LOG_ERR("No person sensor results found on the i2c bus");
            k_sleep(SAMPLING_INTERVAL_MS);
            return;
        }

        LOG_INF("****************");
        LOG_INF("%d face(s) found", results.num_faces);

        for (i = 0; i < results.num_faces; ++i) 
        {
            const person_sensor_face_t* face = &results.faces[i];

            LOG_INF("Face #%d: %d confidence, (%d, %d), (%d, %d), %s",
            i, face->box_confidence, face->box_left, face->box_top,
            face->box_right, face->box_bottom, 
            face->is_facing ? "facing" : "not facing");
        }
        k_sleep(SAMPLING_INTERVAL_MS);
    }
}
/*--------------------------------------------------------------------------*/

Note: The code in the main loop() is less than 40 lines. Extra lines of code in the main app are for setting and configuring the Zephyr RTOS I2C interface for reading the data from the Useful Person sensor via QWIIC connector.

Compiling and Uploading the firmware to the Stratus Device

To compile the application, open a terminal window in the application directory and issue the following west command:

west build -b conexio_stratus_ns

Once the application is compiled successfully, connect the Stratus device to the USB port and put it into DFU mode.

Flash the compiled firmware using newtmgr:

newtmgr -c serial image upload build/zephyr/app_update.bin

Open up a serial console and reset the Stratus device. The following serial UART output will be displayed in the terminal.

*** Booting Zephyr OS build v2.6.99-ncs1  ***
[00:00:00.205,596] <inf> app: Usefulsensors person sensor sample started
[00:00:00.205,627] <inf> app: Setting up i2c
[00:00:00.205,627] <inf> app: I2C device OK
[00:00:03.209,808] <inf> app: ****************
[00:00:03.209,808] <inf> app: 1 face(s) found
[00:00:03.209,838] <inf> app: Face #0: 69 confidence, (100, 56), (126, 108), not facing
[00:00:05.211,944] <inf> app: ****************
[00:00:05.211,944] <inf> app: 1 face(s) found
[00:00:05.211,975] <inf> app: Face #0: 94 confidence, (49, 19), (101, 99), facing
[00:00:06.213,012] <inf> app: ****************
[00:00:06.213,043] <inf> app: 1 face(s) found
[00:00:06.213,043] <inf> app: Face #0: 69 confidence, (98, 58), (128, 106), facing

This is Just the Beginning…

This post demonstrates one of the many applications that you can create with the Conexio Stratus cellular IoT device.

• Build and deploy the full face detection machine-learning model right onto the Stratus device. To get started have a look at how to get started with Edge Impulse using Conexio Stratus.
• Send the face detection data to the cloud securely using the cellular connectivity feature of the Stratus device. Not to forget, it comes with a preloaded 500 MB of cellular data.
• Send and visualize your data easily on the cloud using Datacake. Get inspired by having a look at the Conexio Stratus to Datacake tutorial.

If you want to create and connect your own IoT applications without having to worry about cellular data or contracts, grab your own Conexio Stratus kit today.

Thank you and happy coding. 🙏