Baby Watcher 👶

Behind the scene

Young marriages find them juggling their duties of children in the middle of a never-sleeping world in the fast-paced, high-demanding city centre. Meet a Gen Z couple finding out what it takes to raise a baby in a city where daycare options seem to be targeted towards toddlers than babies.

It can be challenging to strike a balance between work and daycare, particularly when there is a financial crisis and parents are forced to take on second jobs to make ends meet. These are the times when babies are left to play or sleep in another room by themselves, which presents a problem for parents who want to be able to maintain a close eye on their children.

he fundamental novelty of Baby Watcher lies in its seamless connection with Matter, a smart-home platform that unites multiple devices.

Baby Watcher goes one step further and works in tandem with Edge Impulse, a cutting-edge technology that can identify and react to a baby's cries. Parents won't have to guess what their young children require any more. By deciphering and interpreting cries, Baby Watcher can provide valuable information about an animal's needs, such as crying, laughing.

The rationale for this solution

Due to the RAM and ROM specification of nRF5340 SoC, 1Mb of RAM and I try to combine Matter code and Edge impulse ML model into one, there will be lack of RAM and ROM for the SoC to fully functions. The ROM code shouldn’t store on external flash. Therefore, I asked @Marte and she recommend me to seperate the application

This is my old build summary when I try to combining too much in one application. Obviously, there was not enought for normal operations

Memory region         Used Size  Region Size  %age Used
           FLASH:      920391 B     962048 B     95.67%
             RAM:      439824 B       440 KB     99.96%
        IDT_LIST:          0 GB         2 KB      0.00%

This is the system architecture

• I'll use a digital microphone connected to an nRF7002 DK as a Matter node to capture audio data. This data is then sent via UART to another nRF7002 DK serving as an Edge Impulse processor. This node will make predictions based on this data and output the results on the terminal.
• A Raspberry Pi 3 and an nRF52840 dongle will serve as a Thread Border Router. In the context of developing Matter over Thread, this router plays a crucial role in forming networks. Additionally, the Thread Border Router is responsible for forwarding IP frames between Wi-Fi and Thread networks.

Hardware connections

Explain hardware connections

Step 1: Developing the Edge Impulse model

First of all, we need to create a project on Edge Impulse Studio. It should look like this

Data Acquisition

To get started with the application, first we need to capture the data. There are 3 way to do this:

• Customize the forwarder application to use the DMIC module
• Add existing data
• Use the PC’s microphone

The former option could produce a more robust machine learning model, while the latter is much easier for beginners. I chose to go with the latter so that I could spend more time learning Zephyr for further development.

Regarding the data, I labeled it with three categories: laugh, noise, and crying, and split it into a 75/25 ratio."

Model training

In the time series data, you could configure the window size, the window increase and the frequency

In MFE section, you could explore the features

In the NN classifier, you could configure the neural network architecture

Deployment

Step 2: Developing the firmware for the Matter node

Create a new application based on Matter Template

• Create a new application based on Matter Template

Enable the UART and DMIC drivers in prj.conf

/* DMIC */
CONFIG_AUDIO=y
CONFIG_AUDIO_DMIC=y
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_HEAP_MEM_POOL_SIZE=8192
CONFIG_MAIN_STACK_SIZE=2048
/* UART */
CONFIG_SERIAL=y
CONFIG_UART_ASYNC_API=y

• Enable the UART and DMIC drivers in prj.conf

/* DMIC */
CONFIG_AUDIO=y
CONFIG_AUDIO_DMIC=y
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_HEAP_MEM_POOL_SIZE=8192
CONFIG_MAIN_STACK_SIZE=2048
/* UART */
CONFIG_SERIAL=y
CONFIG_UART_ASYNC_API=y

• Include the header of the UART and DMIC drivers

#include <zephyr/drivers/uart.h>
#include <zephyr/audio/dmic.h>

• Include the header of the UART and DMIC drivers

#include <zephyr/drivers/uart.h>
#include <zephyr/audio/dmic.h>

• Define devicetree overlay for this application In my case, the DMIC sensor will connect with the nRF7002 DK through P1.5 for PDM clock and P1.6 for PDM data input. Besides, I’d like to use UART1, which related to I2C1, so I have to disable I2C1 and gpio_fwd

#include <zephyr/dt-bindings/ipc_service/static_vrings.h>
/delete-node/ &gpio_fwd; /* for UART1 usage */
/ {
	chosen {
		nordic,pm-ext-flash = &mx25r64;
		zephyr,shell-uart = &uart1;
	};
};

&ipc0 {
    zephyr,priority = <0 PRIO_COOP>;
};

&clock {
    hfclkaudio-frequency = <12288000>;
};
&pinctrl {
    pdm0_default_alt: pdm0_default_alt {
        group1 {
            psels = <NRF_PSEL(PDM_CLK, 1, 5)>,
                <NRF_PSEL(PDM_DIN, 1, 6)>;
        };
    };
};
dmic_dev: &pdm0 {
    status = "okay";
    pinctrl-0 = <&pdm0_default_alt>;
    pinctrl-names = "default";
    clock-source = "ACLK";
};

&uart1 {
    status = "okay";
};

&i2c1 {
	status = "disabled";
};

• Develop UART functions to send data to the other node. Get the device pointer of the UART hardware and verify that it is ready

const struct device *uart= DEVICE_DT_GET(DT_NODELABEL(uart1));

• Get the device pointer of the UART hardware and verify that it is ready

const struct device *uart= DEVICE_DT_GET(DT_NODELABEL(uart1));

• Define an UART callback (More about UART callback function: https://academy.nordicsemi.com/courses/nrf-connect-sdk-fundamentals/lessons/lesson-4-serial-communication-uart/topic/uart-driver/) and register it in the main loop

static void uart_cb(const struct device *dev, struct uart_event *evt, void *user_data)
{
	switch (evt->type) {
	case UART_TX_DONE: /* Transfer data done */
		LOG_INF("Transmit data successful\\n");
	case UART_TX_ABORTED:
		LOG_INF("Transmit data failed\\n");
	default:
		break;
	}
}

• UART send data function

ret = uart_tx(uart, tx_buf, sizeof(tx_buf), SYS_FOREVER_US);
			if (ret) {
				return EXIT_FAILURE;
			}

• Develop DMIC functions. Get the device pointer of the DMIC hardware and verify that it is ready

const struct device *dmic_dev = DEVICE_DT_GET(DT_NODELABEL(dmic_dev));
	if (!device_is_ready(dmic_dev)) {
		LOG_ERR("DMIC device not ready\\n");
        return EXIT_FAILURE;
	}

• Get the device pointer of the DMIC hardware and verify that it is ready

const struct device *dmic_dev = DEVICE_DT_GET(DT_NODELABEL(dmic_dev));
	if (!device_is_ready(dmic_dev)) {
		LOG_ERR("DMIC device not ready\\n");
        return EXIT_FAILURE;
	}

• Configure the DMIC

/* DMIC configuration */
	struct pcm_stream_cfg stream = {
		.pcm_width = SAMPLE_BIT_WIDTH,
		.mem_slab  = &mem_slab,
	};
	
	struct dmic_cfg cfg = {
		.io = {
			.min_pdm_clk_freq = 1100000,
			.max_pdm_clk_freq = 3500000,
			.min_pdm_clk_dc   = 40,
			.max_pdm_clk_dc   = 60,
		},
		.streams = &stream,
		.channel = {
			.req_num_streams = 1,
		},
	};

	cfg.channel.req_num_chan = 1;
	cfg.channel.req_chan_map_lo = dmic_build_channel_map(0, 0, PDM_CHAN_LEFT);
	cfg.streams[0].pcm_rate = MAX_SAMPLE_RATE;
	cfg.streams[0].block_size = BLOCK_SIZE(cfg.streams[0].pcm_rate, cfg.channel.req_num_chan);

	ret = dmic_configure(dmic_dev, &cfg);
	if (ret) {
		LOG_ERR("dmic_configure failed: %d\\n", ret);
        return EXIT_FAILURE;
	}

• Trigger DMIC

static int do_pdm_transfer(const struct device *dmic_dev, struct dmic_cfg *cfg, size_t block_count) {
	int ret;

	LOG_INF("Starting sampling:");

	/* Start the microphone. */
	ret = dmic_trigger(dmic_dev, DMIC_TRIGGER_START);
	if (ret) {
		LOG_INF("dmic_trigger failed: %d\\n", ret);
		return ret;
	}
	
	for (int i = 0; i < 11; ++i) {
		void *buffer;
		uint32_t size;

		ret = dmic_read(dmic_dev, 0, &buffer, &size, READ_TIMEOUT);
		if (ret) {
			LOG_INF("dmic_read failed: %d\\n", ret);
			break;
		}

		/* Discard first readout due to microphone needing to 
		 * stabilize before valid data can be read. */
		if (i != 0) {
			int16_t tempInt;
			float tempFloat;
			for(int j = 0; j < 1600; j++) {
				memcpy(&tempInt, buffer + 2*j, 2);
				tempFloat = (float)tempInt;
				audio[(i-1)*1600+j] = tempFloat;
			}
		}
		k_mem_slab_free(&mem_slab, &buffer);
	}

	/* Stop the microphone. */
	ret = dmic_trigger(dmic_dev, DMIC_TRIGGER_STOP);
	if (ret) {
		return ret;
	}
	return 0;
}

Step 3: Developing the firmware for the Edge Impulse processor

• Create a new application based on Edge Impulse wrapper
• Develop the EI functions. Initialize the EI

err = ei_wrapper_init(result_ready_cb);
if (err) {
	printk("ei_wrapper_init error: %d\\n", err);
	return 0;
};

• Do the prediction

/* Add data */
ret = ei_wrapper_add_data(&audio_buf, ei_wrapper_get_window_size());
if (ret) {
	printk("Cannot provide input data (err: %d)\\n", ret);
	printk("Increase CONFIG_EI_WRAPPER_DATA_BUF_SIZE\\n");
}
/* Start predictio */
ei_wrapper_start_prediction(0,0);

Step 4: Setup Thread border router

Google has been made a great guide to set up border route: https://openthread.io/codelabs/openthread-border-router#0

Thanks

I am grateful to Nordic for their great hardware, that has made significant improvements to my projects. Additionally, I want to thank you for the Nordic Academy training, which has significantly enhanced my knowledge of your technology. Nordic Marte deserves special recognition and gratitude for her excellent support.