Intercom and smart controller mounted on wheelchair and roll

The goal

I wanted to offer a solution suitable for people using wheelchairs or rollators, to helps when there is someone ring on the doorbell.The portable receiver, who needs to be mounted on the helping device, should be able to receive the ring, receive the image from a camera mounted on the gate, and to communicate, bidirectionally, to the gate.Of course, it's expected to being able to open the electric gate.Since we are using some CPUs, why not adding the capability to mirror the notifications via Telegram to a private group?A caregiver can receive then notifications and or operate, via telegram, on the gate or other device available.

The project

As I have experience on ESP32, I used only these CPUs for this project, 5 in totals.

The project contains 3 components:

• Remote unit: is the external device for the doorbell. It includes a M5stack Atom + I2S Audio and an ESP32-CAM
• Main Unit: is the portable device, who should be mounted on the wheelchair or on the rollator. Include a M5Stack CoreS3, and Atom Echo and some buttons
• Base Unit. It's an Atom Switch from M5stack, a device particularly suitable for electrical panels.

I used two different devices for audio and video as it was easier, cheap and more reliable.

How it works

From guest perspective:

As I reach the door, there is the doorbell with a visible button. Pushing this button will wake up the Audio component, which will turn on the greenlight, and send the audio of the ringtone to the internal device, and a notification to the base unit. After the ringtone, the external audio will start streaming the audio captured from an i2s microphone for 30 seconds, and if no answer from the other side, the device will go in standby after that timer.If the person inside answer at the portable doorbell receiver, I can hear the answer from the speaker, and while the other side is sending audio, the light on the audio device will turn on blue. During all the time, from inside, the user can see who is awaiting on the door, as on the side of the ring button, there is a camera esp32-cam with wide angle sensor, and a torch, so also in the night there is a clear view of the door.

The electric door is controlled by the remote device, no need for the homeowner to reach some button on the entrance.

From Internal perspective.

If someone rings on the door, the M5Stack Core will turn on the video, displaying continuously the images from the outside camera, will play the ringtone, and the light on the audio will be turned on green. While the external audio will be played immediately, it's required to push a button to transmit the internal audio to outside

By pushing that button, the light on the audio device will turn blue while the unit will start transmitting. After releasing the button, the audio transmission will end immediately and the LED will turn green, to indicate we are listening.

On this phase, if we tap on the display, we send the command to the outside unit to turn on the torchlight on the camera. It's powerful and allow seeing the images from outside in the evening.

After 30 seconds from last executed action, both external and internal unit will go in standby, keeping only few components on.

If we decide to open the gate, we can push the red button and the base unit will turn on the relay for 500ms to activate the electric lock.

Anytime, from inside, we can enable the visualization of the external images by tapping the display, who will turn on immediately for 30 seconds. By tapping again, we will enable, as before, the torchlight.

It's also possible to enable the external audio even if no one pushed the external button, it's available function for that on pushin the blue button.

Emergency

Are available two emergency mode:

Mode 1: by pushing the dedicated button, the unit will notify the base unit to send this message to the configured Telegram group.

🚨 Help Requested🚨.Please call back

This notification can be repeated as many times we want.

Mode 2: If the remote unit detects unusual position, like rolling aside, who can accidentally happen using the wheelchair or the rollator, then an automatic emergency message will be triggered.

🚨 Emergency Request🚨.Automatic request from device.Please check if everything is ok

On both emergency modes, the display will show a message confirming the request has been sent.

Components

External Audio

It's based on M5stack Atom, a Max98357 amplifier and a digital microphone INMP441. These I2S devices are connected to the Atom as below:

#define CONFIG_I2S_BCK_PIN 19
#define CONFIG_I2S_LRCK_PIN 33
#define CONFIG_I2S_DATA_PIN 22
#define CONFIG_I2S_DATA_IN_PIN 23

I could have used an M5stack Atom Echo also outside, but the small internal speaker, and the 1W amplifier are not loud enough to be used outside.Beside the small speaker used on my example, the amplifier, with 3W or output power, can drive bigger and more efficient speakers. The amplifier can change the volume by changing the connection on the gain pin. The unit works in half-duplex, and can work either as listener or transmitter. I'm using UDP direct streaming for the audio, with a very low lag. I got inspiration from the great work on https://github.com/GrahamM/ESP32_I2S_Microphone,I added some timer for turning transmission off if no packets are coming from inside, and a watchdog to in standby the unit after 30 seconds. The unit can wake up in two modes:

• By pushing the front button, connected to pin 39, who act as ring request.
• If a signal high is received on the pin 25, connected to the esp32-cam

When waking up via pin 39, the unit will send automatically the ringtone.

wakeup_reason = esp_sleep_get_wakeup_cause();
  if (wakeup_reason == ESP_SLEEP_WAKEUP_EXT0) {
  notify();
  play_ring();
}

Note: the notify function, send an api call to the base unit, every time we push the ring button. Will explain later what notify mean.Funny note, the ringtone is stored on the Atom as PROGMEM, by storing the wav audio as hex values (16bit 44.1Khz)

unsigned const char sounddata_data[] = {
0x52, 0x49, 0x46, 0x46, 0x3E, 0x49, 0x02, 0x00, 0x57, 0x41, 0x56, 0x45, 0x66, 0x6D, 0x74, 0x20,
0x10, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x44, 0xAC, 0x00, 0x00, 0x88, 0x58, 0x01, 0x00,
.........
.........

As the Atom have a SK6812 LED connected on pin 27, I use that to show when the unit is transmitting (green), receiving (blue), having issues on WiFi (red) or in standby (turned off)

Code on GitHub

Camera

The external camera is based on esp32-cam, replacing the default module os2640 with a 160 degree lens. Initially I got a Unit Cam Wi-Fi Camera DIY Kit (OV2640) for the scope, as I like his design and the capability to change the camera according to the need, but, not having PSRam and USB port, it slowed down the development process. I changed the solution for the standard cam, who offers also some spare pin and the powerful white LED for night usage.The pin 13 is connected to the Atom device, acting as wakeup source when required.The software is quite straightforward. It's a weberver, based on a project I found on GitHub (user bnbe-club), not available any more on GitHub. I opted for downloading JPEG images instead of getting a rtsp streaming, as It's easier to visualize on any embedded device and does not require too much time to establish the rtsp streaming. Also, I need anyway the JPEG images for Telegram. The frame rate achieved, around 5 frames per seconds, it's sufficient for the scope.On startup, the cam creates a http server and expose the following methods:

#ifdef ENABLE_WEBSERVER
  server.on("/", HTTP_GET, handle_jpg_stream);
  server.on("/jpg", HTTP_GET, handle_jpg);
  server.on("/ledon",HTTP_GET, ledon);
  server.on("/ledoff",HTTP_GET, ledoff);
  server.on("/audio",HTTP_GET, wake_mic);
  server.onNotFound(handleNotFound);
  server.begin();
#endif

/ (handle_jpg_stream): send continuously jpeg images using the header multipart/x-mixed-replace. Opening with a browser, we can see a smooth stream. It is not used by the remote video, but it can be used from a smartphone as a standard video streaming./jpg (handle_jpg): get one frame from the camera and send via http

void handle_jpg(void) {
  WiFiClient client = server.client();
  cam.run();
  if (!client.connected()) {
    return;
  }
  int img_size = cam.getSize();
  String response = "HTTP/1.1 200 OK\r\n";
  response += "Content-disposition: inline; filename=capture.jpg\r\n";
  response += "Content-Length: ";
  response += img_size;
  response += "\r\n";
  response += "Content-type: image/jpeg\r\n\r\n";
  server.sendContent(response);
  client.write((char *)cam.getfb(), cam.getSize());
}

As the camera can use multiple buffers, I opted for a single buffer to get always latest snapshot, even getting a single image. Testing with multiple buffer, although allow to increase the frame rates, deliver only the oldest frame from the list./ledon: turn on the torchlight connected on pin 4

/ledoff: turn off the torchlight

/audio: set high, for 500 ms, the pin 13, connected to the Atom on pin 25. As that pin is configured as wakeup source, this call will turn on the Audio streaming for 30 seconds.

This software support different model of camera, and different size. I opted for 320x240 to match the M5stack CoreS3 display.

config.xclk_freq_hz = 16500000;
config.pixel_format = PIXFORMAT_JPEG;   // JPEG on hardware
config.frame_size = FRAMESIZE_QVGA;     // 320x240
config.jpeg_quality = 10;               // High Quality
config.fb_count = 1;                    // Only one frame as buffer

Code on GitHub

Internal Audio

This unit is similar to the external Audio unit, with the main difference of listening on UDP by default. It will transmit only by pushing the button connected to pin 39.

It's connected to the Core S3 via grove port, to get power source and to notify the Core when we receive UDP packets from the remote unit.

This unit will never go on standby, as this unit needs to receive the calls from the outside unit. When there are packets coming, will turn on the green light, while pushing the button for enable transmission will turn the LED blue.

Code on GitHub

Internal main unit

This is the core of the internal unit, based on M5stack CoreS3. The main function is to visualize the images from the remote camera, send commands to the camera and to the base unit, and to call for emergency.

The code is quite simple, on the initialisation set the WiFi, the IMU, the touch screen and the Display, then turn off the LCD, awaiting for any local or remote command.The Touch is used to enable the video receiving, and then, tapping again, send the command to turnon the torchlight on the camera.The image reception is made by running, in loop, the command display.drawJpgUrl (see m5gfx doc) who download directly the jpeg from the camera. attached to the CoreS3 I put some buttons, attached to the PORT A and B define in this way:

#define BUTTON1 8
#define BUTTON2 9
#define EMERGENCY_BUTTON 1

Button1 and button2 are used to wake up the external audio and to open the door.Emergency_button, like the name suggest, call the emergency API on the base unit (see below).I decided to not use a menu on display controlled with the touch screen, because, targetting persons with different inabilities and eventually older people, a touch screen, not so big, was not a practical option for everyone.Eventually, if we need more functions, we can increase the number of buttons by using a port extender like PCA9554PW, who can support up to 8 buttons.Of course, if the touch screen is an option for the user, than a graphical user interface can help keeping the unit more compact.

The Port C on the Core3 is connected to the Atom Echo, from which get the wake up signal for the display.The IMU unit is always on and it's used to detect unusual positions of the unit, which is supposed to be connected in stable way to the wheelchair. An unusual position of the unit may reveal that something may have happened, triggering immediately an emergecy request on Telegram. When an emergency request is made, the display will show a message as confirmation.After 30 seconds of inactivity, the unit turnoff the display until new commands.

Code on GitHub

Base unit

This can be considered a standalone project.It's entirely based on M5Atom Switch, a unit specifically designed to be mounted on electric panels. It's running an Atom unit, contains 2 relays and an AC power supply.

The software have two main functionality:

Web server

It's the standard server from <WebServer.h>, offering the following methods

server.on("/", HTTP_GET, handle_help);
server.on("/rele1", HTTP_GET, handle_rele1);
server.on("/rele2", HTTP_GET, handle_rele2);
server.on("/emergency", HTTP_GET, handleemergency);
server.on("/auth_emergency", HTTP_GET, handleauthemergency);
server.on("/notify", HTTP_GET, handlenotification);

• / (handle_help): print the list of API available
• /rele1 and /rele2 (handle_rele1 and handle_rele2): turn on and off the respective relay for 500ms.
• /emergency, and /auth_emergencyare used to communicate with Telegram, sending different emergency messages.
• /notify is called by the external Audio unit when someone is pushing the ringbutton. The base unit will download one image from the camera and will send as attachment to Telegram

void handlenotification() {
  String message = "Server is running!\n\n";
  message += "URI: ";
  message += server.uri();
  message += "\nMethod: ";
  message += (server.method() == HTTP_GET) ? "GET" : "POST";
  message += "\nArguments: ";
  message += server.args();
  message += "\nResponse: 200";
  server.send(200, "text/plain", message);
  sendImg(CHATID);
  message = "Someone knocking at the door!\n";
  bot.sendMessage(CHATID, message, "HTML");
}

Telegram Bot

This is the second function of the base unit. It's a bidirectional bot, based on the library UniversalTelegramBot, but forked on this code as there were a bug working with latest release of esp32 libraries. Will send a message to the author, although this library is not updated from long time.

This BOT continuosly checks for any command sent on a specific chat, and push messages as well when it receive the specific commands.

To setup, it's required to create a BOT and to put the credentials on the config.h file.Also it's required to create telegram group and put the chatid on config.h as well

#define BOT_TOKEN "123456789:AaBbCcDdEeFfGgHhIiLlMmNnOoPpQq"
#define CHATID "-12345678"

We should then invite on this chat everyone we want to be notified and we autorize to execute remote commands from Telegram on the Atom Switch.

After the Bot is connected to Telegram, we can get the list of available commands by sending the command "/start"

if (text == "/start") {
  String welcome = "Welcome Remote Bot, " + from_name + ".\n";
  welcome += "These commands are available.\n\n";
  welcome += "/get_photo : getting a photo from the camera\n";
  welcome += "/open_gate :Open the main gate\n";
  welcome += "/light_on : Turn on external light\n";
  welcome += "/light_off : Turn off external light\n";
  bot.sendMessage(chat_id, welcome, "");
}

The command are self explained on the list. The light mentioned is not the torch light on the camera, but the relay 2, which I was thinking to use for an external main light.This Unit can be extended by adding i2c relays, connected to be Groove port on the Atom, or to the additonal port on the switch, allowing to control more device from remote. The bot can be used to read also informations from any sensor. On my example I can ask for a photo from the external camera, as example, if I want to have a look on the door if I'm not at home.

Of course, as anyone on our chat can send commands without providing any password, the security is delegated to not inviting on that chat non trusted people. Maybe the bot can be improved by configuring some list of people allowed to execute commands, but this is out of the scope for this project.

The base unit can be also connected to some pull cord switches, distribuited on the house, to send specific alarms if one get pulled.

Code on GitHub

Configuration

Every component requires a couple of configurations:WiFi credentials and IP of the peer devices. Every device write on the serial port the IP after connected. We need a first build to get the WiFi working and then we need to update the IP list with the values discovered.We can eventually use specific IPs by configuring properly the DHCP server on the router.This part can be improved by using some captive portal for the WiFi setting, and eventually some broadcast message for autodiscovering the Peers, removing in this way any build operation.

Reference

Full code on GitHub: https://github.com/ecasti-gmail-com/doorbell/tree/master

Youtube video of the utiilisation

Retrocamera for backward mevements of the wheelchair https://www.hackster.io/ecasti/esp32-i2c-camera-b1a347

Schematic of the remote unit