Fall Detection

Project's Concept

This project objective is to monitor aging people who live alone, There are cameras ( in the future plan to place in many rooms) which will feed video to Xilinx ZCU104 to process data using Vitis-AI and Openpose to detect fall detection.

I had developed simple algorithm to detect falls by using the following criteria.

• The Angle of Head and body change more than 45 degrees
• The speed of changing angle more than specific value ( Need to trial and error at this moments)

There are two ways to notify the fall detection.

1) via IFTTT by sending SMS to predefined number.

2) Turn on Siren. This Siren is a simple Network Siren that uses NodeMCU and activated by sending http://<ip of Siren>/SirenOn This will allow us to put Siren anywhere which have Wifi Signal. I will not go into details for this project. Please look into https://invtos@bitbucket.org/eblock-gaia/eblock_alarm for details.

On ZCU104 board, we have run a simple Web Server, which will save the picture and record video from the start of fall.

Project Implementation

I spend 90% of the project time to create SD image which has following components.

• VCU
• DPU
• Vitis-AI with Openpose model
• Curl command for sending request to IFTTT and Network Siren.

However I failed to create complete image, there are a lot of errors in each stage of development. The best image we can test for Vitis-AI comes from ready build image for Vitis-AI https://www.xilinx.com/bin/public/openDownload?filename=xilinx-zcu104-dpu-v2020.1-v1.2.0.img.gz which doesn't have VCU but have all pre-install VItis-AI and model. For more details go to https://github.com/Xilinx/Vitis-AI how to create image and install it.

After prepare environment for Vitis-AI on ZCU104, download source code by using

git clone https://invtos@bitbucket.org/eblock-gaia/fallguard.git

Please beware that this image only supports Display Port Adapter !!. I spent almost two weeks figuring out how to show on a normal screen with HDMI port. (In the picture, I borrowed my friend's monitor which has Display Port for half-day to test)

- Ubuntu or Linux machine run command ssh with -Y option to ZCU104 board. -Y mean enable truster X11 forwarding. If you do not specify -Y option. You may get error "Permission Denied. Cannot Open Display" For example

$ssh -Y root@<ip of zcu104>

After login, use command echo $DISPLAY to get display environment. You can use this value for serial terminal console to display Windows on your Linux machine ( DISPLAY environment will be something like localhost:10.0)

- Windows machine. You need to install X server program. I use VcXsrv program. When start VcXsrv using Xlaunch, don't forget to disable access control (on last screen) to accept connections from all clients. At ZCU104 board, setting environment DISPLAY=<ip of windows machine>:0.0

To using input from USB Camera start program with

$./FallGuard openpose_pruned_0_3 0 -t 8

To using input from video file (At this moment only support.webm format)

$./FallGuard openpose_pruned_0_3 <videofile.webm> -t 8

To compile program on host machine, we need to set environment for Arm 64 bit. By using command $source <Directory that SDK installed with sysroots >/environment-setup-aarch64-xilinx-linux For example my SDK is at /opt/petalinux/2020.1/

build.sh is the script for compile program, after compile use scp to copy file to target ZCU104 board. Ex. scp FallGuard root@<ip of Zcu104>:~/ ~ mean copy to home directory.

There are two ways to notify the fall detection.

1) use IFTTT webhook to send SMS by post FallDetect event via webhook using curl command. Please see line 67 in process_result.hpp for details.

2) Post SirenOn value to Network Siren Address. (line 69) Siren Address is fixed (line 17) at this moment for ease of implementation. This value is fixed by setting router to fix ip address for Network Siren's mac address.

Download source code and implementation details from

git clone https://invtos@bitbucket.org/eblock-gaia/eblock_alarm.git

Use Arduino to compile and don't forget to define SSID and Password inside the program. We use only 5 V for power supply even the siren can support up to 12 V.

The last part of implementation is Web server. we use simple Web server (httpd) from busybox to implement simple Web server which show picture and video when fall detection occurred. This will start by call script startweb.sh in project directory or you can put this script inside.bashrc for automatic start.

The picture is in jpg format and video is in avi format. These operations are done by OpenCV library. Please note that the encoder (MJPG in this project) and container (.avi) and size of the image (2304, 1536) must match. If not the video will unreadable. I still confuse about these combinations. So I can't explain more. I tried with other combination Ex. encode = Mpeg and container = mp4 but it does not support. (This still open area to develop)

Project's results

We had tested Fall detection both on video files and USB camera. Both are working fine when person faces directly to camera. But if person stand beside or some Angle, the Openpose cannot determine gesture correctly. This need to improve by installing different angle camera or use other Algorithm to help classified the side gesture. Notification takes sometimes too, this cause from Network configuration and speed of Network. ( Sometimes fast and Sometimes slow)

The speed of falling needs to adjust and widely test depend on different scenarios. For example, if you intend to sleep ( slow movement), system should not fault alarm and think it's falling. Different algorithms for fall detection will help the accuracy of detection.

From video, you will see some lagging between motion and picture on screen and Alarm activate. This should be improved with VCU.

Wrap Up

This project still has more room to grow. DIfferent scenarios should be tested and the speed of Camera processing should be improved. (If we can use VCU, these problems will be solved) System Robustness will be very important. We don't want System hang when someone falls down and system stop working. This will put life at risk.