Automatic fall detection for elderly people

Introduction

Human falls are a global public health issue resulting in over 37.3 million severe injuries and 646, 000 deaths yearly. Specifically in India, there are thousands of old age homes and very few caretakers who can handle the ever-increasing number of elderly people living in these homes. The elderly people are always at the risk of sudden health issues like increased blood pressure, heart attack or stroke, or paralysis stroke leading to body instability and hazardous falls. In such a situation, if the elderly people are unattended, it can lead to adverse consequences.

To avoid such incidents, quick responses (instantaneous alerting system) are important when a fall happens. The earlier the fall is reported, the higher the probability of saving the patient. There is a need for an intelligent surveillance system, that is capable of accurately detecting falls and immediately sending notifications. This system can significantly reduce the rescue time and fatal damage can be minimized.

The existing approaches mostly focus on wearable IoT based non-vision sensors (like accelerometers). Generally, these sensors are uncomfortable for elderly people and not that accurate. Therefore, we propose to develop a computer vision-based solution that is less intrusive and collects information from cameras.

Smart Video Analytics - Fall Detection

Nowadays, cameras are included in most home assistance/care systems as they are less intrusive and better than sensor-based systems. Cameras can be used to detect multiple actions simultaneously with less intrusion. In this project, we have developed a smart fall detection system that can be easily integrated with PoE/USB cameras. The smart fall detection system can detect and send alerts to an emergency person, whenever an elderly person falls down.

To automate the process of fall detection, we use AI-based models to extract various intermediate features/states from the RGB camera images. These images are fed to an AI model that can detect various actions such as stand, sit, fall, and transition. These codes run at the edge on the standalone Xilinx ZCU104 FPGA board (Fig 1).

Fall Detection Model

To detect the various states of motion - stand, fall, transition, and sit we used a YOLOv3 based model. XilinxVitis-AI consists of several AI models, but none of the models can be used for fall detection. Therefore, we re-trained the YOLOv3 network on a custom fall detection dataset.

Dataset

There are few publically available vision-based fall detection datasets. We used a multiple camera fall dataset by Université de Montréal. It covers a room from multiple angles and several falling positions at the same time. To create a robust model that can detect various states - fall, stand, and sit, we collected approximately 20, 000 images from this dataset and labeled them using LabelImg software. An example of the labeling is shown in Fig. 2.

Training to Compilation

We obtained the Darknet model and pre-trained weights from theYOLOv3 model (Darknet YOLOv3 GitHub) and re-trained it on the custom fall detection dataset. We further used Xilinx Quantizer and XilinxCompiler to quantize the model, and generate the compiled executable files (.elf) that can be deployed on the Xilinx ZCU104 FPGA. The software deployment flow from training to deploying it on the Xilinx ZCU104 FPGA is shown in Fig. 3.

Live Fall Detection for Elderly Homes

Once the algorithms were ready, we integrated the software and hardwaretogether(shown in Fig. 4). We took input from a live USB camera and connected it to the Xilinx ZCU104 FPGA board.

The USB camera records video in the H.265 format, which is fed as input to the Video Codec Unit (VCU)in-built inside the ZU7EVFPGA. After the VCU processes the images, they are processed by the YOLOv3 fall detection algorithm using the Vitis AI APIs. If a fall is detected, an emergency alert SMS is sent to the relative. We used the FAST2SMS API to send messages to the emergencycontact person. A live demo of the fall detection is shown below.

Fall detection on multiple camera video feed

In most old age homes, there are fewcaretakers and more elderly people to monitor them. A single person cannot monitor multiple cameras at the same time. The Xilinx ZCU104 FPGA is capable of processingmultiple camera feeds simultaneously at the edge(shown in Fig. 5).

Therefore, we can deploy multiple cameras and integrate it with the automatic fall detection algorithm running on the standalone Xilinx ZCU104 FPGA. The algorithm will constantly monitor the video feeds simultaneously, detect whether a person is standing, falling, or sitting and send emergency SMS alerts whenever somebody falls down. Since we did not have multiple cameras, we tested the algorithms on multiplevideos stored on the SD card. We modified the Vitis-AI APIs to read and process multiple camera inputssimultaneously. A demo video of the multi-camera fall detection is shown below.

Conclusion

For the more serious cases, a swift response to a fall can mean the difference between life and death. Therefore, our automatic fall detection solution deployed on the standalone Xilinx ZCU104 FPGA can save several lives.

Future scope

The automatic fall detection algorithm can be extended to many other scenarios to provide assistance. Some other applications include:

• Kids in the playground
• At homes, where people who suffer from medical conditions like heart attacks or stroke
• At hospitals, where patients need to be frequently monitored