Contactless SPO2 detection using RGB imaging

Introduction:

During the COVID-19 pandemic, all of us must've come across pulse oximeters, which are contact-based measurement devices that are used to measure Heart Rate and SPO2 levels, and show PPG graphs. However, it became clear over time that these devices were -

• Unreliable, they started malfunctioning within a few weeks of being bought - mostly a result of using cheap sensors.
• Costly, a pulse oximeter costed around 50-100$ around March-August, while it can be made within a few dollars. They have now dropped to 1/10th of their MRP at most marketplaces since October.
• Contact based: All pulse oximeters are inherently contact based, hence they might spread infection.

I aimed at changing this. Thermal guns had made it possible to measure EBT (Elevated Body temperature) contactless (though they had their own share of problems, the shortness of that range being one of them). Since EBT could be caused by any normal fever of daily activity, it was a very unreliable screening method on its own. Many people resorted to just performing thermal screening due to unavailability of pulse oximeters and their high costs at that time. This resulted in a lot of false alarms as well as missed screenings. What I wanted to do was come up with something very simple that can be adopted by anyone, from a industrialist owning a workplace to the general masses. Hence I started studying and surfing to find any concepts on Contactless SPO2 measurement, to no avail... until I came across r-PPG.

remote-PhotoPlethysmography

r-PPG or remote PhotoPlethysmoGraphy is a technique used to contactlessly measure the SPO2 levels and heart rate of a person. It involves shining an IR emitter on the person’s finger and capturing the reflections on a receiver placed away from the subject. The variations in the reflections are then analyzed and the SPO2 levels and heart rate of the patient are estimated. However, this process is cumbersome as the source is required to be kept near the person’s finger, IR sources and receivers are costly and the range of this process is still not sufficient. I thought of applying a new technique, which measures the RGB variations of the patient’s facial skin while breathing, considering the sun as the source of light and a camera lens as the receiver. This method would significantly improve the practicality of this procedure. This means there would be no requirement for a source and anyone could use a normal camera in their smartphones or surveillance devices to screen themselves for SPO2 measurements and HR(Heart Rate).

I knew this was something very new, and I would not find much to work on. Yet, I found a few papers online that had tried a similar concept, like mine.

• Non-contact estimation of heart rate and oxygen saturation using ambient light
• Breathing Analysis Using Thermal and Depth Imaging Camera Video Records
• Detail-preserving pulse wave extraction from facial videos using consumer-level camera

And this link directed me towards an old project conducted by people at MIT to enhance the variations in skin colour due to breathing.

All these resources gave me a place to start. However, I still had to write all the code to implement these concepts. Also, while reading these papers I came across the fact that all of them had tested their methods only on normal people, not hypoxia patients. This would mean that it might not even work on such patients. But since I had no choice and this was the only contactless way of SPO2 measurement I found I went ahead with it. The next step was to see if a similar code existed on Github. Except for a few vague repos that looked like shoddy final year projects, I found nothing of importance. Time to code...

Code Development

The process was simple - well atleast to look at. You had to perform face detection, then face recognition (optional, but nice to have) and then skin segmentation. Once that was done, all you had to do is measure the variations in the Red and Green channels for SPO2 measurement and Blue Channel for HR measurement. This was supposed to be done using FFT and statistical mathematics, which was then fed to a simple formula mentioned in the papers. The constants of the equation were to be selected by trial and error I guess because directly using the constants they gave never worked for me. This was done across 250 frames (that was least I came up with without any significant loss in accuracy). This meant that the higher FPS my camera would give, the faster my measurement would be.

I tried to develop the code, and was able to develop a simple version after a lot headache - I used OpenCV and Dlib heavily for the image processing, SciPy for maths, numpy for storing the SPO2, HR, Face ID and timestamp. I used skin segmentation and face recognition libraries that come with OpenCV for preliminary development, since I just wanted to test the algorithm. It worked well on me, I mean the values predicted by my algorithm were within 2% of deviation from my contact based pulse oximeter. I tested it on around 10 people before fixing my coefficients.

Moving Forward...

I know had a working algorithm, and by working all I mean is it showed some numbers and did not throw any errors. But I had to be sure it actually worked. To do that, I had to test it on actual patients with low SPO2 levels. People suffering from Hypoxia. For that, I talked to a few doctors that I knew, explained them what I aimed at doing and how this project could benefit millions. Not only that, there was no issue or harm to the patient for data collection, I just needed a 20 second video of their face. Privacy issues could be addressed by keeping the patient identity anonymous and asking for their consent. After many rejections, I finally got my chance.

For the data collection, I used the See3Cam and a Jetson Nano, because it would be more convenient and less risky. I could not bear the thought of damaging the ZCU104 in sanitization or data collection process. I had implemented my algorithm on Ubuntu, so it was easy enough to migrate it to Jetson. Finally, the day came and here's me wearing PPE and all in the hospital ward...

My entire setup is pretty visible in the video, I had taken a lot of stuff including the ZCU104 case, to accommodate all of it. The hospital people were kind enough to provide me with a mobile platform, and I cannot thank them enough for their co-operation. Even more so for all the patients that volunteered for this data collection - Not even one of them turned me down, much to my surprise. In fact all of them were extremely happy and supportive of such advancement in technology. I took notes of their comorbidities, some details and took their videos. I was able to take 8 videos, all of them being corona patients out of which 3 had SPO2 levels below 95%

Validation Time!

By this time, I had 16 videos of normal people, 5 videos of corona patients with normal SPO2 levels and 4 videos of patients with SPO2 levels below 95%. This meant my batch size was 25, which is good enough for a POC and preliminary analysis. I ran my algorithm on each video I was able to collect, as shown below. The patient that I scanned in the video below had the lowest SPO2 levels I came across - around 86%.

My algorithm predicted a SPO2 level of around 87-89%. This patient's SPO2 levels were fluctuating while I took the video, and he had a lot of skin covered with masks, tubes and caps. This must have led to accuracy loss, because for all other volunteers I received a deviation of not more than 2% from the contact based SPO2 meter.

Now that I had validated that the algorithm worked on a basic level, it was time for FPGA implementation and Hardware acceleration!

Hardware Acceleration and ZCU104 implementation

A lot of time went in reaching this point, and it was already November by the time I performed my preliminary analysis. Hence, I was aiming to get things done fast, but ZCU104 is a very complex board, and Vitis AI a very new environment. Result - A lot of errors, debugging, surfing and whatnot. Finally, I decided to accelerate the Face recognition and skin segmentation part. I used the Face detection demo example for face detection, as that was the only requirement for SPO2 measurement, face recognition could be kept for later if time permitted. Finally, I installed all required Xilinx tools, purchased all required accessories and started working on the actual migration in December. As expected, I was short of time, hence I was only able to implement the face detection acceleration. Tools downloaded and used:

• Vitis 2020.1
• Vitis AI 1.2

I used petalinux and VART to run my model on the ZCU104. Details are provided in the attached repository. I followed the steps shared here and here to integrate and use the VCU and DPU with Vitis AI on the ZCU104. The DPU was used to hardware accelerate the Face Detection model using VART. The VCU was used to encode the recorded videos and save them in h.264 format.

My setup:

Model Explanation:

The model for densebox used for the face detection part was provided by xilinx themselves. The framework the model is made upon is caffe and the special caffe-xilinx library was needed to compile it for further steps. The model was trained upon the WIDER face dataset while the testing of the model was done using the FDDB dataset. The model parameters are as follows. The data channel is BGR thus 3. Padding was done to make the resolution of every image 360:640. Each image was then resized into 320*320.

Here is how the ZCU104 implementation looked like:

Future Scope and Updates

I was so glad that it worked the way it did! The VCU was used to encode the video in h.264 format. But there are a lot of things still left to do, like -

• Make my custom Vivado Design: I tried my best but I just couldn't finish my Vivado design, it has a lot of documentation and tweaking involved. But i am sure this exercise will significantly improve my system's performance.
• Explore the ZCU104 and Xilinx tools more: The resources provided by Xilinx and the capabilities offered by ZCU104 are endless.
• Implement 8 parallel streams from 8 different cameras and perform SPO2 and HR analysis on each of them. Encode all data in h.265 format. This will help my solution to be used in malls, workplaces, ICUs, etc where multiple people need to be monitored simultaneously. Details on how to use the demo are given on the official Xilinx forum here.
• App deployment of my algorithm for mass usage and scalability.
• Implement HR by analyzing the Blue channel. I didn't get time to perfect it because it is affected a lot more by lighting conditions than SPO2 estimation.
• Adding more ambitious vitals and graphs like PPG, ECG, Blood pressure, etc.

Nonetheless, this project is complete in itself and the repository attached contains all that is required to run it on ZCU104.

BOM

This system only used the ZCU104 kit which included -

• The ZCU104
• A 16GB SD Card
• A Targus USB hub 4 port
• A Cu30 See3Cam
• Adapter and Vivado License

This kit costs around 1300 - 1400$ online.

The following additional components were used:

• Jetson Nano B01: For data collection. 99$
• A contact based pulse oximeter as ground truth. 5$

Relevance to Intelligent Video Analytics

This project is extremely relevant to Intelligent Video Analytics. It offers the beginning of a unique screening and monitoring device that can be used in contactless measurement and monitoring of vital parameters for any person, not just an ICU patient. Since the system uses a simple RGB camera, anyone can afford such kind of software. Even industries will have no problem in adopting such kind of hardware-accelerated Advanced Diagnostics. This is because it is possible that just like HR and SPO2 vitals, many other lucrative parameters and graphs like ECG, PPG and blood pressure can be measured using the same way. There are a few papers exploring such possibilities and I am sure the day is not far when I will be able to develop a screening software and device that will be able to monitor all major vitals of any person without any contact or additional sensor interfacing required. I have already received immense interest in this concept from many authorities since I finished my preliminary analysis. I am looking forward to developing this concept to the fullest - I will soon be conducting an official study involving more than 100 patients to officially publish my research. This study is expected to take place from January next year.