Fatigue & distraction alert for long distance driver

1. Background

The project aims to perform object detection inside the vehicle to monitor on the driver if exhibit symptom of fatigue, which could be identified by yawning, closing the eyes. Besides, it also target to detect object such as handphone and hand, if they are closing to each other.

2. Trial

Begin my journey with Xilinx FPGA following the link https://xilinx.github.io/kria-apps-docs/main/build/html/docs/smartcamera/docs/app_deployment.html. The smartcam application tutorial consists a topic using a different model from default model https://xilinx.github.io/kria-apps-docs/main/build/html/docs/smartcamera/docs/customize_ai_models.html. However when running this ssd-mobilenetv2 model, not only the classification is wrong but extremely low fps. I've been reading here and there looking for clue how to troubleshoot and rectify yet to find solution. The model is downloaded from models zoo and compiled in vitis-AI. I'd tried out other models from models zoo as well, not much difference.

Attempt to try on the sd-mobilenetv2 model based on tensorflow framework, following the same method to prepare the files in kv260, but it didn't run as plan, run into Segmentation Fault error.

One of the admin did come back to me offering guide to this issue. She suggested to troubleshoot the capture pipeline if has issue capture and stream to the output, I had verified by switching to file input and run with the default model, it's okay but I switch to the ssd-mobilenetv2 model, the fps slow down tremendously. She had also advise the sync option should be off, I'd check the pipeline que in jupyter notebook, it's already sync off.

Since all my attempts for solving the issue with smartcam didn't come to good end, I switched my focus to learn building custom app follow https://community.element14.com/technologies/fpga-group/b/blog/posts/kv260-vvas-sms-2021-1-blog. Sorting out the installation error this and that has already consumed quite a lot of time (since it's my experience with Vitis, Vivado & PetaLinux), when I finally get the tools needed and run the code building the platform follow https://community.element14.com/technologies/fpga-group/b/blog/posts/kv260-vvas-sms-2021-1-part2, the build has always come to error which I have to give up because this process is taking really long time to run (more than 12hours), yet never managed a single successful build (I was running it with VM in Ubuntu 18.04, 10GB RAM & 500TB).

The tutorial says extensible XSA should be generated first before building the platform, so follow the steps to make XSA but error too. It's the same error that abort platform building earlier.

Posting question in forum and screening through the xilinx support forum, found this thread. https://support.xilinx.com/s/question/0D52E00006vFXkwSAG/is-there-a-workaround-for-vivado-rdiprog-crash-during-synth-while-running-20211-on-ubuntu-18046-lts?language=en_US. They points out that the RAM issue with linux on VM (especially small RAM machine like mine), it needs swap memory to cater for Vivado large appetite for RAM. I then follow this https://askubuntu.com/questions/920595/fallocate-fallocate-failed-text-file-busy-in-ubuntu-17-04 to create swap memory and finally after running ~5hrs, managed to generate the XSA file.

3. To Be Continued... later

I'm not a person give up easily, but the clock is ticking and running out of time. Don't mean to find excuses, but my reason for DNF on time is:

• Underestimate the learning curve of FPGA. There are many issues along the way, installing the software (Vitis, Vitis-AI) already not an easy task. E.g., I have once attempt to install Vitis in ubuntu 20.04 then only realize need to install patch. Then I rather switch back to ubuntu 18.04.
• Failed to catch up / digest how to create/modify an application for FPGA. I have been following the tutorial/guidelines which is easy to get started. But it's a tremendously huge task to attempt to modify it. E.g., getting a different model to work on the smartcam app is not working per expectation. Needless to mention about creating a new application.
• Doesn't fully understand what the application/KV260 is capable of. I'd have thought at the beginning I can refer to the existing baseline/reference application get something to start with, e.g., modify some software program to change the detector label, once confident then further changes to add custom model and label. However, I don't have working example to achieve that.

I will continue invest my time to learn FPGA after this closure, hoping to come back stronger one day.