The size and frequency of wildland fires in the western United States have increased in recent years. In 2018 alone, 8,527 fires burned an area of 1.9 million acres in California, with an estimated economic cost of $148.5 billion. On high fire-risk days, a small fire ignition can rapidly grow and get out of control. Consequently, the detection of wildfires in the first few minutes after ignition is essential to minimizing their destruction. However, it can take much longer for a fire to be reported using existing methods, especially in areas with less human activity. Deep learning-based wildfire smoke detection systems can accurately and consistently detect wildfires and provide valuable intelligence to reduce the time to alert authorities.
The goal of the wildfire/Fire Hazard detection system is to adopt machine learning (ML) methods in the unmanned aerial vehicles to do fire and fire hazard detection in real-world scenarios in real time.
The DJI Mobile SDK enables you to automate your DJI Product. We can control flight, and many subsystems of the product including the camera and gimbal. Using the Mobile SDK, create a customized mobile app to unlock the full potential of your DJI aerial platform.
With the Apple Vision framework, we can recognize objects in live capture. Vision requests made with a CoreML model return results as VNRecognizedObjectObservation objects, which identify objects found in the captured scene. At their Worldwide Developers Conference in 2019, Apple added object detection support to CreateML, their no-code machine-learning app. This means, in theory, we can get a trained model suitable for use in your iPhone application without writing a single line of code.
In this project, we have trained a new CoreML Model by using a dataset of Wildfire published and novel photo video collected from the internet. We integrate such a Model into an IOS App based on the DJI Mobile SDK, and aim to use the Deep Neural Network Model to analyze video streams captured from autopilot drones and detect wildfire and fire hazards in real-time. The system includes a DJI Mavic Pro Drone, IOS App to control the Drone and to do real-time fire and fire hazards detection.
3. METHODS
3.1 System ArchitectureWe used the Mavic Pro Drone from DJI and ran the IOS FireDetection App, based on the DJI IOS Mobile SDK, and included a newly introduced Deep Neural Networks model for Fire/Fire Hazards detection. The system architecture is in Figure 1.
Figure 1: Fire & Fire Hazards Detect system based on DNNs and Autopilot Drone
3.2 Training and Testing Images.The dataset used to train the model include 2 groups of images, one group is 713 Wildfire images released by AI for Mankind in collaboration with HPWREN [6]. Another group is 86 dry leaves on roof images downloaded from internet photos and video. We divide each group of images into 3 subsets: 70% for training, 20% for validation and the remaining 10% for testing.
Figure 2: HPWREN Wildfire Dataset
Figure 3: Fire Hazards Dataset
3.3 Annotated Dataset with RoboflowRoboflow provides everything we need to turn images into information. They built all the tools necessary to start using computer vision, even if we are not a machine-learning expert.
Figure 4: Whole Dataset Annotated on Roboflow
Figure 5: Fire Detection CoreML Training Report
3.5 Model Training with CreateML and Google ColabYoloV5 is best known for object detection, we trained our FireDetection Model on Google Colab by using an annotated dataset from Roboflow. The Model was tested with testing dataset including the package, and non-related Wildfire Video from the internet, more details in Part 4: Results.
Ideally, we could directly use CoreML format of Object Detection Model generated in above step in our application, but found it not working. So we retrain a CoreML model with an Apple CreateML environment with the same dataset.
3.6 Autopilot function in DJI Mobile SDKDJI provide a IOS Mobile SDK, which could help to control the Drone and Retrieve real-time video stream captured from its Built-in camera, it also support efficient flight paths using predefined waypoint actions and adjustable parameters like altitude, speed, gimbal pitch angle, aircraft heading and more.
Figure 6: DJI's Waypoints intelligent drone flight mode
3.7 Integrate CoreML Model into DJI Fly Control systemTo do real-time fire/hazards detection, it need to integrate the self-trained CoreML Model into the DJI IOS mobile SDK Sample App, our implementation is to borrow two blocks of codes from “Recognizing Objects in Live Capture.” [2] then integrate it into DJI Mobile SDK Framework, diagram below:
4. RESULTS
4.1 Testing results on Testing set of images of Model trainingWhen we trained the model with YoloV5 Notebook on Google Colab platform, it would automatically perform detection on all testing sets of images, and get 95% accurate rate, samples below.
Figure 9: Sample of YoloV5 Model Test Result
4.2 Testing Result with non-related Wildfire Video StreamWe tested a trained model with a video stream of 2018 Holy Fire from Youtube, which was recorded with a similar camera as the dataset used in this project. The fire was detected within a second and kept alarming through the 13 minutes video.
By using non-related Fire/Smoke video downloads from the internet as testing input, we found Fire/Smoke were detected at some point; its accuracy is not as good as related sources. Figure below show right detection of smoke and fire on video downloaded from Youtube.
Figure 10: Correct Detection of non-related Wildfire Video
There are lots of missing detection on the same video, see figure below:
Figure 11: Missed Detection of non-related Wildfire Video
There are some false detections as well, see figure below.
Figure 12: False Detection of non-related Wildfire Video
4.3 Fire Detect SystemWe tested the Wildfire & Hazards Detection System indoor and outdoor. For indoor demo environment, the DJI Mavic Pro camera face a screen which could display Fire/Smoke/Dry leaves on roof photo or video, it could capture recorded Fire/Smoke scene to FireDetect App on IPAD over Wireless link, alarm would be generated on the APP if built-in FireDetection CoreML Model detect interested objects.
After the Application get successfully built and loaded into I-PAD, we can see all functions provided by DJI Mobile SDK.
Figure 14: Screenshot of FireDetection App
We tested Fire Detect feature with Smoke/Fire Hazards printed on paper, and saw all subjects were detected correctly.
Figure 15: Screenshot of Detection with FireDetection App
5. DISCUSSION
Some scientists have done some research to use AI/ML to detect Wildfire at its early stage by using a static camera setup at the top of a mountain. They have collected many valuable data for Machine Learning, including the Wildfire Smoke Dataset used by this project. Another group of scientists did research on UAV to detect wildfire for its flexibility. Our project is to combine them together and apply AI/ML directly into UAV systems.
The aim of our project was to develop an application capable of detecting fire in videos and images from autopilot drones, which is robust and works in any environment. In this regard, we have experimented with video streams of real wildfire scenarios. Using Smart Drone, we can identify various suspicious incidents such as Smoke, Fire and all kinds of Fire Hazards. Of such, fire is the most dangerous abnormal occurrence, because failure to control it at an early stage can lead to huge disasters, leading to human, ecological and economic losses. Inspired by the great potential of DNNs, we can detect fire from images or videos from Drone at an early stage. This project shows two custom APPs for fire/fire hazards detection. Considering the fair fire detection accuracy of the DNNs model, it can be of assistance to disaster management teams in managing fire disasters on time, thus preventing huge losses.
Training the algorithms based on DNNs needs a large amount of data. However, current small-scale image/video fire databases cannot meet the needs. In the future, the proposed solution is to use more advanced UAV, which could capture, analyze and visualize the real environment and conduct real-time object detection on the device instead of a remote controller, and generate large mount quality dataset to improve accuracy of wildfire detection in real world and real scenarios.
When we test our Fire/Fire Hazards system, found the limitation, DJI Mavic Pro Hardware, particularly its camera. To conduct Fire Detection in real world, need more advanced system with more intelligence and could do self-training.
6. CONCLUSIONS
DCNNs have presently dominated computer vision tasks in which region-based object detection methods are state-of-the-art. These methods have different advantages, such as removing the gruesome work of manual feature extraction.
Machine Learning and Artificial Intelligence are the booming topics in the software industry now. The use of AI/ML for testing apps is also becoming the trend as there are many companies working to solve the testing challenges using AI/ML.
In this project, we found the Wildfire & Fire Hazards detection System could be a viable detector of wildfire smoke in videos taken by DJI Drone in terms of both accuracy and speed. Limitations and difficulties found in the project are lack of quality and quantity of Wildfire Images used to train Deep Neural Networks Models. We trained the Fire Hazards detection Model with most of the images captured from very limited dry leaves on roof photo and video from the internet, and we used one of the videos to test for demo purposes. We acknowledge that is not a scientific way for researching.
7. Project Video
2 Minutes Video Uploaded to Youtube:
https://www.youtube.com/watch?v=twtwpD9Ws3Y
1. Experiment Result with Holy Fire 2018:
2. Experiment Result for fire hazards detection with leaves on roof video
3. Experiment Result for fire hazards detection with Wildfire Guymon, Ok- Drone captures
4. Experiment Result for fire hazards detection with Drone footage shows devastating aftermath of wildfires
5. Experiment Result for fire hazards detection with 20200812 Lake Fire Smoke Detection
David Lei, Michael Lei
Bill Lei
Comments