Fire Detecting Drone

Hi Everyone!

As you probably know wildfires are a huge problem, both here in California, and worldwide. In the last decade we witnessed the costliest, the most destructive and the deadliest wildland fires on record. Recent examples include the 2018 Camp Fire -- the deadliest and most destructive fire in California's history, and record 2019–20 bushfire season in Australia. I am working on an autonomous drone that can detect wildfires early using Computer Vision and Convolutional Neural Networks.

1.How it works?

My project uses the camera and a GPU-accelerated Neural Network as a sensor to detect fire.

1.1 Dataset

The major problem with using Machine Learning for solving problems is that they require large datasets to train on. Unfortunately, existing fire datasets usually either images from a lab setting or human-perspective images, which don't transfer well to recognizing wildfires from air. So I created a domain-specific fire dataset 2'000 images myself. It's consists of drone imagery from real-world fire scenarios I collected from youtube.

You can download it here or at firedataset.org.

1.2 Neural Net Training.

The neural network training & evaluation code is available on my github. The training was performed on Nvidia Tesla P100. The pre-trained weights are available here.

1.3 Inference

The images from the RGB camera aimed downwards are captured at 1920×1080 and split into k=N×N segments. The number of segments can be adjusted.

1.4. Performance

I evaluated the image classifier on a number of architectures on Jetson Nano with pretty good results. The FPS numbers are surprisingly high! For instance, flying my drone at 10m/s at altitude of 30m I need to take an image around every 5.4s in order to provide continuous land coverage. That means I can easily run 70 network inferences between captures, including the overhead of capturing & resizing the images.

2. How to build your own?

Hardware

I could describe in detail the steps to reproduce building the exact same hexacopter platform, but in reality you can run that on most DIY drones. The only strict requirements for a drone to be compatible with this project are:

1. The ability to carry 500g of payload (jetson nano + cameras + DC/DC converter + radios).

2. Flight controller that's capable of MAVlink communication. Your easiest bet will be a Pixhawk FC board with Ardupilot/PX4 software.

Jetson with raspberry pi camera. It is connected to Pixhawk 4 Mini with a custom cable, like here. I use a 10A DC/DC converter to power the system. The case is adapted from:

https://www.thingiverse.com/thing:3649612

Here's how my setup looks like mounted on a drone.

This is a version with belly-mounted RGB + IR cameras for mapping

Software

Make sure you can run a basic inference pipeline on Jetson Nano.

2.1. Download the weights file

2.2. Download the code

git clone https://github.com/tomek-l/fire-detect-nn.git

2.3. Install the requirements

pip3 install requirements_for_inference.txt

2.4. To install pytorch on nano, follow the instructions here

2.5. Make sure you can run the simplest inference pipeline

The most basic python code (also available in the github repo) for inference on Jetson Nano is available below:

import torch
import os
import time
import cv2
import numpy as np
from model import Model
# gstreamer_pipeline returns a GStreamer pipeline for capturing from the CSI camera
# Defaults to 1280x720 @ 60fps
# Flip the image by setting the flip_method (most common values: 0 and 2)
# display_width and display_height determine the size of the window on the screen
def gstreamer_pipeline(
capture_width=1280,
capture_height=720,
display_width=224,
display_height=224,
framerate=60,
flip_method=0,
):
return (
"nvarguscamerasrc ! "
"video/x-raw(memory:NVMM), "
"width=(int)%d, height=(int)%d, "
"format=(string)NV12, framerate=(fraction)%d/1 ! "
"nvvidconv flip-method=%d ! "
"video/x-raw, width=(int)%d, height=(int)%d, format=(string)BGRx ! "
"videoconvert ! "
"video/x-raw, format=(string)BGR ! appsink"
% (
capture_width,
capture_height,
framerate,
flip_method,
display_width,
display_height,
)
)
def avgerage(l):
return sum(l)/len(l)
# We can get some nice 8FPS from this image
if __name__ == "__main__":
# To flip the image, modify the flip_method parameter (0 and 2 are the most common)
print(gstreamer_pipeline(flip_method=0))
cap = cv2.VideoCapture(gstreamer_pipeline(flip_method=0), cv2.CAP_GSTREAMER)
device = torch.device("cuda:0")
model = torch.load('resnet50-epoch-15-valid_acc=-1-test_acc=0.9856.pt')
model = model.to(device)
if cap.isOpened():
while True:
start = time.time()
ret_val, img = cap.read()
img = np.swapaxes(img,0,2) # WxHxchannel convention to channelxWxH convention
img_mini_batch = np.expand_dims(img, axis=0)
tens = torch.Tensor(img_mini_batch).to(device)
result = model(tens)
print(result)
print(time.time()-start)
cap.release()
else:
print
(
"Unable to open camera"
)

2.6. If the code above works, you can proceed to deploying on a real drone by following the README here: https://github.com/tomek-l/drone-scan. This code will allow you to scan an area and take images autonomously. When the drone detects fire, it will print an alert on the console. Hopefully soon I will be able to integrate it with a more sophisticated fire-response system 🙂

Visuals - what the drone can see

Here's what the drone 'sees' with a dual camera setup when flying over the fire.

Here's a Grad-CAM visualization, which is a useful technique to determine what's happening inside the neural network.

To understand how this video was created, read more about Grad-CAM visualizations: https://arxiv.org/abs/1610.02391 or read through the notebooks on my github 🙂

4. Code and resources

Hackster allows you on only one repository in the description, so I list them here.

4.1 The code for training and inference:

https://github.com/tomek-l/fire-detect-nn

4.2 The code for controlling the drone:

https://github.com/tomek-l/drone-scan

4.3 The pre-trained weights:

https://drive.google.com/file/d/1N7DwrWABhy7AcayZQggG2Hql706yI7wk/view?usp=sharing

4.4 Custom drone fire dataset:

https://drive.google.com/open?id=1jzMxBJWlZ7jv4z3LEKtQfp84Ptk2VXyV