Pandemic Survey Drone

NavQ VNC server:

Today I received the NXP NavQ board to start off my development for the HoverGames Challenge 2. After unboxing I followed through the getting started guide in https://nxp.gitbook.io/8mmnavq/getting-started/quickstart. And voila I have a Linux computer running and I can access it via USB to Serial connection.

Like any other Linux computer I have I prefer to connect the device to the internet and do the following 3 things:

Update

• Update

Upgrade

• Upgrade

Install a VNC server

• Install a VNC server

The first 2 steps went smooth but I got stuck at remotely running a Linux desktop through my VNC server. My VNC server of choice was tightVNC. I followed the exact same steps I used for my Beaglebone Black or Raspberry Pi. I tcan be found in my blog post here.

But the client application for tightVNC was unable to connect to the server even though the ip and port was correct. I tried to dig deep and try a few other VNC clients. The RealVNC client did manage to log in but I was presented with the following screen.

I quickly realized that there was no desktop environment installed in the Linux distribution that came with NavQ. So after a few google search I decided to install Xfce4 as my desktop environment. Here are the steps.

First kill the already running vncserver using the following commands:

$ vncserver -kill :1

Now install xfce4 and xfce4-goodies :

$ sudo apt install xfce4 xfce4-goodies

Choose default (=N) for all the prompts that appear. It takes a while (around 6-7 minutes for me) based on your internet connection speed. Once everything is installed properly I configured the VNC server startup files to choose Xfce as the desktop environment by modifying the xstartup file. Before modifying make sure to backup the current file.

$ mv ~/.vnc/xstartup ~/.vnc/xstartup.bak

Now create a new xstartup file and open it in your text editor:

$ vi ~/.vnc/xstartup

Commands in this file are executed automatically whenever you start or restart the VNC server. We need VNC to start our desktop environment if it’s not already started. Add these commands to the file:

#!/bin/bash
xrdb $HOME/.Xresources
startxfce4 &

To ensure that the VNC server will be able to use this new startup file properly, we’ll need to make it executable.

$ sudo chmod +x ~/.vnc/xstartup

Now startup the vncserver:

$ vncserver

Connect with proper credentials from the windows client. For me the vncserver was hoste on port 5901 so I used imx8mmnavq::5091 to connect. Use your password used while installing tightVNC and voila! You have a shiny new NavQ Linux Desktop.

Installing MAVSDK:

The NavQ linux distribution comes with Python 3.8 but it is missing pip which is a python package installer. We need pip to install MAVSDK for Python. Here are the steps. References has been taken from here.

Install pip :

$ sudo apt-get install python3-pip

once it completes check pip version with

$ pip --version

To install MAVSDK:

$ pip3 install mavsdk

It takes a while to install.

Next install aioconsole for handling asynchronous tasks in python:

$ pip3 install aioconsole

Assembling the Drone:

Although the instructions are very straightforward I did come across some hurdles due to my own mistakes. Initially my drone was flipping on one side on takeoff.

This can happen because of few reasons:

• The airframe in QGroundControl is not set to S500/S500 Generic.
• The ESC wires are not plugged in proper sequence to the Flight controller.
• The direction of the motors are not as described in the assembly gitbook.

After taking care of the above changes I finally managed to make it fly.

Installing MAVSDK(contd.)

• Follow my tutorial here for complete instruction on how to get MAVSDK_Python up and running with NavQ and FMUK66

Finite State Machine and Asyncio

I won't talk much about FSM here since it is a very popular concept in the Software Development world. I chose to implement an FSM for this project because this is an autonomous drone and it needs a flow of logic which needs to be structured in a way that it works as expected.

Since most of MAVSDK makes use of asyncio I ended up designing a python library to implement an FSM using asyncio. The flow chart for the FSM is as described below

This provides a very basic framework for autonomous drone software. Let's dive deep and see each step in detail.

idle : In this state the companion computer downloads the MissionPlan from PX4 using MAVSDK. There is a basic sanity check which checks for the length of the mission plan which is nothing but a list of MissionItem. For a survey mission the size should be greater than 0 or at least one MissionItem.

start_flight: Starts the mission. Waits for the drone to be in air and then changes state to photo_capture

photo_capture : This state constantly checks for if the drone is in air and if it is in air it transfers control to the capture_manager which is responsible to take the photos while in air. The capture manager makes use of MissionProgress of MAVSDK. MissionProgress gives the count of the current MissionItem and the total number of MissionItems. If at any point in the mission the drone is not in air and the mission has not finished the FSM goes to an error_state

capture_manager : The logic behind the capture_manager is that whenever the mission reaches a new MissionItem the software checks if that MissionItem has any CameraAction to be executed. If it finds a CameraAction.TAKE_PHOTO it requests the on board Coral camera to take photo using openCV library for Python.

landing_state : Checks if the drone has landed and the MissionResult is a SUCCESS. If not the software goes into error_state.

odm_manager : This state is responsible for send the photos we captured to WebODM for processing and it gets back with the processed image in the form of a 3D model of the area. We will talk more on this later. Since this is a time consuming blocking task I wrapped this into a concurrent thread using the technique showed buy the awesome guys at Auterion in this post.

mission_end : This is just a termination state for the entire FSM. This state is reached when the FSM goes through all the above states and work as expected and the mission is a success.

error_state : This is also a termination state which is reached whenever there is a problem in any of the states in the FSM and the mission is failed.

WebODM

WebODM is an open source project for Drone Image Processing. The guys at OpenDroneMap has done an awesome job at creating an open source project to make life easier to generate survey grade ortho rectified images and 3D maps from a collection of 2D images. A big shoutout to them. Do checkout their work here

WebODM is a version of OpenDroneMap code to be run on a webserver. In order to cut the hassle of hosting a webserver and then running WebODM there I chose an easier path which is WebODM Lightning. It is a paid web application service provided by the OpenDroneMaps team to make use their awesome software just through a few lines of code. When you sign up for the service you get 150 free credits which I made use of for this project.

The steps to get started is easy:

• Open WebODM Lightning from

• Click on "Sign up" at the top right corner
• Once you finish your sign up process you will be directed to your dashboard

• In your dashboard go to "Processing Node" section and copy the Hostname, Port and Token. You need these credentials to use the software.
• Go to the project folder cloned from the repository and open auth.py
• In auth.py replace the hostname, port and token with your own credentials.
• Finally run the odm_test.py to test odmmanager.py with the sample images.
• It takes a while to process if you have large number of images.
• You can track your progress in your WebODM Lightning Dashboard

• Once the processing task is complete it will download the results in the folder called "results". The results folder contains the ortho rectified image in different formats like GeoTIFF, 3d object file, etc.
• The result files are compatible with most major GIS software for planning and mapping purpose.

Getting Started

To get started with this project these are the steps to perform

• Clone the project repository in the companion computer
• Install the required dependencies like MAVSDK-Python, PyODM, Opencv-Python
• Using QGroundcontrol create a survey mission with an overlap of more than 60%. 80% is a safe number. As hown below

• Upload the newly created mission to your PX4 flight controller and disconnect the computer running the QGC
• Now SSH over WiFi to your companion computer and run the python script survey_drone.py
• This script will implement the FSM mentioned above and complete the mission following the FSM flowchart above.
• Make sure there is WiFi Internet connectivity on the landing site. A better way of doing things are connecting a GSM module to directly connect to the internet.
• Once the mission is complete go to WebODM dashboard and track the progress of the image processing task.
• Once the processing task is complete download the result from the WebODM dashboard or by SSH ing into the companion computer.
• Use the result files into your survey plotting software like Indvision or QGIS to create route map or calculate the area of densely populated areas for use by first responders.