Programming drones with Raspberry Pi on board easily

Programming is utterly cool and so are drones, so it's hard to imagine more exciting activity than making drones to fulfill our wishes using the plain ol' Python code.

But here's the thing: quadcopters, these nimble bastards, are never willing to do what they're told. It's a lot more fun for them either to refuse to fly, or to fly like hell, crashing into the first chandelier they can find (and they're pretty good at it, trust me). To circumvent this highly unfortunate trait we've created an open source framework, called clover, or just 🍀, which does all the heavy lifting on a quadcopter to allow you to take off with a single line of code. It executes on the onboard Raspberry Pi computer, using its camera for the computer vision navigation.

By the way, this is not an exaggeration about one line of code:

# Take off and hover at 1 m above current position
navigate(x=0, y=0, z=1, frame_id='body', auto_arm=True)

The main documentation is available at https://clover.coex.tech. Github repo: https://github.com/CopterExpress/clover.

Thousands of developers and students use clover to code a drone — and I'll tell you, why you should join them.

Why should I, really?

• Many of you have quadcopters, but is it really cool to pilot them with a remote control? Especially when there's a way to make the drone bring you beer, chase your friends, perform aerobatics... completely on its own! Here are some examples of what an autonomous drone can do:

• Drones are the most involving and interesting way to learn programming. Whether you are learning, teaching your child or work in an educational institution, drones are great tool to teach Python, C++, computer vision and Linux. You don't need gamification or other tricks to make the education appealing anymore. Drones are being used in universities as well — computer vision, machine learning and swarms are trending academic topics.
• The drone programming skills are an extremely valuable nowadays. Not only a lot of companies start their drone deliveries, but drones also use the same technologies as self-driving cars, so you can apply your skills to create other autonomous vehicles, or robots of any kind.
• If you like participating in open source projects, drones are amazing opportunity to have fun and contribute at the same time! Robotics projects usually don't have an abundance of contributors, so every contribution is highly appreciated.

Isn't it too hard?

Autonomous vehicles, as any advanced robots, are incredibly hard to do from scratch because of complexity of the hardware and its interactions with the environment. We created clover framework to reduce the barrier of entry as much as possible — all you need to do is to assemble the drone and flash our preconfigured image on the Raspberry Pi's microSD card. Then you can connect to your drone with Wi-Fi and use either SSH or web interface to make it fly. Thanks to ROS, popular open source robotic middleware, you can use almost any programming language to interact with the drone's API.

I've already shown how to take off using the Python API. Let's try something a bit more complex, for example, that's how to fly by a rectangle trajectory in the endless loop:

while(True):
    navigate(x=0, y=2, z=0, speed=2, frame_id='body')
    rospy.sleep(4)

    navigate(x=0, y=0, z=1, speed=2, frame_id='body')
    rospy.sleep(4)

    navigate(x=0, y=-2, z=0, speed=2, frame_id='body')
    rospy.sleep(4)

    navigate(x=0, y=0, z=-1, speed=2, frame_id='body')
    rospy.sleep(4)

x, y, z indicate the coordinates to which to fly, and frame_id determines what these coordinates are relative to: they can be relative to the drone's current position, or to some recognized object, or to the room, or even to Earth, if we use GPS. speed is pretty self-explanatory. Pretty simple, huh?

If even that is not simple enough, you can use the visual web interface based on Google Blockly:

Thanks to the onboard camera, you can use OpenCV to recognize objects or people and act accordingly, e.g. program the drone to look for some object, grab it using a gripping device, drop in some other location, etc. Here the drone finds the yellow circle and lands on it:

Awesome, where do I start?

Typical consumer drones (like DJI Mavic) are not open source and it's impossible or at least inconvenient to control them via code. Their frames usually don't support installation of additional components like onboard computer — these drones are meant to be used as they are.

To use clover software, you need a drone with a Raspberry Pi 3 or 4 on board — anything with a PX4-compatible flight controller will do.

Hey, what's a flight controller?

Keeping it simple, a quadcopter consists of a frame, a battery, a flight controller (aka autopilot), motors and their ESCs (electronic speed controllers). A flight controller is a device equipped with microcontroller (like Arduino) and a bunch of sensors: accelerometers, angular rate sensors, magnetometers and barometer. These sensors provide data for microcontroller to calculate drone's current attitude, and with addition of an external positioning data, e.g. GPS, it also determines its position and speed. Using this information and control inputs (commands from the remote control or onboard computer), the flight controller calculates required motor commands and passes them to ESCs, which control drone's motors.

PX4 is a popular open source autopilot firmware, supported by Dronecode Foundation, a Linux Foundation Collaborative Project. Its architecture is perfect for autonomous flights, controlled by a companion computer instead of a remote, and allows you to extend its functionality easily.

Clover docs contain info on how to connect Raspberry Pi so you can get your drone in the air in no time. Equip your Raspberry Pi with a camera to fly indoors.

If you don't have a right drone, there are the clover quadcopter kits, perfectly suitable for tinkering.

What if I'm not ready to mess with the real drone yet?

If you don't have a drone yet, you can still learn how to code them using the simulator. You can download clover OVF VM image, run it with Virtualbox or VMware Player and test your code in a virtual environment. It contains Ubuntu, ROS, and a simulated drone, ready to be in the air in a matter of seconds! Here is a short video:

More in-depth documentation on simulation is available in the PX4 docs for developers. The simulation is extremely useful during the drone development, allowing to create elaborated CI pipelines for the onboard software.

What is ROS and how exactly the drone is being controlled?

ROS is the most popular robotics middleware out there. It's a bunch of tools to organize your robot's software: you split your application to separate programs, called nodes. You can think of nodes as microservices for robots. Nodes can pass messages to one another through topics, just like microservices do using RabbitMQ/ZeroMQ, and call each others services, which are essentially RPCs. For example, one node can process camera feed, recognizing some objects and posting their coordinates in a topic, while the other can calculate a trajectory to grab these objects and call the "grab" service of the third node, operating the gripping mechanism.

This decentralized architecture lends itself well to robots, providing convenient introspection and debugging. Using RViz, ROS visualization tool, you can visualize the position of your drone, all the recognized objects and any other topics in real time:

This approach also leads to a loosely-coupled architecture — nodes, having defined interfaces, can easily be shared with others. As a result, there's a lot of open source ROS packages on GitHub and in the ROS Index. Another advantage of the peer architecture is that nodes can be written using any language which has the ROS client libraries, including C++, Java, Python, Go, C#, Rust, and many others. You can even interact with your nodes from web browser using ROS JS library.

Your program, written using clover framework, will run on Raspberry Pi on board. It can use the data from onboard cameras and sensors to determine what to do next — and then issue a command to the drone's flight controller. You can order the drone to fly to a given point, or even directly control its speed or attitude, by calling the ROS services, provided by clover ROS package (more detailed description of available services can be found in the clover docs). Raspberry Pi will then send the commands to PX4, flight controller firmware, using MAVLink protocol.

How the drone navigates indoors?

Flying indoors, a drone can't use the GPS to calculate it's position, but thanks to the camera-equipped Raspberry Pi on board it can use computer vision to determine where is it. clover framework supports indoor navigation using ArUco markers (they're like large QR codes) and optical flow.

When the drone sees an ArUco marker, it can calculate its relative position, which allows to fly with respect to the marker. Let's tell the drone to take position at 1 m above the marker with ID 107:

navigate(x=0, y=0, z=1, frame_id='aruco_107')

Usually we recommend to use multiple markers, located all over the floor of the flying space, so the drone always sees few of them. Markers can also be placed on some objects like an item to pick up or even a moving platform to land. Here the drone is trying to stay above the marker with ID 225 (yep, just one line of code):

navigate(x=0, y=0, z=1, frame_id='aruco_225')

When it's not convenient to place ArUco markers, it's possible to use optical flow navigation. It works like the sensor of an optical mouse: the camera is pointed downward and the drone detects how the floor moves relatively to it. It doesn't require any preparation, but allows to execute commands only relative to the current position ("move 2 meters to the right") and not relative to the marker or the room ("fly to where you took off" or "fly to the center of the room").

The optical flow calculation requires knowing the altitude precisely, so a laser rangefinder is a must. Fortunately, they are cheap and tiny these days — even $5 STM VL53L1X is enough.

What's next?

The quality and quantity of the beginner's documentation in the field of robotics is often less than desirable, especially in comparison to more popular areas of software development, like web or mobile dev. However, thanks to the efforts of the open source community, in recent years robotics has become an increasingly friendly field. Here are some links in case you decide to dive deeper in the field of the aerial robotics:

• ROS wiki covers all the aspects of ROS,
• OpenCV docs contain a lot of tutorials on using C++ and Python for computer vision,
• PX4 user guide combined with their dev guide provides in-depth information about drones, principles of their software control and PX4 software stack,
• clover docs contain a lot of information on clover, ROS, PX4 and their interactions, and you can find webinars and other interesting stuff on our YouTube channel.

If you have any questions, the open source community is always there to help — ROS has its Q&A forum, PX4 project has an enormously active Slack community, and clover framework has a Telegram chat.

I hope you've learned something interesting, whether you become a drone developer or not. :) Stay safe, and stay contributing!