PX4 Gets a Neural Upgrade

Challenges abound when building aerial drones that can take to the sky autonomously. The control and navigation systems of these vehicles must be capable of getting to their destination while avoiding obstacles in dynamic environments in three dimensions. This is not the sort of job a programmer wants to define hardcoded rules for because the possibilities are so immense. So, neural networks are a favorite tool of these developers. The idea is to give them lots of examples of what successful flights look like so that they can learn to handle the details on their own.

This is a great plan, but it comes at a hefty cost. These algorithms tend to be very expensive computationally, yet onboard computing resources are very limited in terms of both processing power and energy consumption. Moreover, many popular flight control systems do not make it easy to deploy a custom neural network to the hardware. But now a team at the Norwegian University of Science and Technology is trying to simplify that process. They have developed and documented a method that makes it easy to deploy TensorFlow Lite models to the popular open source PX4 flight control software, which supports many different hardware platforms.

The approach begins in simulation, using a GPU-accelerated environment called the Aerial Gym Simulator. This platform, built on NVIDIA’s Isaac Gym, allows them to simulate different multirotor drone configurations and train neural network policies using reinforcement learning. They use the Proximal Policy Optimization algorithm to teach the system how to follow position setpoints. The simulator incorporates a detailed physical model of the drone, including its mass, inertia, thrust-to-torque ratios, and motor response times, to closely mirror real-world performance.

Once a policy is trained, the team converts the neural network to a format suitable for embedded systems. First, they translate the model from PyTorch to TensorFlow Lite Micro, a library designed to run inference on resource-constrained microcontrollers. Then, using a custom script, they convert the model into a C-array that can be embedded into PX4’s firmware. This enables the neural network to run directly on the drone’s flight controller without needing additional onboard computing.

This works in conjunction with a new PX4 module that replaces the standard control pipeline. Instead of the traditional cascade of position, velocity, and attitude controllers, a single neural network handles the entire job end-to-end. The module hooks into PX4’s uORB middleware, listens for the drone’s state updates, and publishes motor commands in real time — at a frequency of 650 Hz on the Pixracer Pro board.

The system was tested on a custom quadcopter called the Learning-based Micro Flyer using the Pixracer Pro flight controller. The neural network, composed of two hidden layers with 64 and 32 neurons respectively, runs entirely within the limited RAM of the controller and performs inference in under 100 microseconds. Testing in both simulation and the real world showed that the network achieved tracking performance comparable to traditional control methods.

This open-source work significantly lowers the barrier to experimenting with neural flight controllers, and that may accelerate research and development in autonomous aerial robotics in the years to come.