For Reproducibility, DUCKIENet Fits the Bill

Mobile robots — such as autonomous vehicles — present many challenges when it comes to assessing the performance of, and evaluating differences between, algorithms. Robots are complex, composed of many interacting components to be considered. Unique or uncontrolled environments also add to the difficulties. These are some of the reasons that mobile robot research is often very difficult to reproduce. The algorithms found to be successful in one study may not work as expected when tried in another implementation.

A group of researchers centered at ETH Zurich have put forward a new concept to deal with these issues, and have created an implementation of the concept by the name of DUCKIENet (Decentralized Urban Collaborative Benchmarking Network). The acronym may not quite work, but DUCKIENet certainly sounds more appealing than DUCBNet, so I will let it slide. DUCKIENet is built on top of the Duckietown platform — a reproducible framework for autonomous vehicles operating in model urban environments. Duckietowns consist of standardized, modular environmental elements, such as road segments, traffic signals, and traffic signs. Inhabiting the Duckietowns are Duckiebots — simple robots consisting of DC motors, a camera, LEDs, and a Raspberry Pi for computation. Duckietown also includes a software package that is used to control the Duckiebots, to facilitate communication between processes, and to provide a virtual environment simulator to speed training and debugging.

Extending Duckietown, DUCKIENet adds a Challenges server, evaluation servers, and Duckietown Autolabs (DTA). The Challenges server stores benchmarks and results, computes leaderboards, and schedules jobs to be executed by evaluation servers. Evaluation servers run simulations, and DTAs are physical labs composed of Duckietown hardware ecosystems, either local or remote, that can carry out physical experiments.

The researchers consider one of the primary applications of DUCKIENet to be in hosting competitions. With environment and hardware fixed, a level playing field is set for evaluating performance of algorithms.

DUCKIENet is built from inexpensive components on top of the open source DuckieTown platform, making it quite accessible. However, there are some tasks at DTAs that have not been automated (e.g. resetting experiments, charging Duckiebots) which is likely to limit the availability of remote DTAs for general use. Moreover, DUCKIENet environments are very simplistic model environments, and algorithms built using them will not have much applicability to real-world scenarios. However, as an educational tool, DUCKIENet holds promise for training future generations of machine learning engineers.