Robotic Simulators: What’s the Daily Special?

This article is based and updated from this blog post.

You must have seen it on a robotics newsletter, LinkedIn, or anywhere on the news: Artificial intelligence is in again and they want it to have a robotic body! Embodied AI, ideally, on a humanoid robot.

But what if you don’t trust this body? You might also have seen some videos of a factory robot going haywire? And that is even without the AI? How about putting these robots in a virtual environment or robotic game, so that the robot can do its thing while it's learning away without hurting itself or others.

Trends in robotics come and go, but there is one matter that will always stay in any robotic developer’s toolbelt. Designing a full robotic system with hardware and software in the loop requires a constant iterative process. This makes simulators an indispensable tool.

But which simulator can you choose? How can you be sure it has all the capabilities necessary?

What is a robot simulator?

It is surprisingly difficult to explain. If you ask someone who comes from physics and material science, their definition of the word 'simulation' is very different from how we (robotic engineers) view simulation. The meaning or implementation is very different depending on which field you are from. Basically, the Wikipedia page definition of a simulator is:

A simulation is an imitative representation of a process or system that could exist in the real world.

This is an extremely broad term and can stand for anything, from a pseudo-random dice roll program to a finite element analysis of a steel beam, to a high-fidelity rendering of camera footage of a virtual drone. This is just to illustrate how big the term actually is!

And how about a simulator for robotic systems? Gemini had this to say:

A robotic simulator is a software framework that provides a virtual environment, often leveraging different physics/rendering engines and sensor models, to model the robot's behavior, its interaction and perception with the simulated world for design, evaluative or data-generative purposes.

That seems more like the definition that robotics engineers are most familiar with when talking about simulation. There are some key words in there that will probably be important:

• Virtual environment — To provide the scenario for the simulated robot to act in, depending on the application, like an indoor building, forest, or lunar landscape.
• Behavior, its interaction and perception — The simulated entity should be able to interact with and act upon that virtual environment or world through its simulated sensors and actuators.
• Physics/rendering engines and sensor models — To be able to simulate those interactions and perceptions caused by the robot's behavior, to model how an object will slip while being grasped or the noise from lidar readings.
• Design, evaluative or data-generative — To use this as a development tool, as part of continuous integration to assure quality, or to collect data that can be used for AI training purposes.

For a more in-depth explanation of the definition of robotic simulation and its purposes, you should read the Ekumen blog post on robotic simulators.

A list for aerial robotic simulators

At ICRA 2023, there was a workshop for Aerial Robotic Simulators, and its original purpose was to settle on one simulator for all aerial robotic applications.The organizers (including myself) received 16 papers on all different aerial robotics simulators built or assembled for different purposes. Further investigation (in the form of a survey paper), actually found over 50 robotic simulators developed for aerial vehicles alone.

However, in just two years since the workshop, quite a few of these projects have stopped being maintained. Airsim is planned to be archived, with no continuation of the promised 'Project AirSim'. Many used Gazebo Classic, which has been End-of-Life since early this year... And finally, some simulators had Unity as a backend — Unity Robotics hub on GitHub is now very barren since the robotics team was laid off last year.

So that is a bit tricky? Robotics engineers want to go into a framework that is healthy, well-maintained, and has a development future in robotics. They don't want to constantly do the same investigation again and again, as it is very time-consuming and draining. What if you have put in all the time and effort to learn one framework, just to learn that it will be dropped in support in the next year?

This list needs to be more than just showing which options there are… it needs to show which simulators are still relevant!

Generating a best-of-robot-simulators list

As it turn out, there is an alternative to awesomelist, namely best-of list. The handy thing is that it grabs the metadata from the GitHub repository in terms of stars, downloads, forks, and activity, and makes a ranking of how well these projects are doing, fully automatically. So you can see immediately which are the most relevant and healthy projects to go for.

That was the start of creating a full fledged best-of list... of Robotic Simulators!

The GitHub repository started it out with 80 projects compiled from the earlier mentioned paper and awesome-robotics lists, in a way that the best-of generator can generate a ranking per category. Since then, the list gained traction and with the help of many contributors, it grew to 140 robotic simulator entries in just 2 months!

There are currently 11 categories in this list:

• Generic Robotics Simulators — Generic simulators, tools or SDKs made for robotics
• Robotic Simulators in 2D — Robotic simulators that only work in a 2D environment, for instance navigation
• Aerial Robotics Simulators — Simulator frameworks made especially for aerial robotics
• Maritime Robotics Simulators — Simulator frameworks made especially for maritime robotics
• Automotive Simulators — Simulator frameworks made especially for automotive
• Space Robotics Simulators — Simulator frameworks made especially for space robotics
• AI training Simulators — Simulations made for training for AI-agents like reinforcement learning
• Other Domain Specific Simulators — Robotic simulators build for other domains like automotive or space robotics
• Game Engines — 3D engines designed for games that can be interfaced with robotic frameworks
• Physics Engines — Physics Engines that simulate multi-joint dynamics, gravity etc.
• Rendering Engines — Rendering engines for robotic simulators

There is some overlap between some of these categories, and not all categories are perhaps equally presented, but this is of course a work in progress and feel free to suggest any suggestions of improvements as a GitHub Issue.

Too much choice?

The list can show which projects are still active and have a healthy community. This can be extremely useful for some, but others could be overwhelmed by the vast amount of choice for a robotics problem. It’s like being in a restaurant, getting a huge menu of options of dishes. But at least those dishes will show exactly what they contain and if they are delicious, and if they are “86ed” (out of stock).

In terms of Generic robotic simulators, MuJoCo takes the crown, however, the Open 3D Engine (O3DE) is trending upwards and has a bigger community as it also leverages the game development community as well. Both Gazebo and Webots are the simulation behemoths of the robotics community, and considered the golden standards.

And there are some new kids on the (open-source) block. The Newton physics simulator has just been released, which is a combined effort by Disney, NVIDIA, and Google Deepmind, which utilizes MuJoCo with NVIDIA Isaac Sim. The biggest news that came just last week is that the NVIDIA Isaac Sim’s code has also been open sourced as of recently, with the exception of certain build dependencies. Only time will tell if these emerging open source projects will stand the test of time and commit to their promise.