Simulate Mobile Aloha in Mujoco

Simulate Mobile Aloha in Mujoco

MuJoCo is a free and open source physics engine that aims to facilitate research and development in robotics, biomechanics, graphics and animation, and other areas where fast and accurate simulation is needed.

MuJoCo offers a unique combination of speed, accuracy and modeling power, yet it is not merely a better simulator. Instead it is the first full-featured simulator designed from the ground up for the purpose of model-based optimization, and in particular optimization through contacts.

Agilex achieved Mobile Aloha simulation in Mujoco and will share the instructions in this project.

Build environment

The system version of this project is Ubuntu20.04

Install MuJoCo

1. Download the MuJoCo version 2.1 binaries for Linux or OSX.

2. Unzip the downloaded mujoco210 to ~/.mujoco/mujoco210.

Install and use mujoco-py

For more details: https://github.com/openai/mujoco-py

To install and use mujoco-py, please make sure the below libraries were installed.

sudo apt install libosmesa6-dev libgl1-mesa-glx libglfw3

If you have the above libraries installed but still see -lGL not found errors, you most likely need to create the symbolic link directly:

sudo ln -s /usr/lib/x86_64-linux-gnu/libGL.so.1 /usr/lib/x86_64-linux-gnu/libGL.so

To include mujoco-py in your own package, add it to your requirements like this:

mujoco-py<2.2,>=2.1

Enter the Aloha simulation environment

The directory structure is as follows:

aloha_mujoco
    └── aloha
        ├── CMakeLists.txt
        ├── meshes_mujoco
        │   ├── aloha_v1.xml
        │   └── meshes_mujoco
        ├── package.xml
        └── scripts
            ├── aloha_ctrl.py
            └── aloha_ctrl_test.py
​

Among them,meshes_mujoco mainly stores stl files and description xml files of mujoco.

Open the aloha_v1.xml file in mujoco:

~/.mujoco/mujoco210/bin/simulate /<your path>/aloha_mujoco/aloha/meshes_mujoco/aloha_v1.xml

The following interface will be displayed:

Mujoco interface introduction

Right column:

Joint displays the angle of each joint in radians.

Control is used to control joint angles. Unit: radians.

Drag the slider in control to control the movement of a certain joint of the robotic arm.

Scroll down the left column to see the Rendering tab. Click it and slide down. After the Camera option lights in the Model Elements tab, you can see the views of the three cameras:

Enter Aloha simulation with python files

1. Run 'test'

Note: Please make sure you installed mujoco_py, and add below in bashrc:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.mujoco/mujoco210/bin
export LD_PRELOAD=$LD_PRELOAD:/usr/lib/x86_64-linux-gnu/libGLEW.so:/usr/lib/x86_64-linux-gnu/libGL.so

Run the aloha_mujoco/aloha/scripts/aloha_ctrl_test.py file in the folder. Seven windows as below will pop up, but three of them have no pictures because the camera pictures are sent through another window created by OpenCV.

The robotic arm will move every 2 seconds, and you can see that the camera image changes accordingly.

2. Let two robot arms in Mojuco follow the movement of the main robot arm in the real world

(1) Firstly, open a new terminal and run roscore

(2) Secondly, run the master robotic arm node. Open both robotic arm nodes, and check whether the data of /master/joint_left and /master/joint_right are normal.

(3) Finally, run the aloha_ctrl.py file. If there is no problem with the operation, seven windows will pop up as above, and you can move the main robotic arm to control the robotic arm in the simulation environment.

Troubleshooting

Missing GLFW

Common error during installation is:

raise ImportError("Failed to load GLFW3 shared library.")

This happens when the glfw python package cannot find the GLFW dynamic library.

MuJoCo has a copy of this library that can be used during installation.

Add the path to the mujoco bin directory to the dynamic loader:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.mujoco/mujoco210/bin