Cutting-Edge Convergence:SLAM, ROS, and Vision in Mercury X1

Introduction

In the rapidly evolving world of technology, humanoid robots are becoming increasingly significant, with their applications extending from industrial automation to the service sector. This paper will explore the use of cutting-edge technologies such as large language models (LLM), simultaneous localization and mapping (SLAM), robotic operating systems (ROS), open-source computer vision (OpenCV), and S-Tag marker technology through the case study of the Mercury X1 elephant humanoid robot. This case not only showcases the advanced functionalities of robots but also demonstrates how modern robotics effectively solve practical problems, heralding the future prospects of technology applications in both industry and service sectors. The following sections will detail the technical setup, procedures, and results of the experiment.

Technical Details

In the application case of the Mercury X1, multiple technologies were integrated to enable the robot to perform precise object grasping and moving tasks in complex environments. Here are the key technologies involved and their detailed descriptions:

1. Large Language Model (LLM)

Large language models, like the ChatGPT series, are applied in robotics to enable the understanding and execution of commands based on natural language. These models learn the complexity and context of language through processing vast amounts of text data, allowing robots to parse user commands and translate them into actions. In the Mercury X1, LLM is used to understand natural language operation commands, providing decision-making support for subsequent actions.

2. Simultaneous Localization and Mapping (SLAM)

SLAM technology is an indispensable part of modern robotics and autonomous vehicles, enabling robots to locate themselves and create maps in unknown environments without external help. Utilizing lasers, cameras, and other sensors, SLAM helps the Mercury X1 build environmental maps, update its position in real-time, and plan its movement paths. This is crucial for ensuring autonomous navigation in complex settings.

3. Robotic Operating System (ROS)

The Robotic Operating System (ROS) is a flexible framework for writing robot software. It provides a suite of tools and libraries that enable developers to create complex and reliable robot behaviors. ROS's modular design allows developers of the Mercury X1 to easily integrate different hardware and software, such as arm path planning and mobility base mapping, navigation, and obstacle avoidance, handle data communication, and manage interactions between multiple sensors and actuators. ROS is a powerful tool widely used in robotics applications globally.

4. Open-Source Computer Vision Library (OpenCV)

OpenCV is an open-source library for implementing image processing and computer vision. It plays a crucial role in this case, without which the project would not be feasible. The robot's camera uses OpenCV to analyze visual data, identify, and locate objects. OpenCV algorithms enable the robot to recognize object shapes, sizes, and precise coordinates, which are essential for accurate grasping and manipulation.

5. S-Tag Marker Technology

S-Tag marker technology is a highly reliable identification system designed to provide accurate marker recognition in visually challenging environments.

These markers are used to identify objects and locations within the operational environment of the Mercury X1 robot. Even in low light or obstructed vision conditions, S-Tag ensures the robot accurately identifies target objects through its camera system.

6. pymycobot

pymycobot is a Python library used to control the mechanical arm and end effectors (such as grippers) of the Mercury X1 robot. It allows developers to precisely control the arm's angles, coordinates, and motion modes, including interpolation mode and refresh mode. This library provides high flexibility and customizability, enabling the robot to perform complex grasping and manipulation tasks and adapt to various operational needs.

Through the application of these technologies, the Mercury X1 elephant humanoid robot can perform complex tasks such as autonomous navigation, object recognition, and precise manipulation, which are indispensable capabilities in modern automation and intelligent systems.

Product Introduction

Mercury X1

The Mercury X1 is an advanced humanoid robot developed by Elephant Robotics, specifically designed to handle various automation tasks. It features 19 degrees of freedom (with 7 degrees of freedom per arm), providing it with exceptional flexibility and adaptability in task execution. The Mercury X1 is equipped with a wheeled mobility base, driven by high-performance direct-drive motors, enabling stable movement in complex environments and offering up to 8 hours of battery life, making it suitable for both personal and commercial applications.

This robot utilizes a high-performance main controller system, equipped with the NVIDIA Jetson Xavier for powerful computing support. This setup is tasked with handling complex algorithms for visual ranging, sensor fusion, localization and map building, obstacle detection, and path planning. Additionally, the mobility base of the Mercury X1 is outfitted with LiDAR, ultrasonic sensors, and a 2D vision system, allowing for highly perceptive environmental interaction.

myCobot Pro Adaptive Gripper

The 2-finger adaptive gripper of the myCobot Pro can achieve both internal and external parallel gripping, as well as a unique encompassing grip mode. It is capable of picking up objects of any shape without releasing them. This gripper is used to perform a wide range of applications and can be rapidly deployed into production—no robotics expertise is required. It is a key component of a highly flexible and reliable robotic unit.

myCobot Pro Camera Flange

The camera module on the myCobot Pro uses a USB-B data cable to capture images. This setup allows for the efficient transfer of visual data, which is essential for tasks involving vision-based sensing and processing.

Next, we will explore how these technologies are integrated into the robot and demonstrate their performance in specific tasks.

Scenario Workflow Introduction

1. Mapping and Setting Marker Points

Mapping: Initially, the environment is mapped using the Mercury X1's onboard radar, employing SLAM technology via the ROS framework using the gmapping algorithm for 2D mapping. The robot is activated, and the system is initialized. As it moves through the environment, the radar sensors collect data. The SLAM algorithm transforms this data into a structured map while updating the robot's position in real-time. To ensure the accuracy of the mapping, it is advisable to move around in the environment extensively.

Marking Points A and B: Using the "Publish Point" tool in RViz, click on the desired positions on the map to mark them. Each click publishes a point to ROS's default topic. Repeat this process to mark point B, and record the coordinates of points A and B.

These saved positions can later be used for fixed-point cruising.

2. Moving to Desk A for Object Recognition and Grasping

After completing the mapping and marking, the Mercury X1 moves to Desk A to perform object grasping

Object detection is conducted using the camera mounted at the end of the robot's arm. Initially, if the specific details of the surroundings are unknown, the robot conducts a preliminary survey of the environment to locate targets. Once a target is identified, the robot quickly calculates the position of the object for accurate grasping. This process involves integrating visual data from the camera with the positional data obtained from the mapping to guide the robot to precisely approach and manipulate the object.

#Scan all objects code

def detect_objects_all(flag):
    if flag == 0:
        detect_objects("left", ml_camera_pos_a, camera_data_a)
        detect_objects("right", mr_camera_pos_a, camera_data_a)
        cube_data, tray_data = judge_tray_cube(camera_data_a)      else:
        ml.send_angles([-41.88, 30.62, -1.14, -102.57, 59.89, 58.11, 9.63], 10)
        mr.send_angles([45.94, 25.33, 0.15, -102.31, -58.65, 55.8, -13.94], 10)
        waitl()
        ml.set_gripper_value(0, 10)
        mr.set_gripper_value(0, 10)
        time.sleep(12)
        detect_objects("left", ml_camera_pos_b, camera_data_b)
        detect_objects("right", mr_camera_pos_b, camera_data_b)
        ml.set_gripper_value(50, 10)
        mr.set_gripper_value(50, 10)
        cube_data, tray_data = judge_tray_cube(camera_data_b)      return cube_data, tray_data

In this scenario, each object on Desk A is tagged with an STag makers, allowing for precise acquisition of the object's coordinate parameters. The coordinates obtained from the STag are in a different coordinate system from those of the robot's mechanical arm. Consequently, a global coordinate system is essential within the robotic vision algorithms of the mechanical arm. This system converts both the object's coordinates and the coordinates at the end of the mechanical arm into a common world coordinate system.

This conversion facilitates the mechanical arm in accurately grasping the objects, as it can precisely locate and align its movements with the targeted object's position. The integration of STag makers into the robotic system greatly enhances the efficiency and accuracy of object manipulation, allowing the robot to perform complex picking tasks in a structured and predictable manner.

3. Moving to Desk B

After successfully grasping the object at Desk A, the Mercury X1 holds a storage box and moves towards Desk B to perform sorting tasks.

4. Object Sorting

Similar to the process at Desk A, the method of object recognition and grasping at Desk B follows the same algorithm. The objects are identified through the STag marker codes. Once identified, the precise coordinates of each object are obtained, which are used for sorting.

Technical Challenges in Development

The development of dual-arm humanoid robots, such as the Mercury X1, presents complex challenges, particularly in the coordination of the two arms. These challenges can be analyzed from the following perspectives:

1. Precise Coordination and Timing Synchronization:

The primary challenge is to calculate accurately the movement trajectories of both arms to ensure they simultaneously reach a predetermined position when grasping a tray. This requires the system to synchronously control two independent drive systems to ensure coordinated and synchronized movements. When the Mercury X1 grips the tray, it must clamp the edges simultaneously and lift the tray at the same time. To ensure synchronization and prevent one arm from lifting before the other, a 1-second waiting period is preset.

def put_up_tray(tray_data):
    ml.set_gripper_value(60, 10)
    mr.set_gripper_value(60, 10)
    time.sleep(1)
    for tray in tray_data:
        offset, fact_bcl = get_obj_coords(tray["corners"], tray["ids"], tray["cur_coords"], tray['arm'])
        point = fact_bcl.copy()
        px, py, pz = point[0], point[1], point[2]
        if tray["ids"] == [4] and tray['arm'] == "left":
            af = (offset + 90) * pi / 180
            mc = ml
            Hei = 57
            Len = -20
        elif tray["ids"] == [5] and tray['arm'] == "right":
            af = (offset - 90) * pi / 180
            mc = mr
            Hei = 57
            Len = 30
        else:
            continue
        # p3
        px3 = px + Hei * cos(af) + Len * sin(af)
        py3 = py + Hei * sin(af) - Len * cos(af)
        pz3 = pz
        p3 = [px3, py3, pz3]
        mc.send_base_coord(6, offset, sp)
        mc.send_base_coord(2, py3, sp)
        mc.send_base_coord(1, px3, sp)
        mc.send_base_coord(3, pz - 10, sp)
    waitl()
    ml.set_gripper_value(0, 10)
    mr.set_gripper_value(0, 10)
    time.sleep(2)
    current_coordsl = ml.get_base_coords()
    current_coordsr = mr.get_base_coords()
    waitl()
    ml.send_base_coord(3, current_coordsl[2] + 20, 20)
    mr.send_base_coord(3, current_coordsr[2] + 20, 20)
    waitl()

2. Mechanical Balance and Stability:

Coordination between the arms becomes especially crucial when grasping large or heavy objects. It is vital that the robot remains stable during these operations to avoid tipping over or damage due to mechanical imbalance. This involves complex mechanical calculations and real-time feedback adjustments. For the current project, the actions are not overly complex; simply lifting and lowering the tray simultaneously ensures the tray’s stability. However, in more complex scenarios, such as holding a table tennis racket in a way that ensures the ball does not fall off, considerations of mechanical balance and stability become much more significant.

3. Collision Avoidance:

Preventing collisions between the arms, between the arms and the robot’s body, and between the arms and the environment poses a significant challenge in dual-arm operations. It is essential to precisely plan the movement path of each arm while maintaining a safe distance from each other and the surrounding environment. This requires detailed spatial awareness and careful programming to avoid any potential impacts that could disrupt the task or damage the robot.

Summary

This case study not only highlights the potential practical applications of robotic technology in the industrial and service sectors but also underscores the importance of technological integration. By detailing the technical background, implementation steps, and specific applications, we observe the significant value of the Mercury X1 in enhancing production efficiency, reducing labor costs, and handling complex operational environments.

Your support through likes and comments is greatly appreciated and motivates us to provide further updates and insights. If you are interested in this case study, please show your support by engaging with us. Your encouragement is our driving force for continuous improvement and sharing valuable insights!