Teleoperating the AgileX NERO Arm with Pika Sense

0. Preface

Pika Sense provides an intuitive teleoperation interface for robotic manipulation and data collection. Combined with Pika Station, Pika Gripper, and the AgileX NERO robotic arm, it enables low-latency end-effector teleoperation for both single-arm and dual-arm systems.

This tutorial walks through the complete setup process, and serves as a reference for developers who want to integrate Pika Sense with their own robotic arms.

⚠ warning: Pika Sense currently supports:

• AgileX PiPER
• AgileX PiPER-X
• AgileX NERO
• xArm Lite 6

Support for additional robotic arms can be added by implementing the corresponding teleoperation interface. We welcome community contributions. If you successfully adapt another robotic arm, feel free to submit a Pull Request to our GitHub repository.

⚠ warning: Pika Sense currently supports:AgileX PiPERAgileX PiPER-XAgileX NEROxArm Lite 6Support for additional robotic arms can be added by implementing the corresponding teleoperation interface. We welcome community contributions. If you successfully adapt another robotic arm, feel free to submit a Pull Request to our GitHub repository.

1.Hardware Preparation

This section describes the hardware setup required for teleoperating a single AgileX NERO robotic arm with its default NERO gripper.

⚠ Important Notes

• When assembling the robotic arm, align the red alignment marks on the arm connectors with the corresponding red marks on the cables.
• The textured sleeve on the aviation connector is the locking mechanism. During installation, align the red mark downward with the locating notch and push the connector straight in. To disconnect it, press the textured sleeve and then pull the connector out.
• If you are using a third-party robotic arm, you can follow the same hardware connection procedure described for the NERO arm.
• For third-party robotic arms, you must additionally verify that the power supply, communication interfaces, end-effector interfaces, and control protocols meet the requirements for teleoperation.

⚠ Important NotesWhen assembling the robotic arm, align the red alignment marks on the arm connectors with the corresponding red marks on the cables.The textured sleeve on the aviation connector is the locking mechanism. During installation, align the red mark downward with the locating notch and push the connector straight in. To disconnect it, press the textured sleeve and then pull the connector out.If you are using a third-party robotic arm, you can follow the same hardware connection procedure described for the NERO arm.For third-party robotic arms, you must additionally verify that the power supply, communication interfaces, end-effector interfaces, and control protocols meet the requirements for teleoperation.

Step 1 – Connect the NERO Robotic Arm

Connect the NERO robotic arm as shown in the wiring diagram below. For detailed wiring instructions, please refer to the NERO User Manual.

When powering on the robot for the first time, complete the following steps in order (A–G):

• Connect connector A to the J2 port.
• Connect the CAN cable of aviation connector B.
• Connect the XT30 connector C.
• Align the red dots on aviation connector D, then plug it in with the red dot facing downward.
• Connect the plug of power adapter E.
• Verify that the AC power cord of power adapter E is properly connected, then power on the system. Wait until the indicator LED on the control panel starts flashing green.
• Connect the USB cable to your computer to begin using the device.

Step 2 - Connect Pika Sense

Next, connect the Pika Sense device.

For detailed information about the Pika Sense interfaces and wiring, refer to Section 1.1 of the Pika Positioner & Sense User Guide (Positioning and Calibration).

2.SoftWare Preparation

Step 1.Install System Dependencies

Before running the teleoperation software, install the required system packages:

sudo apt update && sudo apt install ethtool
sudo apt update && sudo apt install can-utils

Step 2.Install Conda

Install either Miniconda or Anaconda before proceeding.

If Conda is already installed, you can skip this step.

Step 3.Create the Pika Teleoperation Python Environment

Create a dedicated Conda environment for Pika teleoperation:

conda create -n pika python=3.10
conda activate pika
conda install pinocchio==3.2.0 casadi==3.6.7 -c conda-forge
pip install lark numpy==2.0.2 empy==3.3.4 meshcat pyyaml piper-sdk opencv-python ur-rtde netifaces catkin_pkg

3.Starting Teleoperation

3.1 Single-Arm Teleoperation

Step 1 – Activate the CAN Interface

Connect the robotic arm's CAN cable to your computer, then activate the CAN interface by running:

cd ~/pika_ros/src/PikaAnyArm/agx_arm/agx_arm_ros/scripts

bash can_activate.sh

Step 2 – Calibrate Pika Sense

Before starting teleoperation, calibrate the Pika Sense device.

For detailed calibration instructions, please refer to Sections 1.2 and 1.3 of the Pika Positioner & Sense User Guide (Positioning and Calibration).

Step 3 – Launch the Teleoperation Program

Choose the appropriate launch procedure according to your hardware configuration.

Option A – Using Pika Sense without the Pika Gripper

If you are not using a Pika Gripper, launch the following programs.

Terminal 1

source ~/pika_ros/install/setup.bash

cd ~/pika_ros/scripts

bash start_single_sensor_whit_teleop.bash

Terminal 2

source ~/pika_ros/install/setup.bash

conda activate pika

# Launch the appropriate teleoperation node for your robot:

# piper
ros2 launch pika_remote_agx_arm teleop_single_piper.launch.py
# piper_x
ros2 launch pika_remote_agx_arm teleop_single_piper_x.launch.py
# nero
ros2 launch pika_remote_agx_arm teleop_single_nero.launch.py

After both terminals are running, double click the Pika Sense trigger to enable teleoperation.

⚠ Important Notes Ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector before enabling teleoperation.

⚠ Important Notes Ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector before enabling teleoperation.

Option B - Using Pika Sense with Pika Gripper

If a Pika Gripper is mounted on the robotic arm and controlled by Pika Sense, first bind one Pika Sense with one Pika Gripper by following the instructions in the Pika Positioner & Sense User Guide (Positioning and Calibration).

Terminal 1

conda deactivate

source ~/pika_ros/install/setup.bash

cd ~/pika_ros/scripts

bash start_sensor_gripper.bash

Terminal 2

source ~/pika_ros/install/setup.bash
conda activate pika

# Launch the appropriate teleoperation node for your robot:
# piper
ros2 launch pika_remote_agx_arm teleop_single_piper.launch.py
# piper_x
ros2 launch pika_remote_agx_arm teleop_single_piper_x.launch.py
# nero
ros2 launch pika_remote_agx_arm teleop_single_nero.launch.py

Once the system is running, double click the Pika Sense trigger to start teleoperation.

⚠ Important Notes Ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector before enabling teleoperation.

⚠ Important Notes Ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector before enabling teleoperation.

3.2 Dual-Arm Teleoperation

Step 1 – Configure the CAN Interfaces

Before starting dual-arm teleoperation, both robotic arms must be assigned to the correct CAN interfaces.

First, connect the left robotic arm to your computer via the CAN adapter, then run:

cd ~/pika_ros/src/PikaAnyArm/agx_arm/agx_arm_ros/scripts
bash find_all_can_port.sh

The terminal will display the USB port corresponding to the left arm.

Next, connect the right robotic arm and run the script again:

bash find_all_can_port.sh

The terminal will display the USB port corresponding to the right arm.

Open the can_config.sh file and copy the two detected USB port IDs into Lines 111 and 112, respectively.

For example, when using PiPER, the configuration should look like:

if [ "$EXPECTED_CAN_COUNT" -ne 1 ]; then
    declare -A USB_PORTS
    USB_PORTS["1-8.1:1.0"]="can_left:1000000"   # Left arm
    USB_PORTS["1-8.2:1.0"]="can_right:1000000"  # Right arm
fi

After saving the configuration, activate both CAN interfaces:

cd ~/pika_ros/src/PikaAnyArm/agx_arm/agx_arm_ros/scripts

bash can_config.sh

Step 2 – Calibrate Pika Sense and Bind Pika Devices

Calibrate the Pika devices before starting teleoperation. For detailed instructions, refer to the Pika User Manual, including the the Pika Positioner & Sense User Guide (Positioning and Calibration) section for base station deployment and Pika Sense calibration, followed by the Pika Device Binding section to bind the left and right Pika Sense devices.

Step 3 – Launch the Dual-Arm Teleoperation Program

Choose the launch procedure according to your hardware configuration.

Option A – Using Pika Sense without the Pika Gripper

If you are not using Pika Grippers, or you are using the robot's original grippers, first bind two Pika Sense devices following the Pika Device Binding Guide.

Terminal 1

conda deactivate
source ~/pika_ros/install/setup.bash
cd ~/pika_ros/scripts && bash start_multi_sensor_whit_teleop.bash sensor

Terminal 2

source ~/pika_ros/install/setup.bash
conda activate pika

# Launch the appropriate teleoperation node for your robot:
# piper
ros2 launch pika_remote_agx_arm teleop_double_piper.launch.py
# piper_x
ros2 launch pika_remote_agx_arm teleop_double_piper_x.launch.py
# nero
ros2 launch pika_remote_agx_arm teleop_double_nero.launch.py

Finally, double click the right Pika Sense trigger to start dual-arm teleoperation.

Important Note Ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector before enabling teleoperation.

Important Note Ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector before enabling teleoperation.

Option B – Using Pika Grippers

If Pika Grippers are installed on both robotic arms, first bind:

• Two Pika Sense devices
• Two Pika Grippers

using the Pika Device Binding section section 2.6, after that:

Terminal 1 —— Start Pika Sense

conda deactivate

source ~/pika_ros/install/setup.bash

cd ~/pika_ros/scripts

bash start_multi_sensor_whit_teleop.bash

Terminal 2 —— Start Pika Grippers

conda deactivate
source ~/pika_ros/install/setup.bash
cd ~/pika_ros/scripts && bash start_multi_gripper.bash #pika gripper

Terminal 3 —— Launch the Robot

source ~/pika_ros/install/setup.bash
conda activate pika

# Launch the appropriate teleoperation node for your robot:
# piper
ros2 launch pika_remote_agx_arm teleop_double_piper.launch.py
# piper_x
ros2 launch pika_remote_agx_arm teleop_double_piper_x.launch.py
# nero
ros2 launch pika_remote_agx_arm teleop_double_nero.launch.py

Finally, double click the right Pika Sense trigger to start teleoperation.

Important Note Ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector before enabling teleoperation.

Important Note Ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector before enabling teleoperation.

4.Configuration Files

The configuration files for each supported robot are organized as follows:

config/
├── piper_rand_params.yaml
│   └── gripper_xyzrpy
│       Offset from Joint 6 to the gripper frame (m, rad)
│
├── xarm_params.yaml
│   ├── eff_position
│   │   Initial TCP pose (mm, rad)
│   └── pika_to_arm
│       Pika coordinate system → Robotic arm end-effector coordinate system (m, rad)
│
└── ur12e_params.yaml
    ├── pika_to_arm
    │   Pika coordinate system → Robotic arm end-effector coordinate system (m, rad)
    └── robot_ip
        IP address of UR arm

5. Coordinate Frames and ROS Topics

To help developers integrate Pika Sense with their own robotic arms, this section describes:

• The Pika gripper coordinate frame
• The /pika_pose ROS topics

5.1 Pika Coordinate Frame

The Pika coordinate frame is defined at the center of the Pika gripper and is published through the /pika_pose topic.

When teleoperation is activated by double-clicking the Pika Sense gripper, the current pose of the Pika Sense is recorded as the reference (zero) pose. All subsequent position and orientation commands are expressed relative to this reference frame.

The coordinate axes of the /pika_pose frame are defined as follows:

• X-axis: Forward
• Y-axis: Left
• Z-axis: Up

5.2 ROS Topics

For single-arm teleoperation, Pika Sense publishes the robot target pose to: /pika_pose For dual-arm teleoperation, the left and right Pika Sense devices publish to separate topics: /pika_pose_l/pika_pose_r All of these topics use the standard ROS message type: geometry_msgs/PoseStamped Since most industrial robot controllers provide Cartesian end-effector control interfaces based on geometry_msgs/PoseStamped, the published pose messages can typically be used directly or converted into robot-specific motion commands with minimal adaptation.

For a complete implementation example, see:

• teleop_xarm.py

FAQ

Q1 Which robotic arms are currently supported by Pika Sense?

Pika Sense currently supports teleoperation for the following robotic arms:

• AgileX PiPER
• AgileX PiPER-X
• AgileX NERO
• xArm Lite 6

Support for additional third-party robotic arms can be implemented by developers or users based on the robot's control interface.

Q2. Can I use Pika Sense with a third-party robotic arm?

Yes.

To integrate a third-party robotic arm, the robot should support Cartesian end-effector pose control, or provide an interface that can convert the geometry_msgs/PoseStamped messages published by Pika Sense into robot-specific motion commands.

Q3.Why do I need to double-click the Pika Sense gripper before starting teleoperation?

Double-clicking initializes the teleoperation reference pose.

The current pose of the Pika Sense is recorded as the zero (reference) pose, and all subsequent position and orientation commands are calculated relative to this reference frame.

Q4.Is there anything I should pay attention to when double-clicking the Pika Sense gripper?

Yes.

Before double-clicking, ensure that the orientation of the Pika Sense matches the orientation of the robot end-effector.

Doing so minimizes the initial pose offset and helps ensure that the robot moves intuitively in the same direction as the Pika Sense.

Q5. Which parameters should I pay attention to when integrating a third-party robotic arm?

The most important parameters are:

• The coordinate frame of the robot end-effector (TCP)
• The transformation between the Pika gripper center frame and the robot TCP frame
• The robot controller's required pose or motion command format

In most cases, the pika_to_arm parameter in the configuration file defines the transformation between the Pika coordinate frame and the robot TCP, making it one of the key parameters to adjust when integrating a new robotic arm.