Real-Time 6D Pose Generation & Grasp Planning Toolkit

This paper introduces GraspGen, a lightweight and real-time 6D pose generation & grasp planning toolkit.

It combines Principal Component Analysis (PCA) with TensorRT-accelerated YOLOv8-Seg inference to enable fast object segmentation, pose estimation, and grasp planning on both desktop PCs and Jetson embedded platforms.

GraspGen integrates with the PiPER robotic arm and supports full forward/inverse kinematics, pose control, gripper control, and more.

Key Features

• TensorRT-accelerated YOLOv8-Seg inference for real-time operation
• PCA-based 6D pose generation
• Forward & inverse kinematics support
• Real-time pose control & gripper control
• Modular ROS interface
• Runs on both Desktop and Jetson platforms
• Compatible with PiPER robotic arm ecosystem

Demonstration

Tag

PiPER Robotic Arm, YoloV8-Seg Model, TensorRT, Forward and Inverse Kinematics, Pose Control, Gripper Control

Repositories

• GraspGen:https://github.com/vanstrong12138/GraspGen
• AgileX-College:https://github.com/agilexrobotics/Agilex-College/tree/master

Environment Configuration

Supports dual-platform deployment: Desktop Platform and Jetson Platform

Desktop Platform

• Ubuntu20.04
• ROS noetic
• NVIDIA Driver Version 470.256.02
• CUDA Version 11.4
• cuDNN 8.4.1
• TensorRT 8.6.1.6
• OpenCV 4.5.3

1. Install NVIDIA Graphics Driver

sudo apt update
sudo apt upgrade
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-driver-470
#reboot the system
reboot

2. Install CUDA Toolkit 11.4

First, download the CUDA runfile from the NVIDIA official websit

• Execute the installation command

wget https://developer.download.nvidia.com/compute/cuda/11.4.0/local_installers/cuda_11.4.0_470.42.01_linux.run
sudo sh cuda_11.4.0_470.42.01_linux.run

• Uncheck the first option “Driver” during installation, as we have already installed the graphics driver in Step 1

3. Add Environment Variables

echo 'export PATH=/usr/local/cuda-11.4/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-11.4/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

4. Verify CUDA Installation, and run the following command to check CUDA information:

nvcc -V

5. Install cuDNN

Download the TensorRT tar file from the NVIDIA official website, extract it, and copy the file

• After extraction, execute the following commands to copy cuDNN to the CUDA installation directory

sudo cp cuda/include/cudnn*.h /usr/local/cuda/include
sudo cp cuda/lib/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*

6. Install TensorRT， Download the TensorRT tar file from the NVIDIA official website, extract it, and copy the files

• After extraction, execute the following commands to move TensorRT to the /usr/local directory

#Extract the file

tar -xvf TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz 

#Enter the extracted directory
cd TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-11.8/

#Copy to /usr/local
sudo mv TensorRT-8.6.1.6/ /usr/local/

• Verify TensorRT Installation

#Navigate to the MNIST sample directory
cd /usr/local/TensorRT-8.6.1.6/samples/sampleOnnxMNIST

#Complie the sample
make

#Navigate to the bin directory and run the executable
cd /usr/local/TensorRT-8.6.1.6/bin
./sample_onnx_mnist

7. Install OpenCV with CUDA Acceleration

Download OpenCV 4.5.3 from the official website, extract it, and enter the directory

• Install OpenCV dependencies

sudo apt install build-essential cmake git pkg-config libgtk-3-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libjpeg-dev libpng-dev libtiff-dev gfortran openexr libatlas-base-dev python3-dev python3-numpy libtbb2 libtbb-dev libdc1394-22-dev

• Download opencv_contrib from GitHub and place it in the OpenCV directory (or another directory; modify the CMake path accordingly)
• Pre-compile with CMake (note: update OPENCV_EXTRA_MODULES_PATH to your opencv_contrib modules path)

cmake \
-D WITH_CUDA=ON \
-D CUDA_ARCH_BIN="8.6" \
-D WITH_CUDNN=ON \
-D OPENCV_DNN_CUDA=ON  \
-D cuDNN_VERSION='8.4' \
-D cuDNN_INCLUDE_DIR='/usr/local/cuda/include' \
-D CUDA_ARCH_PTX="" \
-D OPENCV_EXTRA_MODULES_PATH=/home/khalillee/opencv-4.5.3/opencv_contrib-4.5.3/modules \
-D WITH_GSTREAMER=ON \
-D WITH_LIBV4L=ON  \
-D BUILD_opencv_python3=ON \
-D BUILD_TESTS=OFF \
-D BUILD_PERF_TESTS=OFF \
-D BUILD_EXAMPLES=OFF \
-D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D ENABLE_FAST_MATH=1 \
-D CUDA_FAST_MATH=1 \
-D WITH_CUBLAS=1 \
-D PYTHON3_EXECUTABLE=$(which python3) \
-D PYTHON3_INCLUDE_DIR=$(python3 -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") \
-D PYTHON3_PACKAGES_PATH=$(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") \
-D OPENCV_ENABLE_NONFREE=ON \
-D PYTHON3_NUMPY_INCLUDE_DIRS=/usr/lib/python3/dist-packages/numpy/core/include \
-D PYTHON3_PACKAGES_PATH=/usr/lib/python3/dist-packages \
-D PYTHON_DEFAULT_EXECUTABLE=$(which python3) \
-D OPENCV_PYTHON3_INSTALL_PATH=/usr/lib/python3/dist-packages \
-D BUILD_opencv_python3=ON \
-D BUILD_opencv_python2=OFF \
-D HAVE_opencv_python3=ON \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D INSTALL_C_EXAMPLES=ON \
-D BUILD_opencv_python3=ON \
-D OPENCV_GENERATE_PKGCONFIG=YES \
..

• Compile and install (this may take time; be patient)

#  Use all CPU cores for compilation
make -j$(nproc) 

#Install
sudo make install

• Add OpenCV libraries to the system environment

sudo gedit /etc/ld.so.conf.d/opencv4.conf

• A blank file will open; add /usr/local/lib at the end of the file
• Update the system dynamic linker cache

sudo ldconfig

• Configure environment variables (edit /etc/bash.bashrc)

sudo gedit /etc/bash.bashrc

#Add the following at the end of the file
PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig 
export PKG_CONFIG_PATH 
#Save and exit

• Activate the environment variables and verify OpenCV version

source /etc/bash.bashrc
pkg-config --modversion opencv4

8. Install Dependent ROS Packages

sudo apt install -y ros-noetic-sensor-msgs ros-noetic-image-transport ros-noetic-cv-bridge ros-noetic-vision-msgs ros-noetic-image-geometry ros-noetic-pcl-conversions ros-noetic-pcl-ros ros-noetic-message-filters

Jetson Orin

• Ubuntu20.04
• ROS noetic
• JetPack 5.1.5
• CUDA 11.4.315
• cuDNN 8.6.0.166
• TensorRT 8.5.2.2

1. DownloadSDKManagerJetPack 5.1.5

2. Choose to flash only the system or all components (if only the system is flashed, you can install dependencies manually later)

3. If only the system is installed, install dependencies manually; skip if all components are installed via SDKManager!

sudo apt update && sudo apt upgrade

4. Install CUDA and Configure Environment Variables

sudo apt-get install cuda-toolkit-11-4
# Set environment variables
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.4/lib64
export PATH=$PATH:/usr/local/cuda-11.4/bin
export CUDA_HOME=$CUDA_HOME:/usr/local/cuda-11.4

5. Install cuDNN

sudo apt-cache policy libcudnn8

6. Install TensorRT

sudo apt-cache policy tensorrt

7. Install Dependent ROS Packages

sudo apt install -y ros-noetic-sensor-msgs ros-noetic-image-transport ros-noetic-cv-bridge ros-noetic-vision-msgs ros-noetic-image-geometry ros-noetic-pcl-conversions ros-noetic-pcl-ros ros-noetic-message-filters

Clone and Compile

• Clone the project to your local workspace

cd your_ws

git clone https://github.com/AgilexRobotics/GraspGen.git

catkin_make

Run Instructions

1. Launch Piper_kinematics Inverse Kinematics Node

# Launch Piper_kinematics inverse Kinematics Node
rosrun piper_kinematics piper_ik_node

# Launch interactive pose marker
rosrun piper_kinematics interactive_pose_marker.py

• Or launch both nodes with one command

roslaunch piper_kinematics piper_ik.launch

2. Launch Piper Driver Node and Drag the Interactive Marker to Target Position

cd piper_ros
# Activate CAN interface
./can_activate.sh

#Launch robotic arm driver node
roslaunch piper start_single_piper_ik_rviz.launch

• Drag the interactive marker to the target position to drive the robotic arm to move there

3. Launch GraspGen Node

#Launch RealSense camera node
roslaunch realsense2_camera rs_aligned_depth.launch 

#Launch GraspGen node
roslaunch tensorrtx_yolov8_ros yolov8.launch

4. Place an Object in Front of the Camera to See Detection Results

5. Add Visualization Plugin to View AABB Bounding Box and Grasp Pose

After adding the visualization plugin, you can see the automatically generated AABB bounding box of the object and the grasping pose optimized via Principal Component Analysis

6. Execute Grasp Task

• Press the lowercase “s” key in the GraspGen node terminal to start the grasping task