YOLOv8 & myCobot: Mimicking Human Movement

This article is reprinted from Suzuki Sumiya at SWITCH SCIENCE, and the reprinting of this article has been authorized by the author.

Link from:https://www.switch-science.com/blogs/magazine/jetson-maker-faire-tokyo-2023

1.Introduction

This article from Suzuki Sumiya at SWITCH SCIENCE demonstrates controlling a robotic arm by simulating human arm postures, as presented at Maker Faire Tokyo 2023. The demonstration involved using a USB camera for image recognition to estimate the position of a human arm, and then replicating this with a mycobot in a similar arm shape.

The project primarily utilizes Ultralytics YOLOv8 for posture estimation, integrating machine vision with robotic arm control. In this article, we will rewrite this project, detailing step-by-step how to set up this project.

2.Technical Introduction

Introduction to YOLOv8

Ultralytics YOLOv8 is a cutting-edge, state-of-the-art (SOTA) model that builds upon the success of previous YOLO versions and introduces new features and improvements to further boost performance and flexibility. YOLOv8 is designed to be fast, accurate, and easy to use, making it an excellent choice for a wide range of object detection and tracking, instance segmentation, image classification and pose estimation tasks.

As the latest version in the series, YOLOv8 includes the following features and improvements:

1. Faster Detection Speed: The YOLO series is renowned for its rapid detection speed, with each subsequent version optimized for this aspect.

2. Higher Accuracy: YOLOv8 may increase the accuracy of object detection by using more advanced neural network architectures and learning algorithms.

3. Improved Generalization: Enhanced algorithms may handle different types of image data more effectively, including detection in complex backgrounds and under varying lighting conditions.

4. Enhanced Adaptability and Scalability: The new version might offer more customization options and settings, allowing it to better adapt to different application scenarios and needs.

5. Optimized Resource Usage: Improvements in computational efficiency might enable YOLOv8 to perform better on resource-constrained devices, such as in mobile or embedded systems.

In simpler terms, when given an image to analyze, YOLOv8 can quickly identify objects in the image and mark them accurately.

myCobot 280 M5

The myCobot 280 M5 is the smallest and lightest 6-axis collaborative robot developed in collaboration by Elephant Robotics and M5Stack. It features an integrated modular design, weighing only 850 grams, making it exceptionally lightweight. Equipped with six high-performance servo motors, it boasts characteristics such as quick response, low inertia, and smooth rotation.

The mycobot 280 supports multi-platform development across Linux, Windows, and MacOS, and its control interfaces are fully open-sourced. It supports mainstream programming languages currently popular in the market, such as Python, C++, and C#.

python-pymycobot mycobot's control library.

Jetson Orin Nano

The Jetson Orin Nano is an embedded artificial intelligence computing module from NVIDIA, characterized by high performance and low power consumption. It is based on NVIDIA's Orin chip, which includes 12 Arm cores and an NVIDIA Ampere GPU. This combination enables the Jetson Orin Nano to run complex artificial intelligence applications such as machine vision, natural language processing, and augmented reality.

3. Development Process

Development Environment:

Software:

● Operating System: Linux

● Programming Language: Python

● Python Libraries:

• `import cv2`
• `import math`
• `import time`
• `from ultralytics import YOLO`
• `from pymycobot.mycobot import MyCobot`

● YOLOv8 Model: `yolov8n-pose.pt`

Hardware:

● USB Camera *1

● myCobot 280 M5Stack *1

● Jetson Orin Nano *1

● Monitor, Keyboard, and Mouse *1

YOLOv8 Pose Estimation Model - yolov8n-pose

YOLOv8 comes with a pre-trained pose estimation model that can be used directly. Pose estimation is a task that involves identifying the positions of specific points in an image, often referred to as keypoints. These keypoints can represent various parts of an object, such as joints, landmarks, or other significant features. The positions of these keypoints are typically represented as a set of 2D [x, y] or 3D [x, y, visible] coordinates.

Simple to use code:

import torch
from ultralytics.yolov8 import YOLO
import cv2

#import model
model = YOLO('yolov8n-pose.pt')
#open pic
img = cv2.imread('image.jpg')
# Detect pictures
results = model(img)

# Show result
for r in results:
    im_array = r.plot()  # Detection plot draws BGR numpy array containing prediction results
    im = Image.fromarray(im_array[..., ::-1])  # RGB PIL
    im.show()  # show pic
    im.save('results.jpg')  # save result

In the project, it is necessary to obtain the x and y coordinates of the wrist, elbow, and shoulder of a human hand to calculate angles. These calculated angles are then transmitted to the second and third joints of the mycobot to replicate the same angles.

Code：

if keypoints_tensor is not None and keypoints_tensor.size(1) > 0:
# Extract coordinates and confidence of specific keypoints
    x_mimi = keypoints_tensor[0][3][0]
    y_mimi = keypoints_tensor[0][3][1]
    conf_mimi = confidence_score[0][3]

    x_kosi = keypoints_tensor[0][13][0]
    y_kosi = keypoints_tensor[0][13][1]
    conf_kosi = confidence_score[0][13]

    x_kata = keypoints_tensor[0][5][0]
    y_kata = keypoints_tensor[0][5][1]
    conf_kata = confidence_score[0][5]

    x_hizi = keypoints_tensor[0][7][0]
    y_hizi = keypoints_tensor[0][7][1]
    conf_hizi = confidence_score[0][7]

    x_te = keypoints_tensor[0][9][0]
    y_te = keypoints_tensor[0][9][1]
    conf_te = confidence_score[0][9]

Subsequently, the angle between the waist and the shoulder is calculated. This angle primarily focuses on a simplified model of the upper body, which may be used to simulate the movement of the shoulder or the tilt of the entire upper body.

# Calculate the vector between AB
vector_AB = (x_kata - x_hizi, y_kata - y_hizi)

# Calculate the angle of this vector using a function
angle_rad1 = math.atan2(vector_AB[1], vector_AB[0])

# Convert angle from radians to degrees
angle_deg1 = math.degrees(angle_rad1)

# The calculated and adjusted angle value is used to control the robotic arm
mycobot1 = int(angle_deg1)-90

Next, the angle formed by the three keypoints of the hand, elbow, and shoulder is calculated. This angle involves a more complex posture analysis because it includes the entire chain from the hand to the elbow and then to the shoulder. Combining the calculations of these two types of angles enables the robotic arm to imitate human postures more accurately and naturally.

x1=x_te # Hand x,y
y1=y_te

x2=x_hizi #Elbow X,Y
y2=y_hizi

x3=x_kata # Shoulder x，y
y3=y_kata

# Define three punctuation points
point1 = (x1, y1)
point2 = (x2, y2)
point3 = (x3, y3)

# Calculate vector
vector1 = (x2 - x1, y2 - y1)
vector2 = (x3 - x2, y3 - y2)

# Calculate vector length
length1 = math.sqrt(vector1[0] ** 2 + vector1[1] ** 2)
length2 = math.sqrt(vector2[0] ** 2 + vector2[1] ** 2)

# Calculate dot product
dot_product = vector1[0] * vector2[0] + vector1[1] * vector2[1]

# Calculate angle (radians)
angle_rad = math.atan2(vector2[1], vector2[0]) - math.atan2(vector1[1], vector1[0])

# If the angle is curved to the right then it is 0 degrees
# If the angle is rotated from right to left then it is 180 degrees
# If on a straight line, then -90 degrees (or +90 degrees, whichever)
if angle_rad > math.pi:
angle_rad -= 2 * math.pi
elif angle_rad < -math.pi:
angle_rad += 2 * math.pi

# Convert angle to degrees
mycobot2 = int(math.degrees(angle_rad))

Finally, the angles obtained are used to control the robotic arm through conditional judgments, enabling it to simulate arm movements.

#The acceptable ranges of the robot arm calculated for mycobot1 and mycobot2 are -180 to 180 degrees and -155 to 155 degrees respectively.
if -180 <= mycobot1 and mycobot1 <= 180 and -155 <= mycobot2 and mycobot2 <= 155 and conf_hizi >= 0.75:
    mc.send_angles ([90,-mycobot1,mycobot2,0,-90,0],100)
    mc.set_color(0, 0, 255)
    print("Angle of point A:", conf_hizi)
else:
    print("Angle of point A:", conf_hizi)
    mc.set_color(255, 0, 255)

4. Conclusion

Suzuki Sumiya demonstrated at Maker Faire Tokyo 2023 an intriguing demo that combined the Jetson Orin Nano, mycobot, and yolov8-pose. This project offers a learning opportunity in various areas, including computer vision, robotics programming, mathematical calculations, and how to integrate these elements to create an interactive system.

Below is a video featuring an interview with Suzuki Sumiya discussing this project.

If you have any other interesting projects, feel free to contact us. We would be happy to share your project across various platforms to garner widespread attention.