User Innovations in Face Tracking with myCobot＆OAK-D Camera

This article is edited and reposted with permission from the author Yuichiro Hirano. Original article link: https://www.techlife-hacking.com/?p=956#

Introduction

With the advent of YouTube and Netflix, we have started watching smartphones while lying down. However, holding the smartphone by hand for a long time can be tiring. This time, we made a moving smartphone arm that adjusts its position to maintain a proper distance right in front of you, so you don't have to hold the phone by hand. Be sure to try it!

Materials Prepared

myCobot 280 Pi

This time, we used a robotic arm from Elephant Robotics. Compared to other products, it is inexpensive and relatively easy for beginners to handle.

OAK-D Camera

For the camera to track the face, we used the OAK-D OpenCV DepthAI camera. It not only captures video but also assists in the neural network computations needed for facial recognition. As a result, even if the microcontroller doesn't have a GPU, it can perform facial recognition at high speed.

Project

It is a smartphone arm that constantly adjusts the display to a proper distance right in front of you. It mainly consists of an OAK-D camera and the "myCobot" robotic arm. The OAK-D camera can not only capture video but also obtain depth information, allowing it to calculate the distance from the camera to the face. The myCobot is a robotic arm with six axes of rotation, capable of performing various movements. Using the 3D position of the face derived from the video captured by the OAK-D camera, the myCobot delivers the smartphone display right in front of you. This way, you can enjoy videos without holding the smartphone by hand.

Attaching the OAK-D Camera and Smartphone to myCobot

The myCobot has four M2.6 screw holes at its end. We 3D printed a case for the smartphone and used these screw holes to attach the case to the myCobot, securing the smartphone in place.

On the other hand, the OAK-D camera has a 1/4-inch screw hole. We also made a hole for the 1/4-inch screw in the 3D printed case to secure the camera.

As a reference, here are the STL files used for this project.

https://www.thingiverse.com/thing:5152423

Movement of the Robotic Arm

Movement in the X direction: Rotation of the J1 axis

Movement in the Y direction: Rotation of the J4 axis

Movement in the Z direction: Rotation of the J2 and J3 axes (J2 and J3 rotate in opposite directions)

The J2 and J3 axes are used for movement in the depth direction. Moving only the J2 affects the Y direction, so the J3 is rotated the same amount in the opposite direction to the J2 axis to minimize this effect.

3D Face Tracking

Face Tracking on the XY Plane

By performing face detection on the images obtained from the OAK-D camera, the coordinates (x, y) of the face in the camera frame can be obtained.

The center coordinates of the OAK-D camera frame are set as the target value, and the face coordinates (x, y) obtained through face recognition are used as feedback values for PID control.

Face Tracking in the Depth Direction (Z direction)

The OAK-D camera is equipped with stereo cameras, allowing it to obtain the face coordinates in the depth direction (z) as well as the plane. The optimal distance where the face is neither too close nor too far from the display is set as the target value, and the face coordinates (z) measured by the stereo camera are used as feedback values for PID control.

myCobot and OAK Camera

Connect the OAK camera to the Raspberry Pi that comes with myCobot via USB. The OAK camera determines the target coordinates of the face, and the Raspberry Pi that comes with myCobot performs PID control based on these coordinates to adjust the direction of the camera.

Setup

We will set up the environment for the Raspberry Pi that comes with myCobot.

myCobot

In the Raspberry Pi version of myCobot, it is ready to use as soon as it is powered on. The robotic arm can be operated using Python, and it is officially supported.

# test
from pymycobot.mycobot import MyCobot
 
mycobot = MyCobot('/dev/ttyUSB0')
# go zero
mycobot.send_angles([0,0,0,0,0,0], 80)

OAK-D OpenCV DepthAI Camera

Install the depthai library to operate the OAK-D camera.

# install dependency
sudo curl -fL http://docs.luxonis.com/_static/install_dependencies.sh | bash
 
# get sources
git clone https://github.com/luxonis/depthai.git
 
# install depthai
python3 install_requirements.py

Demo

Once the setup is complete, please run the demo. If the camera tracks the face while maintaining a certain distance, it is working correctly.

# get demo sources
git clone https://github.com/tech-life-hacking/depthai.git
 
# execute demo
python3 depthai_demo.py

PID Adjustment

If the movement of myCobot is unstable, try adjusting the PID values.

# settings
PID_control.PID(P value, I value, D value)
pidX.setTargetPosition(Position of the point in the frame (X direction): Range 0-1, 0.5 is the center)
pidY.setTargetPosition(Position of the point in the frame (Y direction): Range 0-1, 0.5 is the center)
pidZ.setTargetPosition(Distance between the camera and the face (meters), 0.5m = 50cm)

# default
pidX = PID_control.PID(10, 10, 3.75)
pidY = PID_control.PID(6.5, 5, 2.5)
pidZ = PID_control.PID(50, 30, 20)
pidX.setTargetPosition(0.5)
pidY.setTargetPosition(0.5)
pidZ.setTargetPosition(0.5)

Setting Target Values

The target values for where myCobot should point the camera are determined by the following code. nnData[0] represents the coordinates of the four corners of the bounding box that the OAK-D camera detected around the face. Dividing the sum of these four corner coordinates by 2 gives the center point of the bounding box. spatialCoordinates.z is a method that returns the measurement result of the distance between the camera and the face.

x = (self._nnData[0].xmin + self._nnData[0].xmax) / 2
y = (self._nnData[0].ymin + self._nnData[0].ymax) / 2
z = int(self._nnData[0].spatialCoordinates.z) / 1000

Summary

This time, we performed face tracking using the OAK-D camera for face recognition and a robotic arm capable of complex movements. By capturing human movements with computer vision and operating the robotic arm accordingly, you can see that it can perform a wide variety of actions. I hope this will be a useful reference for your development.