Rockchip (EASY EAI) RV1126B Person Detection Solution

1. Scheme Overview

This solution demonstrates how to connect two MIPI-CSI cameras to the 【Human Detection Algorithm】 for analysis and display the algorithm analysis results on the screen.

1.1 Wiring Schematic Diagram

Camera and board connection:

* Wiring work must be carried out when the power is cut off.

* Connect the IMX415 camera to the Camera1 interface with reversed wiring. (Note: Incorrect wiring will burn out the camera or the core board, so exercise extreme caution when making connections.)

* The clip of the terminal block and the blue plastic mark on the FPC wire must be on the same side, as shown in the figure below.

The EASY EAI Nano-TB is equipped with the IMX415 monocular camera by default. It also comes with a 40-pin reverse FPC cable with a 0.5 mm pitch.

reverse line：The blue marks at both ends are not on the same side.

co-directional line：The blue marks on both ends are on the same side.

2. Get Started Quickly

2.1 Preparation of Development Environment

If you are reading this document for the first time, please refer to《Getting Started/Development Environment Setup/Preparation and Update of the Easy-Eai Compilation Environment》，and follow the relevant instructions to deploy the compilation environment.

Execute the run script on a PC with the Ubuntu system to enter the EASY-EAI compilation environment, as shown below.

cd ~/develop_environment
./run.sh 2204

2.2 Source code download and routine compilation

Create a management directory to store the source code repository under the EASY-EAI compilation environment:

cd /opt
mkdir EASY-EAI-Toolkit
cd EASY-EAI-Toolkit

Clone the remote repository in the management directory using the git tool

git clone https://github.com/EASY-EAI/EASY-EAI-Toolkit-1126B.git

Note:

* The page may lag due to network issues. Please wait patiently.

* If you have to download from the GitHub webpage, you need to download the entire repository instead of only the directory corresponding to this example.

Go to the corresponding routine directory and perform compilation. The specific commands are as follows:

cd EASY-EAI-Toolkit-1126B/Solutions/detect-Person
./build.sh

Note:* Since the dependent libraries are deployed on the board, the /mnt mount must be maintained during cross-compilation.

2.3 Model Deployment

To ensure the development example runs smoothly, you also need to download the personnel detection algorithm model first.

The Baidu Netdisk link is：https://pan.baidu.com/s/1ds5ffznYnhtj1S6cXvjdVg?pwd=1234 (Extraction code: 1234).

Meanwhile, copy the downloaded face detection algorithm model to the Release/ directory:

Perform the compilation again（This time we need to include cpres, which means copying the resource files to the board as well.）：

./build.sh cpres

2.4 routine operation

Access the background of the board via serial port debugging or SSH debugging， and navigate to the directory where the routine is deployed, as shown below:

cd /userdata/Solu/detect-Person

The routine commands are as follows.

./detect-Person 23

The display effect on the screen is as follows:

3. Code Parsing

The main logical code of the solution is located at: EASY-EAI-Toolkit-1126B/Solution/detect-Person/src/main.cpp. The code mainly realizes the human detection function quickly by calling our company's easyeai-api library, and the main body of the code is divided into the main thread and the algorithm thread.

3.1 Composition of Component Library

To implement the personnel detection function, the following components of the easyeai-api library are required, as shown below.

The module information is as follows.

These components are compiled into the project via CMakeLists.txt; please refer to the subsequent chapters for details.

3.2 logic block diagram

The overall logical block diagram of the project is shown below.

3.3 main thread

The businesses processed by the main thread are as follows:- Initialize peripherals;- Create a sub-thread for algorithm analysis;- Capture images and send them to the sub-thread;- Capture and display images.

The main logical function code is attached here, while other auxiliary and verification codes are omitted temporarily.

The code for initializing the display screen is shown below.

// 1.初始化显示屏
int width, height;
if (disp_init(&width, &height) < 0) {
    fprintf(stderr, "DRM初始化失败\n");
    return -1;
}

The component initialization operation is as follows, and the mipi camera is called here.

// 2.打开摄像头
#define CAMERA_WIDTH    1920
#define CAMERA_HEIGHT   1080
#define IMGRATIO        3
#define IMAGE_SIZE      (CAMERA_WIDTH*CAMERA_HEIGHT*IMGRATIO)
int cameraIndex = atoi(argv[1]);
ret = mipicamera_init(cameraIndex, CAMERA_WIDTH, CAMERA_HEIGHT, 0);
if (ret) {
    printf("error: %s, %d\n", __func__, __LINE__);
    goto exit3;
}
pbuf = (char *)malloc(IMAGE_SIZE);
if (!pbuf) {
    printf("error: %s, %d\n", __func__, __LINE__);
    ret = -1;
    goto exit2;
}

Create thread mutex locks and threads, as shown below.

// 3.创建识别线程，以及图像互斥锁
pthread_mutex_init(&img_lock, NULL);
pResult = (Result_t *)malloc(sizeof(Result_t));
memset(pResult, 0, sizeof(Result_t));
if(0 != CreateNormalThread(detect_thread_entry, pResult, &mTid)){
	free(pResult);
}

Capture the image and invoke the clone operation.

// 4.1、取流
pthread_mutex_lock(&img_lock);
ret = mipicamera_getframe(cameraIndex, pbuf);
if(ret) {
    printf("error: %s, %d\n", __func__, __LINE__);
    pthread_mutex_unlock(&img_lock);
    continue;
}
algorithm_image = Mat(CAMERA_HEIGHT, CAMERA_WIDTH, CV_8UC3, pbuf);
image = algorithm_image.clone();
pthread_mutex_unlock(&img_lock);

Call the image display function and mark the analyzed target position with Result tags.

// 4.2、显示
char text[256];
for (int i = 0; i < Result.algoRes[0].resNumber; i++) {
    detect_result_t *det_result = &(Result.algoRes[0].detect_Group.results[i]);
    if( det_result->prop < 0.4) {
        continue;
    }

   // 标出识别目标框
   sprintf(text, "%s %.1f%%", det_result->name, det_result->prop * 100);
   printf("%s @ (%d %d %d %d) %f\n", det_result->name, det_result->box.left, det_result->box.top,
                   det_result->box.right, det_result->box.bottom, det_result->prop);
    int x1 = det_result->box.left;
    int y1 = det_result->box.top;
    int x2 = det_result->box.right;
    int y2 = det_result->box.bottom;
    // 标出识别目标定位标记
    /*
    rectangle(image, cv::Point(x1, y1), cv::Point(x2, y2), cv::Scalar(255, 0, 0, 255), 3);
    putText(image, text, cv::Point(x1, y1 + 12), cv::FONT_HERSHEY_SIMPLEX, 0.5, cv::Scalar(0, 0, 0));
    */
    plot_one_box(image, x1, x2, y1, y2, text, i%10);
}

disp_commit(image.data, CAMERA_WIDTH, CAMERA_HEIGHT, HAL_TRANSFORM_ROT_270);

3.4 Sub-thread for algorithm analysis

The sub-thread for algorithm analysis mainly completes the following operations:- Monitor whether the image buffer is empty with a delay;- If it is not empty, it means the main function has sent image data, and the thread performs the image acquisition operation;- Call the personnel analysis function;- Record the data of the target frame for subsequent image composition operations.

The delayed image monitoring operation is shown as follows.

if(algorithm_image.empty()) {
	usleep(5);
	continue;
}

The image acquisition operation is as follows.

pthread_mutex_lock(&img_lock);
image = algorithm_image.clone();
pthread_mutex_unlock(&img_lock);

Invoke the personnel detection function, and the target result obtained by the algorithm is recorded in the algoRes[0] member of pResult, as shown below.

// 算法分析
ret = person_detect_run(ctx, image, &pResult->algoRes[0].detect_Group);

4. Development Guide

4.1 Introduction to the Toolkit Repository

4.2 Sample Files & Directory Structure

4.2.1 Introduction to Development Examples

Each development instance is an independent project, which adopts CMake to realize automatic compilation and deployment. A minimal project must contain at least the following files: build.sh, CMakeLists.txt and main.c, as shown below.

The detailed introduction is as follows.

4.2.2 Expandable directory

An expandable directory means that certain functional modules and functional codes can be added during the development process. There are two types of addition modes:

• Add compiled third-party libraries by placing header files and library files in the include and libs directories.
• Add user-defined functional modules, which are recommended to be placed in the src directory.

The details are as follows. The files related to third-party modules are include/3rd_model/xxx.h and libs/3rd_model/xxx.a. The user-defined functional modules are src/mySrcCode and src/mySrcCode2.

4.3 Parsing of CMakeLists.txt File

4.3.1 Compilation Environment Configuration Section:

The first part is the configuration section, which is shown as follows. (Obtain the current solution directory, configure the toolchain, and extract the solution name):

The configuration information is as follows.

4.3.2 Configuration Section of easyeai-api

Part 2 involves importing our company's functional component library (configure the EASY EAI API header file directory ${XXX_INCLUDE_DIRS}, source code directory ${XXX_SOURCE_DIRS}, and dependent library parameters ${XXX_LIBS} for the current solution):

The configuration information is as follows.

4.3.3 Third-party library configuration section

Part 3 Configuring Third-Party Libraries (for the current solution: configure the third-party header file directories, library file directories, third-party library linking parameters, and source code directories):

The configuration information is as follows.

For example, the directory structure for adding a personal library is as shown below.

The modification method of aux_source_directory is:

aux_source_directory(./src ./src/mySrcCode ./src/mySrcCode2 dir_srcs)

or

aux_source_directory(./src dir_srcs)
aux_source_directory(./src/mySrcCode dir_srcs)
aux_source_directory(./src/mySrcCode2 dir_srcs)

4.3.4 Configuration Section of This Plan

Part Four configures the compilation information of the project, and the content is as follows:

The configuration items are as follows.

4.4 build.sh compilation script:

4.4.1 Path Location Section

The first part is used to extract the directory for compilation operations, with the content as follows: (enter the directory where the build.sh script is located, extract the absolute path of the current directory, and get the name of the current directory)

4.4.2 Clear the compilation section

Part 2: Cleanup operation. The directories to be cleared are build and Release. Details are as follows: (If the parameter "clear" is passed when executing the build.sh script, the compiled output will be cleared.)

4.4.3 compilation operation

Part III: Call cmake directly for compilation, with the details as follows: (recompile to generate the deployment directory and automatically deploy resources to the board)