Rockchip (EASY EAI) RV1126B Speech Recognition

Speech recognition

1. Introduction to Speech Recognition

Speech recognition technology, also known as Automatic Speech Recognition (ASR), aims to convert lexical content in human speech into computer-readable input, such as keystrokes, binary encoding, or character sequences. Unlike speaker recognition and speaker verification (which attempt to identify or verify the speaker who utters the speech rather than the lexical content contained within it), speech recognition focuses on the linguistic content of the speech.

Our speech algorithm is based on Whisper, a model designed by OpenAI. As a general-purpose speech recognition model, Whisper is trained on a large amount of multilingual and multitask supervised data, enabling it to achieve human-level robustness and accuracy in English speech recognition. Whisper can also perform tasks such as multilingual speech recognition, speech translation, and language identification. Whisper’s architecture is a simple end-to-end approach that adopts an encoder-decoder Transformer model, converting input audio into corresponding text sequences and specifying different tasks based on special tokens.

Operation efficiency based on the EASY-EAI-Nano-TB (RV1126B) hardware motherboard:

2. Quick Start

2.1 Development Environment Preparation

If you are reading this document for the first time, please refer to《Getting Started Guide/Development Environment Preparation/Easy-Eai Compilation Environment Preparation and Update》，and deploy the compilation environment according to the relevant operations.

Execute the run script in the PC-side Ubuntu system to enter the EASY-EAI compilation environment, as follows:

cd ~/develop_environment
./run.sh

2.2 Source Code Download

Create a management directory for storing the source code repository in the EASY-EAI compilation environment:

cd /opt
mkdir EASY-EAI-Toolkit
cd EASY-EAI-Toolkit

Clone the remote repository in the management directory using the Git tool:

git clone https://github.com/EASY-EAI/EASY-EAI-Toolkit-1126B.git

Notes:

• The process may get stuck due to network issues; please wait patiently.
• If you have to download from the GitHub webpage, download the entire repository instead of only the directory corresponding to this example.

2.3 Model Deployment

To run the algorithm Demo, you need to download the speech algorithm model first.

Baidu Netdisk link: https://pan.baidu.com/s/18okeW8ZIJ6zapTvuxcFoRw?pwd=1234 (Extraction Code: 1234).

Meanwhile, copy and paste the downloaded speech recognition algorithm model to the Release/ directory:

2.4 Routine Compilation

Enter the corresponding routine directory and execute the compilation command as follows:

cd EASY-EAI-Toolkit-1126B/Demos/algorithm-speech_recognition/
./build.sh cpres

Notes:

• Since the dependent libraries are deployed on the board, the /mnt mount must be maintained during the cross-compilation process.
• If the build.sh script is run with the cpres parameter, all resources in the Release/ directory will be copied to the development board.

2.5 Routine Execution and Effects

Enter the board background via serial port debugging or SSH debugging，and navigate to the routine deployment location:

cd /userdata/Demo/algorithm-speech_recognition/

Run the routine with the following command:

sudo ./test-speech_recognition speech_encoder.model speech_decoder.model filters.txt CN.txt cn 1-10-1_CN.wav

For detailed API descriptions and API calls (source code of this routine), see the instructions below.

3. Speech Recognition API Description

3.1 Reference Method

To facilitate customers to directly call our EASY EAI API library in local projects, the libraries, header files, etc. that need to be linked in the project are listed here for easy addition by users.

3.2 Speech Recognition Initialization Function

The prototype of the speech recognition initialization function is as follows:

int speech_recognition_init(const char *p_encoder_path, const char *p_decoder_path, const char *p_filter_path, 
		const char *p_vocab_path, rknn_whisper_t *p_whisper);

Detailed description:

3.3 Speech Recognition Running Function

The prototype of the speech recognition running function is as follows:

int speech_recognition_run(rknn_whisper_t *p_whisper, audio_buffer_t audio, int task_code, std::vector<std::string> &recognized_text);

Detailed description:

3.4 Speech Recognition Release Function

The prototype of the speech recognition release function is as follows:

int speech_recognition_release(rknn_whisper_t *p_whisper);

Detailed description:

4. Speech Recognition Algorithm Routine

The routine directory is Demos/algorithm-speech_recognition/test-speech_recognition.cpp, and the operation process is as follows:

#include <iostream>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <time.h>
#include "sndfile.h"
#include "speech_recognition.h"
#include "audio_utils.h"


int main(int argc, char **argv)
{
	if (argc != 7){
		printf("%s <encoder_path> <decoder_path> <filter_path> <vocab_path> <task(en/cn)> <audio_path>\n", argv[0]);
		printf("Example: %s speech_encoder.model speech_decoder.model filters.txt CN.txt cn 1-10-1_CN.wav\n", argv[0]);
		return -1;
	}
	const char *p_encoder_path = argv[1];	// 编码模型地址
	const char *p_decoder_path = argv[2];	// 解码模型地址
	const char *p_filter_path = argv[3];	// 滤波器频谱
	const char *p_vocab_path = argv[4];		// 词组文件
	const char *p_task = argv[5];			// 识别语种（cn/en）
	const char *p_audio_path = argv[6];		// 待识别音频
	int task_code = 0;
	std::vector<std::string> recognized_text;

	// Tokenizer 预定义控制符号(切换语言或任务)
	if (strcmp(p_task, "en") == 0){
		task_code = 50259;
	}
	else if (strcmp(p_task, "cn") == 0){
		task_code = 50260;
	}
	else{
		printf("\n\033[1;33mCurrently only English or Chinese recognition tasks are supported. Please specify <task> as en or zh\033[0m\n");
		return -1;
	}
	
	// 读取音频，并对音频进行处理
	audio_buffer_t audio;
	int ret = read_audio(p_audio_path, &audio);
	if (ret != 0){
		printf("read audio fail! ret=%d audio_path=%s\n", ret, p_audio_path);
		return -1;
	}
	if (audio.num_channels == 2){
		ret = convert_channels(&audio);
	}
	if (audio.sample_rate != SAMPLE_RATE){
		ret = resample_audio(&audio, audio.sample_rate, SAMPLE_RATE);
	}

	// speech recognition初始化
	rknn_whisper_t whisper;
	ret = speech_recognition_init(p_encoder_path, p_decoder_path, p_filter_path, p_vocab_path, &whisper);

	int iter = 0;
	for (int i=0; i < 5; i++)
	{
		clock_t start = clock();

		recognized_text.clear();
		// speech recognition语音识别
		ret = speech_recognition_run(&whisper, audio, task_code, recognized_text);
		
		clock_t end = clock();  // 记录结束时间
		double infer_time = ((double)(end - start)) / CLOCKS_PER_SEC;  // 转换为秒

		// 结果输出
		std::cout << "\nspeech recognition output: ";
		for (const auto &str : recognized_text){
			std::cout << str;
		}
		std::cout << std::endl;

		float audio_length = audio.num_frames / (float)SAMPLE_RATE; // sec
		audio_length = audio_length > (float)CHUNK_LENGTH ? (float)CHUNK_LENGTH : audio_length;
		float rtf = infer_time / audio_length;
		printf("%d, Real Time Factor (RTF): %.3f / %.3f = %.3f\n", iter++, infer_time, audio_length, rtf);
	}

	// speech recognition释放 
	speech_recognition_release(&whisper);
	return 0;
}