Hardware components | ||||||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
Introduction
For security cameras, more than 62,000 security cameras have been installed in the capital Bangkok, Thailand [1]. The retrieved frames from most cameras are of low quality, making it difficult to identify the appearance and patterns of objects in the scene. However, these low quality frames may be beneficial for limited capacity storage and transfer rate. Consequently, the performance of current camera quality may lead to the risk of social behavior and classification.
To achieve the value-added in the current scenario, the image enhancement or reconstruction is the priority of choice for most security cameras [2] and product recognition [3]-[4]. Single image super resolution (SISR) is one of the image enhancements that is used to enlarge the low to high quality image. This approach has significantly improved and recovered the unclear image, which is applied to minimize the risk of security and false classification or recognition.
The state-of-the-art SISRs are designed for full image enhancement, which has high cost and computation complexity. Generally, the enlargement of the image has been selected or automatically detected on a specific object of interest in the scene. For that reason, we propose a low-cost and low-complexity multi-functional super resolution system to apply in real-time applications.
[1] https://www.nationthailand.com/in-focus/40012998
[2] Lyu, K., Pan, S., Li, Y., & Zhang, Z. (2022). JSENet: A Deep Convolutional Neural Network for Joint Image Super-Resolution and Enhancement. Neurocomputing.
[3] Yang, X., Wu, W., Liu, K., Kim, P. W., Sangaiah, A. K., & Jeon, G. (2018). Long-distance object recognition with image super resolution: A comparative study. IEEE Access, 6, 13429-13438.
[4] Shamsolmoali, P., Celebi, M. E., & Wang, R. (2020). Deep learning approaches for real-time image super-resolution. Neural Computing and Applications, 32(18), 14519-14520.
The detail implementation steps are as follows:
Installation of Vitis
To install Vitis in pc please follow the steps from https://github.com/Xilinx/Vitis-AI
Next, install Vitis AI Library v2.0 from the link below https://github.com/Xilinx/Vitis-AI/blob/master/demo/Vitis-AI-Library/README.md
To set up the cross-compiler in PC, please follow the step one and set up the target on the Kria KV260, please follow the step two
https://github.com/Xilinx/Vitis-AI/tree/master/setup/mpsoc/VART#step1-setup-cross-compiler
Installation of Image and Video Test Kria Board and Run Vitis AI Library Examples
Implementation of RCAN in demo code
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu, "Image Super-Resolution Using Very Deep Residual Channel Attention Networks", ECCV 2018
Model description light-wieght RCAN for scale=2x. Compared with the original version, the changes include:
- Remove all Channel-attention (CA) modules
- Using less Residual blocks: --n_resgroups 3 --n_resblocks 2 --n_feats 32
On PC, we can get the model by downloading from the Vitis-AI repository. Clink on the link below
In file model.yaml
--
description: Pruned_RCAN for Single Image Super-resolution.
input size: 360*640
float ops: 86.95G
task: super-resolution
framework: tensorflow
prune: '0.98'
version: 2.0
files:
- name: tf_rcan_DIV2K_360_640_0.98_86.95G_2.0
type: float & quantized
board: GPU
download link: https://www.xilinx.com/bin/public/openDownload?filename=tf_rcan_DIV2K_360_640_0.98_86.95G_2.0.zip
checksum: 9b162e9b632bd00cb984ce571a089deb
- name: rcan_pruned_tf
type: xmodel
board: zcu102 & zcu104 & kv260
download link: https://www.xilinx.com/bin/public/openDownload?filename=rcan_pruned_tf-zcu102_zcu104_kv260-r2.0.0.tar.gz
checksum: 82452edf520936e91f6dda7633b110c8
- name: rcan_pruned_tf
type: xmodel
board: vck190
download link: https://www.xilinx.com/bin/public/openDownload?filename=rcan_pruned_tf-vck190-r2.0.0.tar.gz
checksum: 7d4b7020bf8f319657b8ee04a9e409c3
license: https://github.com/Xilinx/Vitis-AI/blob/master/LICENSE
--
We download board: GPU download link: https://www.xilinx.com/bin/public/openDownload?filename=tf_rcan_DIV2K_360_640_0.98_86.95G_2.0.zip
Will get "tf_rcan_DIV2K_360_640_0.98_86.95G_2.0.zip"
In readme.md file tell detail to retain, test and convert model to KV260.
--
# RCAN: Image Super-Resolution Using Very Deep Residual Channel Attention Networks
## Contents
1. [Environment](#Environment)
2. [Preparation](#Preparation)
3. [Test/Train](#Test/Train)
4. [Performance](#Performance)
5. [Model information](#Model information)
6. [Quantize](#Quantize)
## Environment
1. Environment requirement
- Python 3.6
- Tensorflow 1.15.1
- Pillow 6.0.0
- numpy 1.15.0
- scikit-image 0.15.0
2. Installation
- Create virtual envrionment and activate it (without docker):
```shell
conda create -n tf_rcan python=3.6
conda activate tf_rcan
```
- Activate virtual envrionment (with docker):
```shell
conda vitis-ai-tensorflow
```
- Install all the python dependencies using pip:
```shell
pip install -r requirements.txt
```
Note: If you are in the released Docker env, there is no need to create virtual envrionment.
## Preparation
1. Datasets description
- [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/)
- [Benchmarks](https://cv.snu.ac.kr/research/EDSR/benchmark.tar)
2. Dataset diretory structure
+ data
+ DIV2K
+ benchmark
## Test/Train
1. Model description
light-wieght RCAN for scale=2x. Compared with the original version, the changes include:
- Remove all Channel-attention (CA) moduel
- Using less Residual blocks: --n_resgroups 3 --n_resblocks 2 --n_feats 32
2. Evaluation
```
# perform evaluation on 4 benchmarks: Set5, Set14, B100, Urabn100
sh run_eval.sh
```
3. Training
```
# perform training on DIV2K dataset
sh run_train.sh
```
4. Testing on your own dataset
```
# set the test dataset path as your own directory
sh run_demo.sh
```
## Performance
- Note: the Flops is calculated with the output resolution is 720x1280
### Model_info
1. Data preprocess
```
data channel order: BGR(0~255)
input = input / 255.0
```
2. System Environment
The operation and accuracy provided above are verified in Ubuntu16.04.10, cuda-9.0, Driver Version: 460.32.03, GPU NVDIA P100
--
To implement it in the board, we have to build source files from the link below.
https://github.com/Xilinx/Vitis-AI/tree/master/demo/Vitis-AI-Library/samples/rcan
We use putty to command in kria board and Winscp to transfer file from PC.
We implement SR from model type int8 from rcan_pruned_tf.xmodel and rcan_pruned_tf.prototxt by command ./test_jpeg_rcan rcan_pruned_tf womanx2.png
The results should be similar to the one show in picture.
In terminal, show performance of at fps7.56.
The next step of the project is to convert the model in order to run on the PetaLinux 2021.1 platform. In addition, a GUI will be designed and integrated in the system by using Qt. The system will then be tuned for quality enhancement of surveillance videos.
Comments