Setting Up OpenCV on the MaaXBoard

Prerequisites

Go through the tutorial here to get the MaaXBoard setup in headless mode, expand the filesystem, increase the swap file size, and set up remote desktop.

About OpenCV

From the official website:

OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products. Being a BSD-licensed product, OpenCV makes it easy for businesses to utilize and modify the code

This is one of the most comprehensive libraries for computer vision. If we want to use camera or images for our tests, we need OpenCV. This is quite a large package to compile and install. We will show two alternatives:

1. Install the prebuilt package

2. Build from the source code.

We will use OpenCV version 4.2.0. Since OpenCV is used in combination with python most of the time, we will prepare the python environment first.

Python setup

There is a list of required packages that we need to install before being able to compile/install OpenCV. They are documented here.

Some of them are also used for python image processing, so we will install all of them. This will take some bit of time to pull and install the binaries:

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install build-essential cmake unzip pkg-config 
sudo apt-get install libjpeg-dev libpng-dev libtiff-dev 
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev 
sudo apt-get install libxvidcore-dev libx264-dev
sudo apt-get install git libgtk2.0-dev
sudo apt-get install libgtk-3-dev libgtk2.0-dev
sudo apt-get install libcanberra-gtk* 
sudo apt-get install libatlas-base-dev gfortran 
sudo apt-get install python3-dev python3-tk 
sudo apt-get install libhdf5-dev

To save time, you could also do it all in one step, with "yes" flags enabled, like this:

sudo apt-get install build-essential cmake unzip pkg-config libjpeg-dev libpng-dev libtiff-dev libxvidcore-dev libx264-dev git libgtk2.0-dev libgtk-3-dev libgtk2.0-dev libcanberra-gtk* libatlas-base-dev gfortran python3-dev python3-tk libhdf5-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev -y

Virtualenv setup

One of the best practice rules when working with python libraries and opencv is to create a “virtual environment” for your user where you can customize all the library versions you are going to install. This comic perfectly illustrates why this is a best practice.

You can have multiple virtualenv as long as you have space on your drive (virtualenv settings are stored into the ~/.virtualenv folder in your home directory).

We'll create a virtualenv called “cv” to work on python/opencv/tensorflow

From your home folder use these commands to download PIP (package installer for python):

wget https://bootstrap.pypa.io/get-pip.py
sudo python3 ./get-pip.py

then install the package itself and remove the leftovers

sudo pip install virtualenv virtualenvwrapper 
rm -rf ~/get-pip.py ~/.cache/pip

now as user “ebv” (this was created in the prerequisite headless setup) we edit our bash shell configuration file ~/.bashrc using nano

nano ~/.bashrc

and at the very bottom of it we add these lines:

# virtualenv and virtualenvwrapper 
export WORKON_HOME=$HOME/.virtualenvs 
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3 
source /usr/local/bin/virtualenvwrapper.sh

Save with CTRL+O, exit with CTRL+X and make sure you reload your shell config with:

source ~/.bashrc

When you run this for the first time you will see the following output:

Now we can create the “cv” virtual environment itself with the command:

mkvirtualenv cv -p python3

You can enter the “cv” virtual environment by typing:

workon cv

Note that the prompt has now the header “(cv)” in front of it, it means you are “inside” the new env. You can exit from the virtualenv anytime by typing:

deactivate

The last package we need to install before OpenCV itself is NumPy (https://numpy.org/) which is the backbone of any numerical processing done in python. It's included in matplotlib, so we can just install that.

We want to make sure we are in the cv virtual environment before proceeding so if you deactivated it in the previous step, reactivate it now:

workon cv
pip install -U matplotlib

Since there are no prebuilt packages for ARMv8-a architecture this will take quite some time to fetch, compile and install all the required packages. (expect ~20min for both)

HINT: open another console and hit the command “htop”, or use the GKrellMonitor to keep an eye on your running activity so you know what is happening on the board, here is an example of htop while installing matplotlib:

Once all the packages are installed you can double check your python installation like this: In your console type “python” to enter the python interpreter, then type:

import numpy as np 
a = np.eye(3) 
print (a)

This will print a 3x3 identity matrix:

Hit CTRL+D to exit the python interpreter.

INSTALLING OPENCV

NOTE: OpenCV is now available from pip as a package that will run on arm.

You can simply install it with this command:

pip install opencv-python

Continue with the tutorial if you'd like to install from prebuilt sources, otherwise you're able to start using OpenCV now. Note that every time you create a new virtual environment, you'll have to simlink

OpenCV pre-built install

In this section we install and test OpenCV from.deb prebuilt packages that we have prepared for MaaxBoard. They're linked in the "Code" section.

These package have been built from source code with the instructions provided in the specific section of this howto. The files provided are:

• -rw-r--r-- 1 ebv ebv 1380918 Apr 2 16:07 OpenCV-4.2.0-aarch64-dev.deb
• -rw-r--r-- 1 ebv ebv 17355658 Apr 2 16:07 OpenCV-4.2.0-aarch64-libs.deb
• -rw-r--r-- 1 ebv ebv 15990 Apr 2 16:07 OpenCV-4.2.0-aarch64-licenses.deb
• -rw-r--r-- 1 ebv ebv 7088 Apr 2 16:07 OpenCV-4.2.0-aarch64-main.deb
• -rw-r--r-- 1 ebv ebv 1636152 Apr 2 16:07 OpenCV-4.2.0-aarch64-python.deb
• -rw-r--r-- 1 ebv ebv 1042 Apr 2 16:07 OpenCV-4.2.0-aarch64-scripts.deb

You can transfer these files on the MaaxBoard using the SCP command from your Windows (using PuttySCP) or linux laptop. For example with the command

scp ./OpenCV.zip ebv@10.0.0.245:

Do not forget the “:” at the end of the command. Once they are transferred on the board, unzip them and then use your console on the MaaxBoard as user “ebv” and sudo to install them all, like this:

unzip OpenCV.zip
cd OpenCV
sudo dpkg -i ././OpenCV-4.2.0-aarch64*

Now we have to create a link for our virtual environment to the installed OpenCV library. Follow these commands:

ebv@maaxboard:~$ cd ~/.virtualenvs/cv/lib/python3.7/site-packages/ 
ln -s /usr/lib/python3.7/site-packages/cv2/python-3.7/cv2.cpython-37m-aarch64-linux-gnu.so ./cv2.so

now enter your virtual environment and launch python:

workon cv

We are in python “interactive” mode, we can import the OpenCV library and print the version with the following lines:

python -c 'import cv2; print(cv2.__version__)'

Python example

Now that we have the full python/numpy/opencv installed we can make one simple example using the remote desktop and show how to load an image in numpy. First we will retrieve one simple image for test in our home folder

cd 
wget https://www.element14.com/community/themes/images/2019/MaaXBoard_front.png -O imtest.png

Now open the Geany editor by typing

geany

and type the following code:

import numpy as np

import cv2

img = cv2.imread('imtest.png',3)

cv2.imshow('image',img)

k = cv2.waitKey(10000)

Save the file in your home folder as “test1.py”

In terminal, launch the script from your virtual environment like this:

cd 
workon cv 
python ./test1.py

if everything is working you should see an image of the MaaxBoard popping up for 10s and then disappear.

Now let’s use matplotlib to show the image as if it was a matrix. We will import all the three installed libraries. In geany editor create a new file and save it with name “test2.py”. The script looks like this:

import numpy as np

import cv2

from matplotlib import pyplot as plt

img = cv2.imread('imtest.png',3)

plt.imshow(img, cmap = 'gray', interpolation = 'bicubic')

plt.axis()

plt.show()

Launch the script from the console window using python:

python ./test2.py

you will see a new image popping up where the axis will show the size (in pixels) of the image itself.

C++ Example

Create a new folder called "test3"

mkdir test3

Open geany and copy and paste the following code:

#include <stdio.h>
#include <opencv2/opencv.hpp>
using namespace cv;
int main(int argc, char** argv )
{
    if ( argc != 2 )
    {        
        printf("usage: test3.out <Image_Path>\n");
        return -1;
    }
    Mat image;
    image = imread( argv[1], 1 );
    if ( !image.data )
    {
        printf("No image data \n");
        return -1;
    }
    namedWindow("Display Image", WINDOW_AUTOSIZE );
    imshow("Display Image", image);
    waitKey(0);
    return 0;
}

Save the file as "test3.cpp" in the test3 folder.

Create another file and copy and paste this code:

cmake_minimum_required(VERSION 2.8)

project( test3 )

find_package( OpenCV REQUIRED )

add_executable( test3 test3.cpp )

target_link_libraries( test3 ${OpenCV_LIBS} )

Save the file as CMakeLists.txt. Generate the executable with the following commands:

cd test3
cmake .
make

Now you can run the test by typing

./test3 ~/imtest.png

OpenCV source build

OpenCV will take about ~3h to be built from source code on the target (i.e MaaxBoard). First we create a work folder “ocv” in our home folder, then we enter the folder itself:

cd 
workon cv 
mkdir ocv 
cd ocv

We then pull the source code for opencv and opencv_contrib package:

wget -O opencv.zip https://github.com/opencv/opencv/archive/4.2.0.zip 
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.2.0.zip 
unzip ./opencv.zip 
ln -s ./opencv-4.2.0 ./opencv 
unzip ./opencv_contrib.zip 
ln -s ./opencv_contrib-4.2.0 ./opencv_contrib 
rm *.zip

In the above list of commands, we have pulled the source code, unzip and sim-linked the code (and remove the original zip files). We can now create a build subfolder and configure the environment:

cd opencv 
mkdir build 
cd build

The full command to build OpenCV has a lot of options specific to the architecture and path for contrib folder. This is the one used for this tutorial specific to MaaxBoard:

cmake -D CMAKE_BUILD_TYPE=RELEASE \  -D CMAKE_INSTALL_PREFIX=/usr/local \  -D OPENCV_EXTRA_MODULES_PATH=~/ocv/opencv_contrib-4.2.0/modules \  -D ENABLE_PRECOMPILED_HEADERS=OFF\  -D ENABLE_NEON=OFF \  -D ENABLE_VFPV3=OFF \  -D BUILD_TESTS=OFF \  -D OPENCV_ENABLE_NONFREE=ON \  -D INSTALL_C_EXAMPLES=OFF \  -D INSTALL_PYTHON_EXAMPLES=OFF \  -D BUILD_EXAMPLES=OFF ..

You can copy and paste this command in your terminal on the MaaXBoard. This will launch the configuration of the build.

Once the cmake configuration is done you can launch the build with the command:

make

This will take about 2h to complete. You then install OpenCV with the command:

sudo make install

OpenCV packages

Once you have compiled OpenCV you can also prepare the.deb packages so you can reinstall this build on a different board. There is a minor bug in OpenCV 4.2.0 regarding packetization which require you to first launch a normal installation (sudo make install) and then launch the packetization. To do so you have to modify few things:

1) In your build folder open the CMakeCache.txt and modify the line where the CPACK BINARY_DEB variable is set. It must be written as BINARY_DEB=ON

2) Edit the following files: CPackConfig.cmake, CMakeVars.txt, CPackSourceConfig.cmake

geany CPackConfig.cmake
geany CMakeVars.txt
geany CPackSourceConfig.cmake

If the opencv version variable is set as “unknown” change it to “4.2.0”. In geany you can use replace (ctrl-H).

Now you can launch the packaging of the build with the command:

sudo make package

Debian packages will be available in the same build folder of OpenCV.

You can now symlink your build of OpenCV in /usr/local/lib/ to virtual environment with these commands:

cd ~/.virtualenvs/[ VIRTUAL ENVIRONMENT NAME ]/lib/python3.7/site-packages/ 
ln -s /usr/local/lib/python3.7/site-packages/cv2/python-3.7/cv2.cpython-37m-aarch64-linux-gnu.so ./cv2.so

Note: I ran into one error with setting up the symbolic links for virtual environments. After running the command to create the symlink, it didn't work until removed cv2.so and then linked it again:

rm ./cv2.so
ln -s /usr/local/lib/python3.7/site-packages/cv2/python-3.7/cv2.cpython-37m-aarch64-linux-gnu.so ./cv2.so

Good job! You can now use OpenCV on MaaXBoard. If you're not sure where to go next, check out these tutorials from the OpenCV docs, or look at the OpenCV page on Hackster for some more inspiration.