DeepLabV3 Board Application

DeepLabV3 Board Application

The application demonstrates the Image semantic segmentation. DeepLabV3 is used in this application, which is an algorithm implemented by Google for semantic segmentation.

This will classify the object in each pixel from given portrait image and assigns the label to each of them. Image semantic segmentation is helpful in applications like background change, portrait mode, etc.

To develop a DeepLabV3 model for image segmentation we use Snapdragon mobile platforms(SD835) and Qualcomm Neural Processing SDK.

Recommended setup for model training

Hardware prerequisite

• Intel i5 or greater
• NVIDIA 10 series or greater
• Ram 16 GB or more

System software requirements

• Ubuntu 14.04 LTS or above
• Cuda
• CuDNN

How to train the model

Tensorflow’s DeepLab API setup is the prerequisite for training the DeepLabV3 model.For installation steps follow the below instruction,

• Execute the following commands for installing the dependencies,

# Run this command from <PROJECT_ROOT_DIR>
$ sudo pip install -r dependencies/requirement.txt

• Clone the tensorflow and its model repositories by executing the below commands,

$ git clone https://github.com/tensorflow/tensorflow.git
$ cd tensorflow
$ git clone https://github.com/tensorflow/models.git

• Update the Python Environment setup

$ cd <path to tensorflow setup>/tensorflow/models/research
$ export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim

Note: The environment setup needed to be made for every new terminal session.

To avoid updating the environment path for every new session, add above lines at the end of.bashrc file with an absolute path.

Testing the Installation

Run the below commands to test if installation is successful,

$ cd <path to tensorflow setup>/tensorflow/models/research
$ python deeplab/model_test.py

After completion of installation, execute the below commands to start the training,

$ cd <path to tensorflow setup>/tensorflow/models/research/deeplab
$ sh local_test.sh 

Note: By default the iteration count in local_test.sh is 10, if required kindly modify.

How to convert Tensorflow's model into DLC?

Prerequisites: Qualcomm Neural Processing SDK setup. Use the instructions from the below link to make the setup,

https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk/getting-started

• Initialize the environmental variables of Neural Processing SDK SDK with tensorflow.
• The model is trained using TensorFlow framework and exported to graph file with.pb extension.
• Once you’ve.pb file convert it into.dlc using the following command:

Inference on Ubuntu using Qualcomm Neural Processing SDK

SDK doesn’t support direct images as an input to the model for inference.

Qualcomm Neural Processing SDK requires the NumPy array stored as a raw file. In order to run the application in SDK we need to preprocess the input image.

Following are the details of pre-processing steps followed in src/deep_input_raw.py file,

(We have used the opencv for pre-processing the image)

• Resize the image with the dimensions of 513x513x3.
• Pad the smaller dimension with the mean value of 128 to produce an image of 513x513x3.
• Convert the image to type float32.
• Multiply the image element-wise with 0.00784313771874 and subtract 1.0 respectively.
• Store this pre-processed array as a raw file (blob.raw).

On executing the src/deep_input_raw.py script, blob.raw file is generated.

Procedure to change the background using DeepLab

Following is the detailed description of steps followed to change the background for a pre-processed image using src/post_process_deeplab_output.py,

• The output (output/Result_0/ArgMax:0.raw) from the previous section is a NumPy array of dimension 513x513x1.
• Each element in the array contains the predicted class index of the corresponding pixels for the given input image.
• The index number of person is 15.
• Read the NumPy array into a data buffer of type float32.
• To change the background to grayscale, the pixel values are assigned such that R, G, and B components have identical values.For example, the pixel values R(123), G(93) and B(49) are modified to R(123), G(123), B(123).
• Loop through the NumPy array and change the pixel values in the original resized image (as per step 5 above) other than pixels of class index 15.
• Resize the image to the original size.

post_process_deeplab_output.py script will change the background to grayscale for a pre-processed image.