Created February 2, 2020 © GPL3+

Training Assistant for Melanoma Signs Detection

Nano provides the necessary tools to construct portable devices that can be used for educational purposes and as mass screening assistants.

AdvancedFull instructions provided2 days41

Training Assistant for Melanoma Signs Detection

Things used in this project

Hardware components

NVIDIA Jetson Nano Developer Kit

Story

Application Scope

In recent years AI assisted diagnostic technologies are building up rapidly in the medical sector, whether inference is based on signals or images. Such a prolific area is skin cancer, although as stated recently in https://www.aad.org/news/ai-and-skin-cancer-detection, “AI systems for skin cancer detection are still in their very early stages”. However, hardware advances such as the Jetson Nano and the increasing availability of large datasets such as the "HAM10000 dataset of pigmented skin lesions" provide the necessary tools to construct portable (Edge positioned) devices that can be used initially for educational purposes (in Hospitals, Universities, Conferences etc.) and subsequently as mass screening assistants, in particular at remote areas where population is difficult to receive professional specialized medical services.

Hardware Setup

In order to prepare the Jetson Nano for inference the following steps are required:

1. Perform initial setup of the board as described in:

https://developer.nvidia.com/embedded/learn/get-started-jetson-nano-devkit

2. Setup the inference environment by going through the instructions at:

https://github.com/dusty-nv/jetson-inference/blob/master/docs/building-repo-2.md

3. Install additional python libraries

a. xlrd to manipulate Excel Files:

$ sudo pip3 install xlrd

b. OpenCV:

$ sudo apt-get install python3-opencv
$ sudo apt-get remove python3-opencv

(this will bring up the correct version, it will not uninstall opencv!)

c. Matplotlib:

$ sudo apt-get install python3-matplotlib

d. Python Imaging Library (PILLOW):

$ sudo apt-get install python3-dev libjpeg-dev libfreetype6-dev zlib1g-dev
$ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libfreetype.so /usr/lib/
$ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libjpeg.so /usr/lib/
$ sudo ln -s /usr/lib/`uname -i`-linux-gnu/libz.so /usr/lib/
$ sudo pip3 install Pillow==6.1

4. Prepare the training Data

Training Data can be downloaded from:

https://www.kaggle.com/kmader/skin-cancer-mnist-ham10000

The unzipped file will produce two image directories: HAM10000_images_part_1 and HAM10000_images_part_2, and a number of csv files.

For the training we need the image files and the csv file: HAM10000_metadata.csv which is opened by Excel. Create a copy of the file, delete all columns except column B and column C (i.e. image_id and dx), delete the first row (which are labels), sort the data according to the second column (dx) and you will end up with file names sorted according to lesion type. Save this new file as an excel file.

The python code ‘excellCopy.py’ processes the new excel metadata file and splits the images in 7 sub directories, namely akiec, bcc, bkl, df, mel, nv, vasc which are the different classes the NN will be trained on. The scientific names are:

Actinic keratosis

Basal Cell Carcinoma

Benign Keratosis

Dermatofibroma

Malignant Melanoma

Melanocytic Nevi

Vascular Lesions

In line 9:

loc = ("YOUR_PATH/Metadata_2_columns.xlsx")

the location of this file is provided, and in line 14:

imgpath='YOUR_PATH\HAM10000_images_part_1'

the location of the first images directory is given. The script should be run again on the second part of the images once finished with the first by changing the above line to

imgpath='YOUR_PATH\HAM10000_images_part_2'

The code can be made more compact by creating and iterating a list of the above classes, but at the moment it is in a more brute force mode (job is done though)!

Once the seven sub directories are created you can proceed to the final organisation of your data as follows:

YOUR PATH/train

YOUR PATH/val

YOUR PATH/test

All seven directories must be copied under YOUR PATH/train and then create the same directories, but EMPTY, under YOUR PATH/eval and YOUR PATH/val. Cut a few images (approx. 20) from each directory in YOUR PATH/train and place them in the corresponding sub directory in YOUR PATH/val so that you create the evaluation dataset. Repeat the same procedure with YOUR PATH/test so that you create the test data set. It should be noted here that the data set is heavily biased towards the Melanocytic Nevi class (over 60% of images) and this reflects heavily on the classification of new images from the other categories.

At this point we are ready to perform "Transfer learning" to one of the pretrained image classification networks for the JETSON NANO. Please follow the instructions at (use sudo for the torch install script):

https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-transfer-learning.md

If you get errors in importing torch because of older numpy version then you need to install Cython

$ sudo pip3 install Cython

and then update numpy:

$ sudo pip3 install numpy --upgrade --ignore-installed

Be patient, it takes time for the above to complete.

The PyTorch training scripts are located in the repo under jetson-inference/python/training/classification/. These scripts aren't specific to any one dataset and by default are set to train a ResNet-18 model, but you can change that with the --arch flag.

Once finished creating the proper environment (stop at the Training Datasets heading), issue the command:

$python3 train.py --model-dir=YOUR_PATH1 YOUR_PATH --epochs n

The command must be issued from the location of the jetson-inference tools installed in step 2 above (probably in ~/jetson-inference). YOUR_PATH1 is the location where the retrained model will be saved, and YOUR_PATH is the location of the 3 sub directories: train, test and val with the images.

It will take about 8 hours to complete the tranfer learning process so sit back and relax, or do it overnight! However, I would suggest that first you train 4 epochs only, and test the whole tool chain for possible errors which creep up often because of the updates in the various models. I have tried to provide such fixes in all problems I encountered during two tests set 40 days apart! When the whole tool chain behaves as expected then train properly for 40 or more epochs.

To run the re-trained ResNet-18 model with TensorRT for testing and realtime inference, you need to convert the PyTorch model into ONNX format format so that TensorRT can load it. ONNX is an open model format that supports many of the popular ML frameworks, including PyTorch, TensorFlow, TensorRT, and others, so it simplifies transferring models between tools.

PyTorch comes with built-in support for exporting PyTorch models to ONNX, so run the following command to convert the HAM10000 model with the provided onnx_export.py script:

$ sudo python onnx_export.py --model-dir=YOUR_PATH1

This will create a model called resnet18.onnx under YOUR_PATH1 (eg. jetson-inference/python/training/classification/HAM1000/

You can also download the retrained model (I trained it for 60 epochs, approx. 15 hours) from:

https://drive.google.com/open?id=1Nj7wTkgyBxVyC6cnIXucvLyc3Nx59TgI

Note: If later on (when you execute image.py), you get TENSORT errors and classification can not run, you need to reinstall the torchvision package as follows:

$ sudo pip uninstall torchvision
$ python3 -c "import torchvision"   # should produce error if succesfully uninstalled
$ git clone -bv0.3.0 https://github.com/dusty-nv/vision
$ cd vision
$ sudo python3 setup.py install

After that you need to retrain and re-export your model.

Application Framework

The Python script image.py provides the testing/usage environment for the new resnet18 retrained model. It utilises a modified version of the imagenet-console.py tool provided by NVIDIA, so make sure its location is accessible when executing image.py.

Please make sure you provide your paths for PATH, PATH_TO_CONVERTED_MODEL, PATH_TO_LABELS, and TEMP_PATH in image.py script. The TEMP_PATH must also be provided at the imagenet-console_mine.py script.

The 'labels.txt' file contains the seven classes in whatever terminology suits you, for example:

Actinic keratosis

Basal Cell Carcinoma

Benign Keratosis

Dermatofibroma

Malignant Melanoma

Melanocytic Nevi

Vascular Lesions

The image.py script provides the following interface:

The Open File button selects and displays the image to be classified.

The process button performs the classification and overlays the result on the image.

A result is provided if confidence is above 60% as defined in imagenet-console_mine.py (line 64):

if confidence*100>60:

The Camera Frame captures a frame each time it is pressed in order to create new images to be processed.

The camera is interfaced in image.py by the

cap=cv2.VideoCapture(0)

command and for the particular instance controls a USB Camera (/dev/video0 in my case). There are numerous sites offering info on interfacing USB or RPI cameras to the JETSON NANO but things can be kept simple for the particular application. The camera quality is what really matters.

Conclusions

This project provides a framework to expand upon with the inclusion of more images for the less represented skin lesion classes, that can develop to a powerfull tool for medical training and assistive diagnosis. The model should also be trained on normal lesions in order to be more effective, while a backround removal preprocessing would benefit it, since it is currently biased with a skin background.

The required infrastructure, i.e. the NVIDIA JETSON NANO and a USB camera fall well below the $150 mark maiking it affordable to everyone interested in experimenting or building powerful medical diagnostic assistants.

Code

#!/usr/bin/env python
##############################################################################
# Modified by Tasos from Hajime Nakagami<nakagami@gmail.com>
# (http://www.freebsd.org/copyright/freebsd-license.html)
#
# 2020
##############################################################################
import PIL
import PIL.Image
import numpy as np
import jetson.inference
import jetson.utils

import cv2
import matplotlib.pyplot as plt

import argparse
import sys
import os

try:
    from Tkinter import *
    import tkFileDialog as filedialog
except ImportError:
    from tkinter import *
    from tkinter import filedialog
import PIL.ImageTk

class App(Frame):

    def process(self):
        os.system('PATH/imagenet-console_mine.py --model=PATH_TO_CONVERTED_MODEL/resnet18a.onnx --input_blob=input_0 --output_blob=output_0 --labels=PATH_TO_LABELS/labels.txt TEMP_PATH/current.jpg')
        self.im = PIL.Image.open('TEMP_PATH/processed.jpg')
        self.chg_image()
        
       
    def chg_image(self):
        self.num_page=self.num_page+1
        if self.im.mode == "1": # bitmap image
            self.img = PIL.ImageTk.BitmapImage(self.im, foreground="white")
        else:              # photo image
            self.img = PIL.ImageTk.PhotoImage(self.im)
        
        w = int(self.img.width())
        h = int(self.img.height())
        if w>1280 or h>720:
            wd=int(w*0.3)
            hg=int(h*0.3)
            newsize = (wd, hg) 
            self.im = self.im.resize(newsize) 
        elif w<640 or h<480:
            wd=640
            hg=480
            newsize = (wd, hg) 
            self.im = self.im.resize(newsize)
        else:   
            wd=w
            hg=h    

        if self.im.mode == "1": # bitmap image
            self.img = PIL.ImageTk.BitmapImage(self.im, foreground="white")
        else:              # photo image
            self.img = PIL.ImageTk.PhotoImage(self.im)   
        
        self.la.config(image=self.img, bg="#000000", width=wd, height=hg)
       

    def open(self):
        filename = filedialog.askopenfilename()
        if filename != "":
            self.im = PIL.Image.open(filename)
            self.num_page=self.num_page+1
            self.im.save('TEMP_PATH/current.jpg')
        self.chg_image()
        if self.num_page==2:
            Button(self, text="Process", command=self.process, fg='green').pack(side=TOP)
        

    def capture(self):
        cap=cv2.VideoCapture(0)    
        ret, capimg=cap.read()
        capimg=cv2.cvtColor(capimg, cv2.COLOR_BGR2RGB)
        self.im=PIL.Image.fromarray(capimg)
        self.num_page=self.num_page+1
        self.im.save('TEMP_PATH/current.jpg')
        self.chg_image()
        if self.num_page==2:
            Button(self, text="Process", command=self.process, fg='green').pack(side=TOP)

    def quit(self):
        self.destroy()
        exit()
        

    def __init__(self, master=None):
        Frame.__init__(self, master)
        self.master.title('Skin Lesions Classifier Assistant')
        self.num_page=0
        

        fram = Frame(self)
        Button(fram, text="Open File", command=self.open).pack(side=LEFT)
        Button(fram, text="Camera Frame", command=self.capture, fg='orange').pack(side=LEFT)
        Button(fram, text="Quit", command=self.quit, fg='red').pack(side=LEFT)
        
        fram.pack(side=TOP)
        self.la = Label(self)
        self.la.pack()

        self.pack()

if __name__ == "__main__":
    
    root=Tk()
    root.minsize(640,480)
    app = App(root)
    app.mainloop()

#!/usr/bin/python
#
# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
#

#Adapted by Tasos 2020

import jetson.inference
import jetson.utils

import argparse
import sys
import cv2
import matplotlib.pyplot as plt
import PIL
from PIL import ImageDraw, ImageFont
from PIL import Image

# parse the command line
parser = argparse.ArgumentParser(description="Classify an image using an image recognition DNN.", 
						   formatter_class=argparse.RawTextHelpFormatter, epilog=jetson.inference.imageNet.Usage())

parser.add_argument("file_in", type=str, help="filename of the input image to process")
parser.add_argument("file_out", type=str, default='TEMP_PATH/processed.jpg', nargs='?', help="filename of the output image to save")
parser.add_argument("--network", type=str, default="googlenet", help="pre-trained model to load (see below for options)")

try:
	opt = parser.parse_known_args()[0]
except:
	print("")
	parser.print_help()
	sys.exit(0)

# load an image (into shared CPU/GPU memory)
img, width, height = jetson.utils.loadImageRGBA(opt.file_in)
im_t = PIL.Image.open(opt.file_in)

# load the recognition network
net = jetson.inference.imageNet(opt.network, sys.argv)

# classify the image
class_idx, confidence = net.Classify(img, width, height)

# find the object description
class_desc = net.GetClassDesc(class_idx)

# print out the result. Set Classification Level over 60%
if confidence*100>60:
    print("image is recognized as '{:s}' (class #{:d}) with {:f}% confidence\n".format(class_desc, class_idx, confidence * 100))
else:
    print('Insufficient Classification')	

# print out timing info
net.PrintProfilerTimes()

# overlay the result on the image
w = width
h = height
if w>1280 or h>720:
            wd=int(w*0.3)
            hg=int(h*0.3)
            newsize = (wd, hg) 
            im_t = im_t.resize(newsize) 
elif w<640 or h<480:
            wd=640
            hg=480
            newsize = (wd, hg) 
            im_t = im_t.resize(newsize)
else:   
            wd=w
            hg=h  
if opt.file_out is not None:
	im_t.save(opt.file_out)
	img = Image.open(opt.file_out)
	# Select Font type and size
	font=ImageFont.truetype('/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf', 18)
	d = ImageDraw.Draw(img)
	if confidence*100>60:
		conf=round((confidence * 100), 2)
		caption='Image Classified as: '+class_desc+' @ '+str(conf)+'% Confidence'
		#fontsize=16
		d.text((10,10), caption, font=font, fill=(255, 255, 255, 255))
	else:
		conf=round((confidence * 100), 2)
		caption='Inconclusive @ '+str(conf)+'% Confidence'
		
		d.text((10,10), caption, font=font, fill=(255, 255, 0, 255))
	img.save(opt.file_out)

#!/usr/bin/python

#Tasos 2020

import xlrd 
import os
  
# Location of the excel file 
loc = (‘PATH to Excel File/Metadata_2_columns.xlsx’) 
wb = xlrd.open_workbook(loc) 
sheet = wb.sheet_by_index(0)
flag=1
i=0
imgpath= ‘PATH to first Images Files +'ham10000\HAM10000_images_part_1'

try:
    os.chdir(imgpath)
    print(os.getcwd())
except:
    print('Failed')    

#Start with First class "Actinic keratoses"
i=1
while flag:
    cval=sheet.cell_value(i, 1)
    if cval=='akiec':
        name=sheet.cell_value(i, 0)+'.jpg'
        
        os.system('copy '+name+’ ‘+'PATH to training data location+’+’\akiec') 
        i=i+1
    else:
        flag=0
flag=1

#Continue with Second class "Basal cell carcinoma"

while flag:
    cval=sheet.cell_value(i, 1)
    if cval=='bcc':
        name=sheet.cell_value(i, 0)+'.jpg'
        
        os.system('copy '+name+’ ‘+'PATH to training data location+’+’\bcc') 
        i=i+1
    else:
        flag=0

flag=1
#Continue with Third class "Benign Keratosis"

while flag:
    cval=sheet.cell_value(i, 1)
    if cval=='bkl':
        name=sheet.cell_value(i, 0)+'.jpg'
        
        os.system('copy '+name+’ ‘+'PATH to training data location+’+’\bkl') 
        i=i+1
    else:
        flag=0

flag=1
#Continue with Fourth class "Dermatofibroma"

while flag:
    cval=sheet.cell_value(i, 1)
    if cval=='df':
        name=sheet.cell_value(i, 0)+'.jpg'
        
        os.system('copy '+name+’ ‘+'PATH to training data location+’+’\df') 
        i=i+1
    else:
        flag=0

flag=1
#Continue with Fifth class "Melanoma"

while flag:
    cval=sheet.cell_value(i, 1)
    if cval=='mel':
        name=sheet.cell_value(i, 0)+'.jpg'
        
        os.system('copy '+name+’ ‘+'PATH to training data location+’+’\mel') 
        i=i+1
    else:
        flag=0

flag=1
#Continue with Sixth class "Melanocytic nevi"

while flag:
    cval=sheet.cell_value(i, 1)
    if cval=='nv':
        name=sheet.cell_value(i, 0)+'.jpg'
        
        os.system('copy '+name+’ ‘+'PATH to training data location+’+’\nv') 
        i=i+1
    else:
        flag=0

flag=1
#Continue with Seventh class "Vascular skin lesions"

while flag:
    cval=sheet.cell_value(i, 1)
    if cval=='vasc':
        name=sheet.cell_value(i, 0)+'.jpg'
        
        os.system('copy '+name+’ ‘+'PATH to training data location+’+’\vasc') 
        i=i+1
    else:
        flag=0

Credits

Tasos

10 projects • 2 followers

Training Assistant for Melanoma Signs Detection

Things used in this project

Hardware components

Story

Schematics

JETSON NANO

Code

image.py

imagenet-console_mine.py

excellCopy.py

Credits

Tasos

Comments

Embed the widget on your own site

Training Assistant for Melanoma Signs Detection

Training Assistant for Melanoma Signs Detection

Things used in this project

Hardware components

Story

Schematics

JETSON NANO

Code

image.py

imagenet-console_mine.py

excellCopy.py

Credits

Tasos

Comments

Related channels and tags