Piotr TyneckiWojciech GłażewskiIwona SwietochowskaYana Minina
Created February 8, 2020 © GPL3+

Virus Activity Detector for Education and Research (VADER)

VADER is an educational and research solution for microbial activity detection. This AIoT system is supported by Image Recognition and AI.

IntermediateFull instructions providedOver 2 days324
Virus Activity Detector for Education and Research (VADER)

Things used in this project

Hardware components

NVIDIA Jetson Nano Developer Kit
NVIDIA Jetson Nano Developer Kit
×1
Arducam UC-544
×1
Samsung EVO Plus
×1
Mean Well RS-25-12
×1
Mean Well RS-25-5
×1
MUSBR-A511-40
LOW PROFILE SHELL
×1
HE04-150 Goobay
×6
GOOBAY-30584
×3
Marquardt 1801.1908
×1

Software apps and online services

VGG Image Annotator (VIA)
Jupyter Notebook
Jupyter Notebook
TensorFlow
TensorFlow
OpenCV
OpenCV
keras-retinanet
pandas
matplotlib
SOLIDWORKS
or Open Source alternative
nanoCAD
KeyShot
or Open Source alternative
3D-Tool
Google Draw IO
or fritzing.org alternatively

Hand tools and fabrication machines

3D Printer (generic)
3D Printer (generic)
SLS is preferred technology for VADER corpus
Petri Dish
Petri dish 90x15 or smaller

Story

Read more

Custom parts and enclosures

Step CADs archive

VADER steps in CAD format

Module Jetson

Module LED

Module Power

LED sockets plate

3D base

Petri Dish ring (83)

Module Arducam

Petri Dish ring (87)

Schematics

VADER render 1

VADER render 2

VADER render 3

Front view

Cross section

General diagram

Connections diagram

VADER - Technical Documentation

PDF

Code

VADER Python 3.6+ module

Python
VADER module support Deep Learning model loading, image pre-processing and prediction as well as plotting boxes with probabilities
# Virus Activity Detector for Education and Research (VADER)
#### Version: 0.3.1 (13.02.2020)

#### Authors:
# Piotr Tynecki
# Yana Minina
# Wojciech Glazewski
# Marcin Plociennik
# Iwona Swietochowska

import warnings
warnings.filterwarnings("ignore")

import keras
import tensorflow as tf

tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)

from keras_retinanet import models
from keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_image
from keras_retinanet.utils.visualization import draw_box, draw_caption
from keras_retinanet.utils.colors import label_color

import matplotlib.pyplot as plt
import cv2

import numpy as np


class VaderModelLoader:  
    """
    Class is responsible to load the model
    once and support multiply fast usage
    """
    
    def __init__(self, model_path: str) -> None:
        """
        Set model full path
        """
        
        self.model_path = model_path
         
    def _get_session(self) -> tf.compat.v1.Session:
        """
        Prepare setup for training session
        with GPU support
        """
        
        config = tf.compat.v1.ConfigProto()
        config.gpu_options.allow_growth = True
        
        return tf.compat.v1.Session(config=config)

    def _set_session(self) -> None:
        """
        Start session with prepared setup
        """
        
        keras.backend.tensorflow_backend.set_session(self._get_session())
        
    def load_model(self):
        """
        Load and return the model
        """

        self._set_session()
        
        model = models.load_model(
            self.model_path,
            backbone_name='vgg16'
        )

        return model
    
    
class VaderDetector:
    def __init__(self, model) -> None:
        """
        Set model object and supported predicted classes
        """
        
        # It can be extended by 'mini-spot' or 'bacterial'
        self.label_dict = {0: 'spot'}
        self.model = model
    
    def detect(
        self,
        image_path: str,
        score_th: float,
        output_path: str,
        resize: bool = True,
        crop: bool = True
    ) -> None:
        """
        Read, pre-process image and execute the model prediction
        
        This method support resizing and cropping as well
        """
        
        image = cv2.imread(image_path)

        if crop:
            image = image[170:2360,550:2750]

        if resize:
            image = cv2.resize(image, (1000, 1000))

        im_copy = image.copy()

        draw = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

        image = preprocess_image(image)
        image, scale = resize_image(image)

        # Model prediciton
        boxes, scores, labels = self.model.predict_on_batch(np.expand_dims(image, axis=0))
        boxes /= scale
        counter = 0
        
        # Draw boxes and probabilities on image
        for box, score, label in zip(boxes[0], scores[0], labels[0]):
            if score < score_th:
                break

            color = label_color(label)
            b = box.astype(int)
            draw_box(draw, b, color=color, thickness=6)
            caption = "{} {:.3f}".format(self.label_dict[label], score)
            draw_caption(draw, b, caption)
            counter += 1
        
        print(f'{counter} spots detected')

        # Draw original image with the images containing
        # predicted spots / mini-spots
        detected_img = cv2.cvtColor(draw, cv2.COLOR_BGR2RGB)
        cv2.imwrite(output_path, detected_img)
        fig, axs = plt.subplots(1, 2, figsize=(20, 20))

        images = [im_copy, detected_img]

        titles = ['Original image', 'Detected spots']

        for ax, title, image in zip(
            axs,
            titles,
            images,
        ):
            ax.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
            ax.set_title(title.capitalize())
            ax.grid(False)
            ax.axis("off")

        plt.show()

VADER Jupyter Notebook

Python
VADER step-by-step experiment with example
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Virus Activity Detector for Education and Research (VADER)\n",
    "#### Version: 0.3.1 (13.02.2020)\n",
    "\n",
    "#### Authors:\n",
    "* Piotr Tynecki\n",
    "* Yana Minina\n",
    "* Wojciech Gaewski\n",
    "* Marcin Pciennik\n",
    "* Iwona witochowska\n",
    "\n",
    "\n",
    "Use this Notebook when you Python 3.6+ virtual environment will be ready. All the details about installation were covered in **VADER technical documentation**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 (GPU server side)\n",
    "All steps from that section should be executed on GPU server.\n",
    "You can use free [Google Colab](https://colab.research.google.com/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1. Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import os\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert data to RetinaNet format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read CSV export from VIA tool\n",
    "data = pd.read_csv('vader_via_annotated_data.csv') \n",
    "\n",
    "# Create empty cols\n",
    "data['x1'] = None\n",
    "data['y1'] = None\n",
    "data['x2'] = None\n",
    "data['y2'] = None\n",
    "data['class_name'] = None\n",
    "\n",
    "# Extract data from the CSV into new data structure\n",
    "for i, row in data.iterrows():\n",
    "    atr = row.region_shape_attributes\n",
    "    res = json.loads(atr)\n",
    "    label = json.loads(row.region_attributes)\n",
    "    try:\n",
    "        data.loc[i, 'x1'] = res['x']\n",
    "        data.loc[i, 'y1'] = res['y']\n",
    "        data.loc[i, 'y2'] = res['y'] + res['height']\n",
    "        data.loc[i, 'x2'] = res['x'] + res['width']\n",
    "        data.loc[i, 'class_name'] = label['class']\n",
    "    except KeyError:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Final dataset\n",
    "data_fin = data[['filename','x1', 'y1', 'x2', 'y2','class_name']]\n",
    "\n",
    "# Replace 'image_phage' with your data folder name\n",
    "# with Petri dishes annotated images\n",
    "data_fin['filename'] = ['images_mini/' + x for x in data_fin.filename]\n",
    "\n",
    "# Get unique names and split \n",
    "names = data_fin.filename.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train/Test/Validation split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split ready to use data into train, test and validation sets\n",
    "train_names, test_val_names = train_test_split(\n",
    "    names,\n",
    "    test_size=0.3,\n",
    "    random_state=42,\n",
    "    shuffle=True\n",
    ")\n",
    "test_names, val_names = train_test_split(\n",
    "    test_val_names,\n",
    "    test_size=0.8,\n",
    "    random_state=42,\n",
    "    shuffle=True\n",
    ")\n",
    "\n",
    "# Split original dataset \n",
    "train = data_fin[data_fin.filename.isin(train_names)]\n",
    "test = data_fin[data_fin.filename.isin(test_names)]\n",
    "val = data_fin[data_fin.filename.isin(val_names)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save data into CSV files\n",
    "train.to_csv('vader_train.csv', header=False,index=False)\n",
    "test.to_csv('vader_test.csv', header=False, index=False)\n",
    "val.to_csv('vader_val.csv', header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2. Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Params: https://github.com/fizyr/keras-retinanet/blob/master/keras_retinanet/bin/train.py\n",
    "\n",
    "Weights: https://github.com/fizyr/keras-retinanet/releases/download/0.5.1/resnet50_coco_best_v2.1.0.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model\n",
    "# More details: https://github.com/fizyr/keras-retinanet#training\n",
    "!nohup retinanet-train  --backbone vgg16 --weights snap_vgg/vgg16_csv_10.h5 --steps 60 --random-transform --epochs 10 --batch-size 1 --snapshot-path snapshots_vgg_new/ --gpu 0 --freeze-backbone  --tensorboard-dir tb_vgg_new/ csv dataset/vader_train.csv dataset/classes.csv > train_vgg_new.out &"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert it (choose the snapshot with higher accuracy)\n",
    "# More details: https://github.com/fizyr/keras-retinanet#converting-a-training-model-to-inference-model\n",
    "!retinanet-convert-model snapshots/best_snapshot_name.h5 out/model_jetson.h5 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 (NVIDIA Jetson Nano side)\n",
    "\n",
    "All the steps from that section need to be executed on NVIDIA Jetson Nano."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1. Installation and Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install system packages\n",
    "!sudo apt-get install libatlas-base-dev gfortran\n",
    "!sudo apt-get install libhdf5-serial-dev hdf5-tools\n",
    "!sudo apt install libfreetype6-dev\n",
    "\n",
    "# Install pip\n",
    "!sudo apt-get install python3-pip\n",
    "\n",
    "# Install virtualenv\n",
    "!sudo pip3 install virtualenv\n",
    "\n",
    "# Create and activate virtualenv \n",
    "!virtualenv -p python3.6 vader\n",
    "!source vader/bin/activate\n",
    "\n",
    "# Install requirements \n",
    "!pip install matplotlib\n",
    "!pip install numpy\n",
    "!pip install jupyterlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Follow the instructions to make OpenCV work in a virtual environment:\n",
    "\n",
    "https://github.com/devedse/JetsonNanoGuide#3-setup-opencv\n",
    "\n",
    "To install keras-retinanet:\n",
    "\n",
    "https://github.com/devedse/JetsonNanoGuide#4-setup-keras-retinanet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To connect via SSH:\n",
    "1. In PC terminal run `ssh vader@192.168.0.135` (replace with expected ip address and jetson username) \n",
    "2. In PC terminal run `jupyter lab --port=8888 --no-browser --ip=0.0.0.0`\n",
    "\n",
    "Now you are able to access Jupyter Notebook (lab) from yor browser.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. How to take a picture to your Petri dish?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[gst-launch-1.0](https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html) is a tool that builds and runs basic GStreamer pipelines. We're using it to making pictures with Jetons and our camera system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gst-launch-1.0 nvarguscamerasrc do-timestamp=true num-buffers=30 ! 'video/x-raw(memory:NVMM), width=(int)3280, height=(int)2464, framerate=(fraction)21/1' ! nvvidconv ! jpegenc ! multifilesink location=image/test_vader.jpeg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Result will be saved as \"test_vader.jpeg\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3. Spots detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import custom vader modules\n",
    "from vader import VaderModelLoader, VaderDetector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load final model\n",
    "model = VaderModelLoader(\n",
    "    model_path='model_jetson.h5'\n",
    ").load_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare detector instance\n",
    "detector = VaderDetector(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Detect spots with 0.9 treshold\n",
    "# and save classification results into \"detected_test_vader.jpg\" file\n",
    "detector.detect(\n",
    "    image_path='image/test_vader.jpeg',\n",
    "    score_th=0.9,\n",
    "    output_path='detected_test_vader.jpg',\n",
    "    resize=True,\n",
    "    crop=True\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

Credits

Piotr Tynecki

Piotr Tynecki

1 project • 0 followers
ML and Python Developer / Software Architect / Ph.D student of AI and NLP usage in Fintech, Healthcare and BioTech.
Wojciech Głażewski

Wojciech Głażewski

0 projects • 0 followers
Mechanical and structure engineering, prototyping, overall robot concepts, design, esthetics.
Iwona Swietochowska

Iwona Swietochowska

0 projects • 1 follower
Yana Minina

Yana Minina

0 projects • 0 followers
Thanks to Bartosz Butler.

Comments

Related channels and tags