About the project
Motivation - Why we decided to make this project
Working
Results
Problems Faced
1. Installation of Vitis on Ubuntu
2. TensorFlow Frozen graph problem and its installation
3. OpenCV error
4. Installing Vitis AI
5. Real time webcam interfacing during YOLO v2-v3 implementation on Kria Kv-260

Team NirmaAgri: Ruchi Gajjar, Nagendra Gajar, Prof. Manish Patel, PARIN SHAH, Gayatri Rathod

Published April 18, 2022

Agriculture crop disease detection using ML & Kria KV260

To combat loss due to disease in crops, a machine learning based system with edge AI which detects disease can aid farmers for higher yield.

AdvancedOver 8 days1,113

Agriculture crop disease detection using ML & Kria KV260

Things used in this project

Hardware components

AMD Kria KV260 Vision AI Starter Kit

Raspberry Pi Camera Module

Software apps and online services

AMD Vitis Unified Software Platform

AMD PetaLinux

Jupyter Notebook

Story

About the project:

Crop disease damage is a major source of concern for farmers and India's agriculture-based economy accounts for a considerable amount of the country's GDP. This project work utilizes machine learning to determine the type of illness present in a crop based on leaf photos. First, using the Single Shot Detector, the leaves are individually detected in real-time from the video feed captured from the field. Images of the leaves taken from the field can also be fed as an input to this system. A convolutional neural network is presented to categorize the type of illness present in the crop, which is trained using the PlantVillage dataset, and the proposed hybrid network is implemented on Kria KV-260 for real-time detection and identification. So this platform is developed for advanced vision application development without requiring complex hardware design knowledge. KV260 also provides the benefit of differentiating our design at software level with Vitis AI. The accuracy of disease classification reached is around 95.88 percent.

Motivation - Why we decided to make this project?

Agriculture is extremely important to the world economy. Agriculture is well-known in India, and it employs over 60% of the population. Nowadays, the bulk of crop vegetation fails because crop disease detection is not successfully integrated into farmer harvesting processes. Every year, farmers battle disease damage to their crops. Farmers may benefit greatly from early detection and treatment of these diseases. It is difficult to find skilled experts in the field that can detect any type of plant disease. It would be a boon to farmers if an automated system could identify crop illnesses and other problems (such as malnutrition, weeds, or insect damage) through handheld devices or hardware on agricultural equipment in real time.

As a result, an system that can predict illness in crops before the entire harvest is ruined is required. Machine learning can be used to detect disease in crops and assist farmers in identifying the disease. Using the notion of deep learning, this research project intends to construct a real-time plant disease detection system. The model can be deployed on an embedded platform like Kria KV260 to provide real-time detection of the disease present in the crop. The main aim is to predict plant disease efficiently and so farmers can take effective measures before the disease spread in the crops. So, the objectives of the project are as follows:

Collecting crop disease dataset for Indian crops(Gujarat)
Developing a machine learning (ML) models for crop disease detection and classification.
Porting the ML model on Kria KV 260

Working:

The project's major goal is to detect leaves on plants effectively and then accurately identify the type of illness present on the leaf. The SSD model was used to recognize plant leaves and the a new architecture based on Convolutional Neural Network (CNN) is used to identify leaf diseases. The SSD model and proposed CNN model are integrated to create a hybrid model that can detect leaves and diagnose diseases at the same time. Furthermore, this suggested hybrid model is deployed on Kria KV-260 for real-time testing to answer the problem of real-time detection of plant leaf disease. Figure below depicts a block diagram of the proposed system for leaf identification and disease categorization.

Block Diagram of the proposed system

Results:

The proposed system was tested on the leaf images of PlantVillage dataset as well as the data captured from a real Tomato farm located nearby. By taking the system to a disease infected tomato leaf, the model's effectiveness for leaf detection and illness recognition was tested. The results are presented in figure below:

Disease Detection Results

As evident from the figure, the model is able to accurately identify the type of disease from the leaves - Spider Mites, Early Blight, Tomato Mosaic Virus and Leaf Mold. The suggested model performs well in all situations, including atmosphere, background, soil, and illumination, as evidenced by these real-time field tests.

Set-up

Team

Problems Faced:

1. Installation of Vitis on Ubuntu

Installation of Vitis and Vivado on Ubuntu is quite tiring. They are a lot of dependencies that are prerequisites and are must for installing Vitis. No such proper documentation or links are available for the proper installation. Moreover it takes a long time to get installed in the device. It took us more than 12 hours for the complete installation after referring to some dead links.

2. TensorFlow Frozen graph problem and its installation

For getting the final froze graph, the inputs are.pb and.ckpt file which gives us the output frozen_graph.pb. This was a major problem of TensorFlow library, without this graph further processing was not being done.

This instruction was not being executed:

freeze_graph --input_graph yolov2-tiny.pb --input_checkpoint yolov2-tiny.ckpt --output_graph frozen/frozen_graph.pb --output_node_names yolov2-tinyconvolutional9/BiasAdd --input_binary true

Installation issue of TensorFlow

TensorFlow installation issue

3. OpenCV error

For real time interfacing of camera module and the processing, open CV is the most extensively used Python Library. Without this there will be no camera interfacing taking place and installing it in Linux environment was very time consuming.

OpenCV error

4. Installing Vitis AI

For installing Vitis AI, we were trying to git clone the KV-260 ml accelerated library, but due to some issues the cloning was getting stucked at some point. For the installation purpose we need to create a Docker in which the Vitis-AI is to be installed. There were errors related to index-pack, GnuTLS and early EOF.

Vitis-AI installation error

5. Real time webcam interfacing during YOLO v2-v3 implementation on Kria Kv-260

Before beginning with our own model, we thought of implementing and checking the hardware with YOLO. However, the main problem for real time camera interfacing with KV-260 while implementing YOLO pre-trained models is that the kit was not supporting real time webcam interfacing.

YOLO v3 real-time webcam error

Code

'model.py' for project Agriculture crop disease detection system using machine learning & Edge AI

from google.colab import drive
drive.mount('/content/gdrive')

# -*- coding: utf-8 -*-
"""


# -*- coding: utf-8 -*-
"""

import tensorflow as tf
from tensorflow import keras

def create_model(input_shape,activation_fn,Batch_norm,kernel_init,dropout_rate,target_class):
    input_layer=keras.layers.Input(shape=input_shape)
    layer_1=keras.layers.Conv2D(filters=32,kernel_size=3,strides=1,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(input_layer)
    layer_2=keras.layers.Conv2D(filters=32,kernel_size=3,strides=2,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(layer_1)
    pool_1=keras.layers.MaxPool2D(pool_size=2)(layer_2)
    layer_3=keras.layers.Conv2D(filters=64,kernel_size=5,strides=2,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(pool_1)
    inception_1=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(layer_3)
    inception_1a=keras.layers.Conv2D(filters=32,kernel_size=3,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(inception_1)
    inception_1b=keras.layers.Conv2D(filters=32,kernel_size=3,strides=2,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(inception_1a)
    inception_2=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(layer_3)
    inception_2a=keras.layers.Conv2D(filters=32,kernel_size=3,strides=2,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(inception_2)
    inception_3=keras.layers.MaxPool2D(pool_size=2)(layer_3)
    inception_3a=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(inception_3)
    inception_4=keras.layers.Conv2D(filters=32,kernel_size=1,strides=2,padding='same'
                                ,kernel_initializer=kernel_init)(layer_3)
    concat=keras.layers.Concatenate()([inception_1b,inception_2a,inception_3a,inception_4])
    layer_4=keras.layers.Conv2D(filters=128,kernel_size=5,strides=1,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(concat)
    
    _inception_1=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(layer_4)
    _inception_1a=keras.layers.Conv2D(filters=32,kernel_size=3,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(_inception_1)
    _inception_1b=keras.layers.Conv2D(filters=32,kernel_size=3,strides=2,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(_inception_1a)
    _inception_2=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(layer_4)
    _inception_2a=keras.layers.Conv2D(filters=32,kernel_size=3,strides=2,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(_inception_2)
    _inception_3=keras.layers.MaxPool2D(pool_size=2)(layer_4)
    _inception_3a=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(_inception_3)
    _inception_4=keras.layers.Conv2D(filters=32,kernel_size=1,strides=2,padding='same'
                                ,kernel_initializer=kernel_init)(layer_4)
    concat=keras.layers.Concatenate()([_inception_1b,_inception_2a,_inception_3a,_inception_4])
    
    
    layer_5=keras.layers.Conv2D(filters=128,kernel_size=5,strides=2,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(concat)
    layer_6=keras.layers.Conv2D(filters=256,kernel_size=7,strides=2,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(layer_5)
    layer_7=keras.layers.Conv2D(filters=256,kernel_size=7,strides=2,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(layer_6)
    flat=keras.layers.Flatten()(layer_7)
    dense1=keras.layers.Dense(512,activation='relu',kernel_initializer=kernel_init)(flat)
    drop_1=keras.layers.Dropout(rate=0.25)(dense1)
    dense2=keras.layers.Dense(256,activation='relu',kernel_initializer=kernel_init)(drop_1)
    drop_2=keras.layers.Dropout(rate=0.3)(dense2)
    dense3=keras.layers.Dense(128,activation='relu',kernel_initializer=kernel_init)(drop_2)
    drop_3=keras.layers.Dropout(rate=0.5)(dense3)
    output=keras.layers.Dense(target_class,activation='softmax')(drop_3)
    
    model=keras.Model(inputs=[input_layer],outputs=[output])
    return model
    
    # -*- coding: utf-8 -*-


import tensorflow as tf
from tensorflow import keras

def create_model(input_shape,activation_fn,Batch_norm,kernel_init,dropout_rate,target_class):
    input_layer=keras.layers.Input(shape=input_shape)
    layer_1=keras.layers.Conv2D(filters=32,kernel_size=3,strides=1,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(input_layer)
    layer_2=keras.layers.Conv2D(filters=32,kernel_size=3,strides=2,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(layer_1)
    pool_1=keras.layers.MaxPool2D(pool_size=2)(layer_2)
    layer_3=keras.layers.Conv2D(filters=64,kernel_size=5,strides=2,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(pool_1)
    inception_1=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(layer_3)
    inception_1a=keras.layers.Conv2D(filters=32,kernel_size=3,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(inception_1)
    inception_1b=keras.layers.Conv2D(filters=32,kernel_size=3,strides=2,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(inception_1a)
    inception_2=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(layer_3)
    inception_2a=keras.layers.Conv2D(filters=32,kernel_size=3,strides=2,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(inception_2)
    inception_3=keras.layers.MaxPool2D(pool_size=2)(layer_3)
    inception_3a=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(inception_3)
    inception_4=keras.layers.Conv2D(filters=32,kernel_size=1,strides=2,padding='same'
                                ,kernel_initializer=kernel_init)(layer_3)
    concat=keras.layers.Concatenate()([inception_1b,inception_2a,inception_3a,inception_4])
    layer_4=keras.layers.Conv2D(filters=128,kernel_size=5,strides=1,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(concat)
    
    _inception_1=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(layer_4)
    _inception_1a=keras.layers.Conv2D(filters=32,kernel_size=3,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(_inception_1)
    _inception_1b=keras.layers.Conv2D(filters=32,kernel_size=3,strides=2,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(_inception_1a)
    _inception_2=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(layer_4)
    _inception_2a=keras.layers.Conv2D(filters=32,kernel_size=3,strides=2,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(_inception_2)
    _inception_3=keras.layers.MaxPool2D(pool_size=2)(layer_4)
    _inception_3a=keras.layers.Conv2D(filters=32,kernel_size=1,strides=1,padding='same'
                                ,kernel_initializer=kernel_init,activation=activation_fn)(_inception_3)
    _inception_4=keras.layers.Conv2D(filters=32,kernel_size=1,strides=2,padding='same'
                                ,kernel_initializer=kernel_init)(layer_4)
    concat=keras.layers.Concatenate()([_inception_1b,_inception_2a,_inception_3a,_inception_4])
    
    
    layer_5=keras.layers.Conv2D(filters=128,kernel_size=5,strides=2,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(concat)
    layer_6=keras.layers.Conv2D(filters=256,kernel_size=7,strides=2,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(layer_5)
    layer_7=keras.layers.Conv2D(filters=256,kernel_size=7,strides=2,padding='same',
                                kernel_initializer=kernel_init,activation=activation_fn)(layer_6)
    flat=keras.layers.Flatten()(layer_7)
    dense1=keras.layers.Dense(512,activation='relu',kernel_initializer=kernel_init)(flat)
    drop_1=keras.layers.Dropout(rate=0.25)(dense1)
    dense2=keras.layers.Dense(256,activation='relu',kernel_initializer=kernel_init)(drop_1)
    drop_2=keras.layers.Dropout(rate=0.3)(dense2)
    dense3=keras.layers.Dense(128,activation='relu',kernel_initializer=kernel_init)(drop_2)
    drop_3=keras.layers.Dropout(rate=0.5)(dense3)
    output=keras.layers.Dense(target_class,activation='softmax')(drop_3)
    
    model=keras.Model(inputs=[input_layer],outputs=[output])
    return model
    
    #!/usr/bin/env python3
# -*- coding: utf-8 -*-


from tensorflow import keras
import tensorflow as tf
import numpy as np
import os
import matplotlib.pyplot as plt
import matplotlib.image as mpimg



#from dataset_tf import dataset

#from clr_callback import CyclicLR
classes=np.array(os.listdir("/content/gdrive/MyDrive/KRIA_PROJECT_2022/project_94%/dataset_final/main_dataset_20/Test"))
learning_rate=1e-3
batch_size=1
momentum=0.9
nesterov=True
metrics=[keras.metrics.categorical_accuracy]
epochs=30
valid_step=137
train_step=1200
epoch_step=1    #no of epoch after which lr decays
  #factor by which lr decays
activation_fn="elu"
Batch_norm=False
kernel_init="he_normal"
dropout_rate=0.5
target_class=20
steps=123
image_="/content/gdrive/MyDrive/KRIA_PROJECT_2022/Tomato/A.JPG"
image_1="/content/gdrive/MyDrive/KRIA_PROJECT_2022/Tomato/E.jpg"
image_2="/content/gdrive/MyDrive/KRIA_PROJECT_2022/Tomato/2.jpg"
image_3="/content/gdrive/MyDrive/KRIA_PROJECT_2022/Tomato/5.jpg"
#image_="/media/NIK/project_94%/dataset_final/main_dataset_20/Test/Apple___Black_rot/0bb8fb61-d561-43fd-89a3-d24d253adef9___JR_FrgE.S 2849.JPG"
#image_1="/home/Documents/project-118,68/dataset_final/main_dataset_20/Test/Corn_(maize)___Cercospora_leaf_spot Gray_leaf_spot/2aca721c-1600-4f8c-8e79-df33973c671c___RS_GLSp 9348.JPG"
#image_2="/home/Documents/project-118,68/dataset_final/main_dataset_20/Test/Potato___Late_blight/127521ec-3cc5-4f98-ac73-ad561ec29e73___RS_LB 3219.JPG"
#image_3="/home/Documents/project-118,68/dataset_final/main_dataset_20/Test/Tomato___Leaf_Mold/1c03ffcb-9476-453c-96be-5bd1f629b2d1___Crnl_L.Mold 8853.JPG"
img=[image_,image_1,image_2,image_3]
model=create_model(input_shape=[256,256,3],activation_fn='relu',Batch_norm=False,
                   kernel_init='he_normal',dropout_rate=dropout_rate,target_class=target_class)
optimizer=keras.optimizers.Nadam(learning_rate=learning_rate,beta_1=0.9,beta_2=0.999,epsilon=1e-07)
model.compile(loss="categorical_crossentropy",optimizer=optimizer,metrics=metrics)
latest=tf.train.latest_checkpoint("/content/gdrive/MyDrive/KRIA_PROJECT_2022/project_94%/weights")
model.load_weights(latest)

#obj=dataset("test",batch_size=batch_size)
#test_ds=obj.test_dataset_ready()
for i in range(len(img)):
    image=tf.io.read_file(img[i])
    image=tf.image.decode_jpeg(image,channels=3)
    # print(image.shape)
    image/=255
    image=tf.reshape(image,[1,256,256,3])
    conf=model.predict(image,steps=1)
    # print(conf)
    index=np.argmax(conf.reshape(20))
    # print(index)
    print(classes[index])

fig=plt.figure(figsize=(8,8))
col=2
rows=2
for i in range(1,col*rows+1):
    img_=mpimg.imread(img[i-1])
    fig.add_subplot(col,rows,i)
    plt.imshow(img_)
plt.show()

Agriculture crop disease detection using ML & Kria KV260