AgriAI : Pest Detection for Plants Using Deep Learning

Situation

In modern Agriculture and Forestry purposes Plant Pathological Research is very much essential for the crop improvement. In every year, huge crop is damaged by pathogen attack. In most of the cases, Farmers and lay men suffers huge loss due to severe pathogenic infestation. For crop improvement, the first and foremost essential key point is diseases identification. After proper identification of diseases we will be able to treat the diseases. Most of the cases lab oriented research is time taking and very much expendable but instant identification of plant diseases with its causal organism by using modern technology is most suitable and cost effective and also authentic. This type identifying methodology will help us to identify the diseases with its proper control measures. Above 95 % accuracy and authentic identification is possible by using this methodology.

As per Wikipedia The impact of pesticides consists of the effects of pesticides on non-target species. Pesticides are chemical preparations used to kill fungal or animal pests. Over 98% of sprayed insecticides and 95% of herbicides reach a destination other than their target species, because they are sprayed or spread across entire agricultural fields.Runoff can carry pesticides into aquatic environments while wind can carry them to other fields, grazing areas, human settlements and undeveloped areas, potentially affecting other species. Other problems emerge from poor production, transport and storage practices.Over time, repeated application increases pest resistance, while its effects on other species can facilitate the pest's resurgence.

Each pesticide or pesticide class comes with a specific set of environmental concerns. Such undesirable effects have led many pesticides to be banned, while regulations have limited and/or reduced the use of others. The global spread of pesticide use, including the use of older/obsolete pesticides that have been banned in some jurisdictions, has increased overall.

Thus we can conclude that even though pesticides and insecticides are pretty useful but now a days the reverse of it i.e. harmfulness is getting increased in a rapid rate.

More information can be easily expressed through this video below -

Check out the video from Ted-Ed below to know more about the problem -

Idea

So the point arises - " What can we do to control usage of pesticides ? ".

Over here we should keep one thing in mind is that we need to make a system which can lower down or even maybe track down the usage of pesticides cause we can't pull it down for some obvious reasons to maintain the agriculture industry.

Theory

In our project we will be using the feature and characteristics of being warm-blooded mammal called thermoregulation. Here is a image showing the presence of thermoregulation and being scanned by a IR scanner camera -

So basically we will scan up using the AMG8833 Sensor Camera breakout using the balenaFin and it will show up whether the pest is present there or isn't. And it will then tell us where to exactly spray the insecticide or pesticide and where shouldn't.

Hardware

balenaFin

The balenaFin has been designed with field deployment in mind. It is a carrier board for the Raspberry Compute Module 3 and 3+ Lite (CM3L/CM3+L), that can run all the software that the Raspberry Pi can run, hardened for field deployment use cases.

Raspberry Pi Compute Module

The balenaFin supports the Raspberry Pi Compute Module 3 and 3+ lite (CM3L/CM3+L).

Storage

The storage on the balenaFin is based on an industrial grade eMMC storage with 8GB, 16GB, 32GB and 64GB options available.

Power

The balenaFin features a wide input voltage range (6V-24V), especially suitable for applications where a reliable 5V is usually not available.

Co-processor

The balenaFin includes a low-power co-processor (32-bit ARM® Cortex M4) with Bluetooth support. The co-processor can run on its own or in parallel and allows the main processor to be powered on and off programmatically. This is especially useful in applications where low power consumption or real-time processing is required.

Connectivity

The balenaFin's wireless chip supports 802.11ac/a/b/g/n WiFI and Bluetooth 4.2 (including SMART features). There is a dual-band embedded antenna included in the board and an external antenna connector for improved signal coverage.

I/O

The Mini PCI Express port on the balenaFin brings seamless connectivity to a number of different modules. Third party modules are readily available for LTE, Zigbee, LoRA and CANBus and extra storage can be achieved by leveraging the USB interface on the mini PCI Express connector (this will probably require a custom design). The balenaFin HAT header can be used to connect any Raspberry Pi HAT compatible module (PoE, RS232, ZWave, etc). A smaller 18-pin header exposes the co-processor's analog and time-sensitive I/O. V1.1 includes an extra 4-pin USB header that allows wire-free design applications such as additional storage, secondary ethernet port, multimedia readers, additional radio interfaces, etc.

Raspberry Pi Compute Module 3+ Lite

The CM3+ Compute Module contains the guts of a Raspberry Pi 3 Model B+ (the BCM2837 processor and 1GB RAM) as well as an optional eMMC Flash device of 8GB, 16GB or 32GB (which is the equivalent of the SD card in the Pi).

This is all integrated onto a small (67.6mm × 31mm) board that fits into a standard DDR2 SODIMM connector. The Flash memory is connected directly to the processor on the board, but the remaining processor interfaces are available to the user via the connector pins. You get the full flexibility of the BCM2837 SoC (which means that many more GPIOs and interfaces are available than with a standard Raspberry Pi), and designing the Module into a custom system should be relatively straightforward because we’ve put all the tricky bits onto the Module itself. This needs to be connected with the balenaFin.

Adafruit AMG8833 IR Camera Breakout Module

This sensor from Panasonic is an 8x8 array of IR thermal sensors. When connected to your microcontroller (or Raspberry Pi) it will return an array of 64 individual infrared temperature readings over I2C. It's like those fancy thermal cameras, but compact and simple enough for easy integration.

This part will measure temperatures ranging from 0°C to 80°C (32°F to 176°F) with an accuracy of +- 2.5°C (4.5°F). It can detect a human from a distance of up to 7 meters (23) feet. With a maximum frame rate of 10Hz, It's perfect for creating your own human detector or mini thermal camera. We have code for using this breakout on an Arduino or compatible (the sensor communicates over I2C) or on a Raspberry Pi with Python. On the Pi, with a bit of image processing help from the SciPy python library we were able to interpolate the 8x8 grid and get some pretty nice results!

The AMG8833 is the next generation of 8x8 thermal IR sensors from Panasonic, and offers higher performance than it's predecessor the AMG8831. The sensor only supports I2C, and has a configurable interrupt pin that can fire when any individual pixel goes above or below a threshold that you set.

Intel Neural Compute Stick 2

Intel Neural Compute Stick 2 is powered by the Intel Movidius X VPU to deliver industry-leading performance, wattage, and power. The NEURAL COMPUTE supports OpenVINO, a toolkit that accelerates solution development and streamlines deployment. The Neural Compute Stick 2 offers plug-and-play simplicity, support for common frameworks and out-of-the-box sample applications. Use any platform with a USB port to prototype and operate without cloud compute dependence. The Intel NCS 2 delivers 4 trillion operations per second with an 8X performance boost compared to previous generations.

What It Does:

Bringing computer vision and AI to the Internet of Things (IoT) and edge device prototypes is easy with the enhanced capabilities of the Intel NCS 2. For developers working on a smart camera, a drone, an industrial robot or the next must-have smart home device, the Intel NCS 2 offers what’s needed to prototype faster and smarter.

What looks like a standard USB thumb drive hides much more inside. The Intel NCS 2 is powered by the latest generation of Intel VPU – the Intel Movidius Myriad X VPU. This is the first to feature a neural compute engine – a dedicated hardware neural network inference accelerator delivering additional performance. Combined with the Intel Distribution of the OpenVINO toolkit supporting more networks, the Intel NCS 2 offers developers greater prototyping flexibility. Additionally, thanks to the Intel AI: In the Production ecosystem, developers can now port their Intel NCS 2 prototypes to other form factors and productize their designs.

How It Works:

With a laptop and the Intel NCS 2, developers can have their AI and computer vision applications up and running in minutes. The Intel NCS 2 runs on a standard USB 3.0 port and requires no additional hardware, enabling users to seamlessly convert and then deploy PC-trained models to a wide range of devices natively and without internet or cloud connectivity.

The first-generation Intel NCS, launched in July 2017, has fueled a community of tens of thousands of developers, has been featured in more than 700 developer videos and has been utilized in dozens of research papers. Now with greater performance in the NCS 2, Intel is empowering the AI community to create even more ambitious applications.

Features:

Boost productivity:

• Reduce time to prototype or tune neural networks with versatile hardware processing capabilities at a low cost.
• Enhanced hardware processing capabilities vs. the original Intel Movidius Neural Compute Stick.
• Take advantage of 16 cores instead of 12 plus a neural to compute engine, a dedicated d deep neural-network accelerator.
• Up to 8X performance gain on deep neural network inference, depending on network.
• Affordability accelerates deep neural network applications.
• Transform the AI development kit experience.
• Plug and Play Simplicity.
• Affordable price point.
• Supports common frameworks and includes out-of-box and fast development.

Discover efficiencies:

• Exceptional performance per watt takes machine vision to new places.
• Run “at the edge” without reliance on a cloud computing connection.
• Deep learning prototyping is now available on a laptop, a single board computer or any platform with a USB port.
• Accessible and affordable — take advantage of more performance per watt and highly efficient fanless design.
• Combine the hardware-optimized performance of the Intel® Movidius™ Myriad™ X VPU and the Intel® Distribution of OpenVINO” Toolkit to accelerate deep neural network-based applications.
• First in its class to feature the Neural Compute Engine — a dedicated hardware accelerator.
• 16 powerful processing cores, called SHAVE cores, and an ultrahigh-throughput intelligent memory fabric together make the Intel Movidius Myriad X VPU the industry leader for on-device deep neural networks and computer vision applications.
• Featuring an entirely new deep neural network (DNN) inferencing engine on the chip.

Simpler versatility for prototyping :

• Intel Distribution of OpenVINO toolkit streamlines the development experience.
• Prototype on the Intel Neural Compute Stick 2 and then deploy your deep neural network onto an Intel Movidius Myriad X VPU-based embedded device.
• Streamline the path to a working prototype.
• Extend workloads across Intel hardware and maximize performance.
• The robust, Intel Distribution of OpenVINO toolkit enables simpler porting and deployment of applications and solutions that emulate human vision.
• The Intel Distribution of OpenVINO toolkit streamlines the development of multiplatform computer vision solutions — increasing deep learning performance.
• It’s now easier to develop applications for heterogeneous execution across the suite of Intel acceleration technologies. Develop once and deploy across Intel CPU, VPU, Integrated Graphics or FPGA.
• If desired, users can implement their own custom layers and execute those on the CPU while the rest of the model runs on the VPU.

Software

Raspbian (currently Raspberry Pi OS)

It is the Foundation’s official supported operating system.

Raspberry Pi OS comes pre-installed with plenty of software for education, programming and general use. It has Python, Scratch, Sonic Pi, Java and more.

The Raspberry Pi OS with Desktop image contained in the ZIP archive is over 4GB in size, which means that these archives use features which are not supported by older unzip tools on some platforms.

NOTE: Here we have to use the Raspbian Image which is specially curated for the balenaFin. Download from here.

balenaEtcher

balenaEtcher is a free and open-source utility used for writing image files such as .iso and .img files, as well as zipped folders onto storage media to create live SD cards and USB flash drives. It is developed by balena, and licensed under Apache License 2.0. Download from here.

OpenVINO

OpenVINO toolkit is a free toolkit facilitating the optimization of a Deep Learning model from a framework and deployment using an inference engine onto Intel hardware. It is written i Python and C++.

Explanation & Working

So here comes the basic demo of our project being explained in a paragraph. So at first the AMG8833 Camera Sensor Board wll be attached to the balenaFin and also the Intel Neural Compute Stick 2 needs to be attached to the Pi for increasing the performance of the model and training it using Intel's OpenVino. The usage of sensor has been explained below through the GIF.

Now the full device consisting of the Fin will be placed on the bottom of a metal rod (such as one in metal detector) such as the one given below.

Evaluation

If a large heat source like a nest isn’t obvious, sensitive thermal cameras may be able to detect irregular heat patterns, moisture, and other signs of damage that indicate the presence of a pest, like missing insulation or holes in walls that indicates an entry point. Understanding what patterns to look for requires training and practice, as it’s easy to incorrectly interpret an image.

Here’s some indicators to look for when trying to locate the following common intruders:

Termites

Termites nests may be visible as hotspots (left), and termite tunnels have high moisture content that can be detected with thermal imaging (right).

Termites can be found by looking for nests, moisture sources, and evidence of damage in walls. When termites enter a home, they release heat from their digestive system in the form of carbon dioxide and construct mud tubes that have high moisture content, creating irregular heat patterns on the surface of walls, ceilings, and floors.

Rodents

Rodents (like the possum on the left) or their nests may be visible, or wildlife activity may leave clues like missing insulation (right).

Rodents and other wildlife might create nests that can be detected, or might have damaged walls or moved insulation and created cold spots that can be seen in thermal. Nocturnal animals that emerge at night can also be tracked with thermal imaging, which works in complete darkness.

Hornets and Other Social Insects

Hornet nests show up as hotspots in thermal.

Insects are cold-blooded, but they do generate heat. The heat of a wasp nest, beehive, or other large cluster of social insects will usually generate enough heat to be detected by a thermal camera.

1) Connect the AMG8833 IR Camera Sensor Breakout to the Raspberry Pi using the following circuit connection -

balenaFin AMG8833 IR Camera

3V3 VIN

GND GND

GPIO 2 SDA

GPIO 3 SCL

GPIO 4 INT

2) Install Intel OpenVino Toolkit from this link .

3) Follow this tutorial to download and run demos on balenaFin connected with Intel Movidius Neural Compute Stick 2.

4) Or if you want to deploy this on a Intel PC then follow this guide made by me.

5) Collect thermal data from various resources available on Internet such as this. Or you can collect your own customised image data while using the IR Camera.

6) You can train your model in the OpenVino SDK itself or you can use Google Colab or Jupyter Notebooks to do so.

7) Then you can export this data to OpenVino SDK Development Platform using this tutorial and guide from OpenVino team.

Training the Model

PART 1 - Distinguishing between a Damselfly and a Butterfly

We are using Google's Teachable Machine to train the model which will be able to classify in between a Damselfly and Butterfly.

STEP 1 - First download the thermal images of butterfly and damselfly from our Google Drive.

Butterfly dataset

Damselfly dataset

STEP 2 - Go to https://teachablemachine.withgoogle.com/ and starting training the data as of below steps.

STEP 3 - Click on Image Project and then in the Class 1 and Class 2 boxes name them as Damselfly and Butterfly respectively as of below.

STEP 4 - Train the data by the images given in our Google Drive. You can either upload from the file or else u can use your webcam and then train by displaying the image from the mobile such as below.

STEP 5 - Then click on Train Model button present in the middle of the screen. Wait for sometime and then it will start training the model.

Classifying the Data

After successful training it will pop up and open your webcam for live classification such as below.

Then click on Export Model option present at the top of the screen. It will display the following screen when being clicked.

Then click on TensorFlow Lite option as shown below.

Congratulations! You have successfully trained and classified the model on web. Now its time to run it on the hardware.

Running on Hardware

Over here we are testing on two situations :

Running on balenaFin with Google Coral USB Accelerator

The Coral USB Accelerator uses Coral Edge TPU , thus increasing the model when running in TensorFlow Lite environment.

Step 1

Install the edgetpu library following Coral's official instructions

Step 2

pip install the following packages like so:

pip3 install Pillow opencv-python opencv-contrib-python

Step 3

Download model from TM2

Step 4

Use this code from our repository to run the model

Running on balenaFin with Intel Neural Compute Stick

Not completed because of limitations in both hardware and software :(

Limitations Faced while making this project

After reading this project post most of you will be asking us for the part where we are running on the NCS 2. Well we faced some severe scenarios while developing this. Here are some below -

1) NONE of us had a Thermal/IR Camera back at home. We asked in many places for the help by collecting some imagery data for us. None of the forums or the places did that. So I (Arijit) had to read 2 full Research papers to get some thermal imagery data of Damselfly and Butterfly. We only got 2 types of thermal images and we then reduced it brightness and all and then made it to around 650 images for training the model.

2) NONE of us including David W and Sahaj Sarup had a NCS. Finally we got help from David Tischler as he generously gave access to a Pi 4 connected with a Movidius stick (NCS 1).

3) We did make the model which we have even tested and it runs pretty fast and then the model was converted to TensorFlow Lite model.

4) We had very little time to spend with the Pi and also we faced difficulty in having some discussions and talks as we belonged from different time zones.

5) We tried for 2 constant days since what we received access to the Pi. But we couldn't make the model to convert to OpenVINO and make it up and running :(

Step-By-Step Explanation

1. As the first step we will scan up and take the data from the AMG8833 Camera Sensor Breakout board by connecting it to the Fin.

2. Performing the desired analysis using Intel's OpenVino software and it will get accelerated i.e. the fps(Frames Per Second) speed will gradually increase.

3. Then the region will be scanned fully after which the Fin will guide the farmer where to put up the insecticides for a better use.

Demonstration Video

You can also check and try out our model being hosted on Teachable Machine website here.

Features

1. Low in cost and can be used very easily.

2. No need of a technical background to use this device

3. Will enable correct usage of chemicals in farming.

4. Prevent harmful diseases cause by spraying unwanted chemicals.

Future Work

For further development we are thinking of using some high power cameras which will be able to even track up and classify pests even if they are flying. We are thinking of reducing the cost and even trying to make our model run on cheap Android phones with a cheap Thermal Camera connected with it which will help the poor farmers to easily buy and use this technology to enhance the human life.

-X-

Thank you for viewing our project. Don't forget to drop down any comments about queries or any questions if you have. We will try to answer them surely :)

Resources / References

https://www.balena.io/fin/

https://www.adafruit.com/product/3538

https://ark.intel.com/content/www/us/en/ark/products/140109/intel-neural-compute-stick-2.html

https://store.balena.io/collections/accesories/products/raspberry-pi-compute-module-3-lite-plus-cm3-l