Created February 2, 2020 © GPL3+

Auto Recyclables Sorting Machine based on A.I.

Make recycling FUN and the recycling industry GREAT again by accurately sorting recyclables and filtering out inbound contamination.

IntermediateFull instructions provided106

Auto Recyclables Sorting Machine based on A.I.

Things used in this project

Hardware components

NVIDIA Jetson Nano Developer Kit

Raspberry Pi Camera Module V2

DFRobot 7'' HDMI Display with Capacitive Touchscreen

Jetson Nano Metal Case

NEMA17 Stepper Motor 0.89N-m

The torque of the motor is proportionate with the size of the turning mechanism that is used. That is, the larger the board or the longer the shaft, the higher the torque. The number of motor sets(stepper motor, stepper motor driver, stepper motor mount, coupler, shaft, shaft mounting bracket, 5 x boards, 2 x 3D-printed board-to-shaft connector) depends on the number of categories you would like to sort. The sorting is binary, so if there are 3 types of recyclables to sort, two sets of motor sets are required.

Stepper Motor Driver DM320

5mm-8mm Coupler

8mm Diameter Shaft 500mm

Stepper Motor Mount for NEMA17

Shaft Mounting Bracket for 8mm Shaft

4mm Acrylic Board

Drawings attached: one circular board, four identical rectangular boards. Other materials, such as wood or metal are also possible depending on the use.

5mm Acrylic Board

Drawings attached. For the casing of AI processing modules (Jetson Nano and camera)

3D-Printed Parts

.stl file attached. Board-to-Shaft Connector, 4040-to-MotorMount Adaptor, 4040-to-ShaftMount Adaptor

Metal frame

Any frame will do. Here we use 4040 Aluminium extrusion for flexibility: 1400mm x 4pcs 1350mm x 5pcs 520mm x 6pcs Connectors with T-nuts and M6x16mm bolts x 30sets CAD model is attached.

Bolts and Nuts

For each motor set: M3x5mm 2pcs for each 4040-MotorMount Adaptor M4x8mm 2pcs for each 4040-ShafyMount Adaptor M4x12mm + M4 nuts 4 sets for each stepper motor mount M6x15mm + M6 nuts 8 sets for every two board-to-shaft connectors

Arduino Mega 2560

Other Arduino board is also applicable depending on the number of motors and sensors used.

12V 30A power supply

Can use lower current ones depending on the number of stepper motors you plan to use

USB-A to B Cable

To connect Jetson Nano to Arduino

Audio / Video Cable Assembly, Ultra Slim RedMere HDMI to HDMI

For 7" touch screen

Jumper wires (generic)

To connect stepper motor driver to Arduino

Digilent 2-pin MTE Power Cable

To power Arduino (connect to 12V power supply)

Electric Cable

For power supply, connect to stepper motor driver

Ultrasonic Sensor - HC-SR04 (Generic)

Software apps and online services

LabelImg

Amazon Web Services AWS EC2

YOLO

Hand tools and fabrication machines

Allen keys

M3, M4 and M6 socket head

3D Printer (generic)

For 3D printing customised parts

Laser cutter (generic)

For laser cutting acrylic boards

Breadboard, 270 Pin

For easy plug and insert during testing

Multitool, Screwdriver

Leveling Ruler

Story

1. Why We Started the Project

Driven by the rapid growth of population and economic activities, waste becomes a severe issue worldwide. According to a World Bank report, global annual waste generation is expected to surge by 70% to 3.4 billion tonnes over the next 30 years. The world economy and technology are advancing every day but the problem of waste is deteriorating at an even faster rate. What happens?

In face of this pressing issue, everyone knows "3-R" - Reduce, Reuse, Recycle. We are encouraged to minimise our consumption of goods and waste footprint, to reuse products until the end of the product cycle, and to put into the right bins for recycling. What's next?

More importantly, when looking into the current way of recycling in the society (at least in Hong Kong where GrinBean is founded), it is simply inefficient and, ironically, not helpful to the problem we are faced with. Research from the World Economic Forum suggests that even for the country with the best recycling rate, only below 60% of waste are recycled properly - not to mention some other places such as Hong Kong with only 24%. What to do?

GrinBean is founded to change the status quo. An AI-powered recycling hardware is developed with a vision to boost environmental sustainability. By innovating waste management solutions, educating and taking the community to the next level beyond 3-R, and streamlining waste management to raise recycling rate, GrinBean aims to make recycling accessible everywhere.

We begin with the smallest thing around us - rubbish bin. With the help of image recognition, AI and mechatronics design, GrinBean develops a smart recycling machine, that improves itself while operating, to accurately sort recyclables and filter out inbound contamination.

2. Computer Vision Model Training

Model Training with NVIDIA Jetson Nano

Rather than just inference the model on edge, training on edge is our target, so we tried to train a model with Jetson Nano and at the same time, we train another model on Colab with more computing power.

Step 1: Preporation

Settings on Jetson Nano

With many requirements preinstalled in Jetson Nano, It makes our work much easier. For example, CUDA, openCV and cuDNN are built-in. However, we still do some modifications on the software of Jetson Nano so that it can proceed training and have better detection.

Due to limited amount of RAM in Jetson Nano, an 8G swap file is set on Jetson Nano so it will not run out of memory during training or processing.

The pre-installed opencv-3.3.1 on Jetson Nano does not support gstreamer functionalities. While we are using it for Picamera streaming, we update opencv to 3.4.8 which also gives higher-quality streaming.

Preparing Datasets

Images of recyclable were taken manually one-by-one with the aid of a self-made lightbox to create a uniform shooting environment for producing high-quality data for model training. All the images have a black background color, which matches the environment where detection is carried out inside our recycling machine.

1 / 2 • Images for AI model training

About 4000 recyclables covered including aluminum cans, steel cans, PET bottles, HDPE bottles, cardboard, white paper and color paper. With data augmentation, about a total of 12,000 images are prepared for training.

As we aimed to produce an object recognition model, image annotation is required. The labeling and drawing of the object bounding boxes of images were done using LabelImg, an open-sourced graphical image annotation tool. (More about LabelImg: https://github.com/tzutalin/labelImg)

Graphical user interface of LabelImg

Step 2: Model Training

We are using darknet framework to train YOLOv3. It is dedicated to real-time object detection. (More info about YOLO model training could be found: https://pjreddie.com/darknet/yolo/, https://github.com/AlexeyAB/darknet, and https://jkjung-avt.github.io/ ) By following the configuration in AlexeyAB/darknet repo, we train a yolov3-tiny model with 3 classes (Paper, Can, Bottle).

Training Statistics with 1258 data

Graph of training average loss vs Iteration (@5500 iterations)

On the other hand, we are training another model in Colab with the SAME set of images but different categorisation and configuration. It trains with a full YOLOv3 model with 12 classes.

The training process was also done on AWS EC2 P2 8xlarge Instance. The model training ran through a total of 500 epochs, that took roughly 3 days with aids of NVIDIA GPUs. A PyTorch weight was produced as a result. The outcome was taken as reference for comparison.

AI model training using AWS

Step 3: Testing

Mean Average Precision (mAP) is our key indicator for the model testing. It is associated with IoU which is set to 0.5.

Comparison of mAP@0.5

Comparison of average IoU

**Although they are trained under different configurations, It shows a reference for the average performance on the same image set.

It is found that the model with 4000 iterations performs the best in mAP.

Screenshot of testing outcomes:

-Testing for YOLOv3-tiny model:

Testing for 2000 iterations

Testing for 4000 iterations, which is the best model

-Testing for full YOLOv3 model:

Testing for 6000 iterations, which is the best model

Step 4: Applying

With the models trained, the best YOLOv3 model trained in Colab, which has tested for 6000 iterations, was used.

Detection of a PET bottle

Detection of a white paper

The result of the YOLOv3-tiny model is not satisfying that it can’t detect objects accurately. Although the testing statistics show good results, yet in real-world applications, it needs more improvement.

It could be due to the environment and the limitation of the YOLOv3-tiny model. We will keep trying different models and feed in more data for the model.

3. Coordination with NVIDIA Jetson Nano

NVIDIA Jetson Nano is the core processor for our sorting machine that provides a graphical user interface for machine users, detects the recyclables with our trained model and finally processes the items.

Part 1: User Interaction

We have built a desktop application for user interaction that runs on Jetson Nano. When users start recycling, users can put anything into the machine. We will have the detection process running in the backend while the user interface on the machine screen shows the detection result. Users can save their recycling effort record with their phone number or scan QR code with our mobile application and all data will be stored in our database.

Part 2: AI Object Recognition

When the machine is initialised, the AI object recognition programme will be started and running in the background continuously with a camera on. When a recyclable is inserted into the machine, our pre-trained model will classify the object. If the result is any of our predefined classes, the result will be returned to the desktop application. Based on the result, the sorting process is triggered.

1 / 2 • recyclable type classification achieved by image processing

Part 3: Hardware Control

When a valid recyclable is detected, the desktop application will send signals to Arduino and it will control all hardware movement in order to sort the recyclable into the designated bin. When this cycle has completed, it is ready to sort the next recyclable.

4. Electronics

Step 1: Set up the Jetson Nano

Instructions can be found here: https://www.waveshare.com/wiki/Jetson_nano_case_(B)

1 / 2 • Physically connect all the components, including camera and touch screen

Step 2: Connect all the components according to the schematic below:

1 / 2 • Schematic Diagram

Step 3: Test Serial Communication with Arduino

Connect Arduino to Jetson Nano and run the code, check:

1. If the motors are turning at desirable speed and degree when Serial command was sent. If not, check the switches on the side of DM320 which can change the speed and step interval of the motors (details can be found in the documentation of DM320: https://www.omc-stepperonline.com/download/DM320T.pdf, more information about controlling stepper motor with Arduino can be found here: https://forum.arduino.cc/index.php?topic=627348.0);

2. If the ultrasonic are sending back correct data, i.e. move the sensor around and verify the distance.

If everything is working fine, proceed to Part 5.

5. Hardware - Assembling the Machine

The machine mainly consists of (1) the AI and processing module, (2) the mechanical sorting module, (3) sensors, (4) the machine frame and interior structures and (5) optional parts.

AI and processing module

The AI and processing module, consisting of an NVIDIA Jetson Nano connecting to a camera and a touchscreen, is placed next to the machine opening where people put in recyclables. Therefore, when a recyclable is inserted, the camera above can capture its image for image processing and triggering the sorting process. The module is cased in acrylic boards of 5mm thickness for rigidity.

Photo of Casing installed in place above Motor Set

Mechanical Sorting Module

Underneath the AI and processing module is the mechanical sorting module, where the 2 layers of mechanical sorters enable recyclables to be sorted binarily for 2 times, and subsequently sorted into 4 different bins (for plastics, paper, metals, and trash). Please refer to the SolidWorks CAD model and the video:

Video showing the sequence

1 / 2 • CAD model of Motor Set

Motor Turning Test

Sensors

Sensors are installed in various locations, ultrasonic sensors are placed above storage bins to detect their remaining capacities. Information about HC-SR04 (the ultrasonic sensor we use) could be found here: https://howtomechatronics.com/tutorials/arduino/ultrasonic-sensor-hc-sr04/.

Other sensors such as infrared sensors could be installed next to the machine opening for detection of the presence of recyclable.

Sensors are fixed to the frame with tape and Blu Tack.

Machine Structure

The metal frame of the machine is constructed with aluminum extrusions, which serves as the main structure of the machine that allows acrylic boards and other modules to be attached.

When putting 4040 Al extrusions together, be careful with the leveling and make sure they fastened firmly.

Bare Aluminium Frame

It is a bit tricky when installing the motor module. First, fasten the 3D printed part to the motor mount/shaft mount, force is required when necessary because the 3D printed part is not threaded. The most convenient way is to turn the screw directly into the 3D printed part. Otherwise, other options include using a tap drill or use heat insert of the respective thread. More information can be found here: https://markforged.com/blog/heat-set-inserts/.

Then, fasten the 3D printed part along with the Motor Set to the Frame.

1 / 3 • Frame with Motor Sets installed

Optional Parts

The following are suggested additional parts (not included in the material list) which would not affect functionality if not adopted: a) Machine exterior can be of acrylic or plates of other materials, fixed with bolts and nuts; b) A pair of doors can be added in the front, allowing janitors to collect recyclables from the machine easily; c) A waterproof box to house all electronic components including the power supply and Arduino, for better protection against dirt or moisture.

Final Test

Final Test before Covering up the Frame

Putting Everything Together

Lastly, the machine should look something like this:

Outlook of machine

Schematics

Code

Arduino code

Arduino code

Arduino

For controlling stepper motors and communicating with Jetson Nano
The code is for sorting 4 types of recyclables, which means, 3 sets of motors and 4 ultrasonic sensors.

When Arduino receives Serial input from Jetson Nano (P for Paper, B for Bottle, C for Can, R for Rubbish), it will control respective motors to turn and use gravity to transfer the item to the correct bin. Once the ultrasonic sensor which is installed at the rim of the bin, detects a distance of less than 5 cm, it will print out warning message.

//defines pins
//Master turnstile motor
const int stepPin = 10;  //PUL -Pulse
const int dirPin = 11; //DIR -Direction
const int enPin = 12;  //ENA -Enable
//Secondary left motor
const int LstepPin = 2;  //PUL -Pulse
const int LdirPin = 3; //DIR -Direction
const int LenPin = 4;  //ENA -Enable
//Secondary right motor
const int RstepPin = 6;  //PUL -Pulse
const int RdirPin = 7; //DIR -Direction
const int RenPin = 8;  //ENA -Enable

//Ultrasonic sensors
const int PtrigPin = 15;  //trigger pin for paper bin
const int PechoPin = 16;  //echo pin for paper bin
const int CtrigPin = 18;  //trigger pin for paper bin
const int CechoPin = 19;  //echo pin for paper bin
const int BtrigPin = 22;  //trigger pin for paper bin
const int BechoPin = 23;  //echo pin for paper bin
const int RtrigPin = 26;  //trigger pin for paper bin
const int RechoPin = 27;  //echo pin for paper bin

int motorpulse = 1200;


//IR sensor
#define IRpin A0            //IR singal pin

//classification of recyclables
char item;

// defines variables
long USduration;
int USdistance;
unsigned long US_int;
unsigned long US_frac;



void setup() {
  Serial.begin(9600);
  
  //Sets the pins as Outputs
  pinMode(stepPin,OUTPUT); 
  pinMode(dirPin,OUTPUT);
  pinMode(enPin,OUTPUT);
  digitalWrite(enPin,HIGH);
  
  pinMode(LstepPin,OUTPUT); 
  pinMode(LdirPin,OUTPUT);
  pinMode(LenPin,OUTPUT);
  digitalWrite(LenPin,LOW);
  
  pinMode(RstepPin,OUTPUT); 
  pinMode(RdirPin,OUTPUT);
  pinMode(RenPin,OUTPUT);
  digitalWrite(RenPin,LOW);

  pinMode(PtrigPin, OUTPUT); // Sets the trigPin as an Output
  pinMode(PechoPin, INPUT); // Sets the echoPin as an Input
  pinMode(BtrigPin, OUTPUT); // Sets the trigPin as an Output
  pinMode(BechoPin, INPUT); // Sets the echoPin as an Input
  pinMode(CtrigPin, OUTPUT); // Sets the trigPin as an Output
  pinMode(CechoPin, INPUT); // Sets the echoPin as an Input
  pinMode(RtrigPin, OUTPUT); // Sets the trigPin as an Output
  pinMode(RechoPin, INPUT); // Sets the echoPin as an Input

  pinMode(IRpin, INPUT); // Sets the IRPin as an Input
  
  Serial.println("Program start!");

}

void loop() {
  
  //detection of motion to trigger camera
  IRdetect();

  //detection of bin level
  USdetect();
  
  if (Serial.available())
  {
   
    //input from camera/user to tell which recyclables
    item = Serial.read();
    Serial.println(item);
    switch (item){
      case 'P':
         
         LCW();
         CCW();
         delay(1000);
         LCCW();
         
      break;
      
      case 'C':
         CCW();
         delay(1000);
      break;
      
      case 'B':
         
         RCW();
         CW();
         delay(1000);
         RCCW();
         
      break;
      
      case 'R':
         CW();
         delay(1000);
      break;
    }    
              
  }
  else
    Serial.println("Pending for input...");
    delay(100);

  Serial.println("----------End of loop----------\n\n");

}

//function for Master turnstile motor to go CLOCKWISE, pulse decide how much it turns
void CW()
{
  //Enables the motor direction to move
  digitalWrite(enPin, LOW);
  digitalWrite(dirPin,HIGH);
  Serial.println("Master Going clockwise");
  //400 Pulses for making one full cycle rotation
  for(int x = 0; x < motorpulse; x++){
    //Serial.println(x);
    digitalWrite(stepPin,HIGH); 
    delayMicroseconds(50); 
    digitalWrite(stepPin,LOW); 
    delayMicroseconds(50); 
  }
  //digitalWrite(enPin, HIGH);
}

//function for Master turnstile motor to go ANTI-CLOCKWISE, pulse decide how much it turns
void CCW()
{
  //Enable and change the rotations direction
  digitalWrite(enPin, LOW);
  digitalWrite(dirPin,LOW);
  Serial.println("Master Going anti-clockwise");
  //400 pulses for making one full cycle rotation
  for(int x = 0; x < motorpulse; x++) {
    //Serial.println(x);
    digitalWrite(stepPin,HIGH);
    delayMicroseconds(50);
    digitalWrite(stepPin,LOW);
    delayMicroseconds(50);
  }
  //digitalWrite(enPin, HIGH);
}

//function for Left motor to go CLOCKWISE, pulse decide how much it turns
void LCW()
{
  //Enables the motor direction to move
  digitalWrite(LdirPin,HIGH);
  Serial.println("Left motor going clockwise");
  //400 Pulses for making one full cycle rotation
  for(int x = 0; x < motorpulse; x++){
    //Serial.println(x);
    digitalWrite(LstepPin,HIGH); 
    delayMicroseconds(500); 
    digitalWrite(LstepPin,LOW); 
    delayMicroseconds(500); 
  }
  //One second delay
  delay(1000);
}

//function for Left motor to go ANTI-CLOCKWISE, pulse decide how much it turns
void LCCW()
{
  //Changes the rotations direction
  digitalWrite(LdirPin,LOW);
  Serial.println("Left motor going anti-clockwise");
  //400 pulses for making one full cycle rotation
  for(int x = 0; x < motorpulse; x++) {
    //Serial.println(x);
    digitalWrite(LstepPin,HIGH);
    delayMicroseconds(500);
    digitalWrite(LstepPin,LOW);
    delayMicroseconds(500);

  }
  //One second delay
  delay(1000);
}


//function for Right motor to go CLOCKWISE, pulse decide how much it turns
void RCW()
{
  //Enables the motor direction to move
  digitalWrite(RdirPin,HIGH);
  Serial.println("Right motor going clockwise");
  //400 Pulses for making one full cycle rotation
  for(int x = 0; x < motorpulse; x++){
    digitalWrite(RstepPin,HIGH); 
    delayMicroseconds(500); 
    digitalWrite(RstepPin,LOW); 
    delayMicroseconds(500); 
  }
  //One second delay
  delay(1000);
}

//function for Right motor to go ANTI-CLOCKWISE, pulse decide how much it turns
void RCCW()
{
  //Changes the rotations direction
  digitalWrite(RdirPin,LOW);
  Serial.println("Right motor going anti-clockwise");
  //400 pulses for making one full cycle rotation
  for(int x = 0; x < motorstep; x++) {
    digitalWrite(RstepPin,HIGH);
    delayMicroseconds(500);
    digitalWrite(RstepPin,LOW);
    delayMicroseconds(500);

  }
  //One second delay
  delay(1000);
}


void USdetect() {  

  for (int n = 0; n < 4; n++)
  {
    int trigPin;
    int echoPin;
    switch(n)
    {
      case 0:
        trigPin = PtrigPin;
        echoPin = PechoPin;
      break;

      case 1:
        trigPin = CtrigPin;
        echoPin = CechoPin;
      break;
      
      case 2:
        trigPin = BtrigPin;
        echoPin = BechoPin;
      break;
      
      case 3:
        trigPin = RtrigPin;
        echoPin = RechoPin;
      break;
      
    }
    Serial.print(trigPin);
    // Clears the trigPin
    digitalWrite(trigPin, LOW);
    delayMicroseconds(2);
    
    // Sets the trigPin on HIGH state for 10 micro seconds
    digitalWrite(trigPin, HIGH);
    delayMicroseconds(10);
    digitalWrite(trigPin, LOW);
    
    // Reads the echoPin, returns the sound wave travel time in microseconds
    USduration = pulseIn(echoPin, HIGH);
  
    if((USduration < 60000) && (USduration > 1))
    {
      // Calculating the distance
      USdistance= USduration*34/2;
      US_int=USduration/100;
      US_frac=USduration%100;
      
      // Prints the distance on the Serial Monitor
      Serial.print(" Ultrasonic Distance: ");
      Serial.print(US_int, DEC);
  
      if (US_frac<10)
      {
        Serial.print(".");
        Serial.print(US_frac, DEC);
      }
      else
      {
        Serial.print(".0");
      }
      Serial.println (" cm");
      delay (500);
    }
    if (US_int <= 5)
    {
        Serial.println("Warning!!");
        Serial.print(n);
        Serial.println("Bin is almost full!");
    }
    
    USduration = 0;
    Serial.println(n);
  }
  
}

Credits

Grin Bean

1 project • 0 followers

JAmes Lam

1 project • 2 followers

Engineer during weekdays, maker during weekends

Thanks to Glenn Jocher, Tom Leung (member of GrinBean team), Archie Loong (member of GrinBean team), Ivan Lee (member of GrinBean team), and AlexeyAB.

Auto Recyclables Sorting Machine based on A.I.

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

Custom parts and enclosures

Rectangular 4mm Acrylic Board

Circular 4mm Acrylic Board

5mm Acrylic Board

3D Printed Board-to-Shaft Connector

3D Printed 4040-to-MotorMount Adaptor

3D Printed 4040-to-ShaftMount Adaptor

Assembly File (SolidWorks 2018)

Schematics

Electronics Connections Schematic Diagram

Code

Arduino code

Machine UI

Credits

Grin Bean

JAmes Lam

Comments

Embed the widget on your own site

Auto Recyclables Sorting Machine based on A.I.

Auto Recyclables Sorting Machine based on A.I.

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

Custom parts and enclosures

Rectangular 4mm Acrylic Board

Circular 4mm Acrylic Board

5mm Acrylic Board

3D Printed Board-to-Shaft Connector

3D Printed 4040-to-MotorMount Adaptor

3D Printed 4040-to-ShaftMount Adaptor

Assembly File (SolidWorks 2018)

Schematics

Electronics Connections Schematic Diagram

Code

Arduino code

Machine UI

Credits

Grin Bean

JAmes Lam

Comments

Related channels and tags