COVID Security with OpenCV and TensorFlow

For Hack the North, my team and I thought of the brilliant idea of creating a device to mitigate those who do not have masks on. It uses machine learning to determine whether or not you have a mask on, and lights a respective LED depending on the result (green LED for yes!, and red LED for no). This device can be used as a friendly reminder to put a mask on, as a security system, or just a reminder for yourself while your leaving your house to take one!

Since we have already created the project, I'll link the devpost/github here:

https://devpost.com/software/mask-detector-xi4v7lhttps://github.com/sherwinchiu/mask-detector

In this link, our team cover more in-depth on what the device can be used for! I hope you view it and find it interesting.

My focus on this article though is to guide you how to implement OpenCV and TensorFlow on Python for Raspberry Pi! While researching on how to get this software working, my team found that a lot of sources online were very brief and did not describe the process very well. I want to break down this problem in steps, and very simply explain every line of code and how to use the library to your advice. Research took up about 20 hours for this project, so I hope by just simplifying everything we did in this article, you can replicate our results in about an hour or two.

Step 1: Planning

If you just want to get straight into it, you can skip this step. I will just be explaining the choices that I made. First things first, the language we chose was base off of needs. Since this was a hackathon, we chose to use Python since it is very simple and easily written language. It is also very compatible with the Raspberry Pi we chose. After that, we just had to figure out the proper packages to use alongside Python.

We knew that we needed some sort of software that could recognize a persons face, and another software that could recognize a mask object. This way, when combined together, we could very easily recognize someone without a mask, and someone with a mask. This is where OpenCV and TensorFlow come in. OpenCV is an open-source computer library that is very good at facial detection. By implementing OpenCV, we would be able to track someone's face! Exactly what we need. It uses a.xml that is gives us parameters for a frontal face detector. I didn't make it myself, and full credits go to Rainer Lienhart. You definitely should check it out from this GitHub repo from them. This entire project is due to their hard work! Thank you!

TensorFlow is an open-source machine-learning software, training neural-networks to react to different parameters given to it. This pretty much means we give it pictures of masks, and pictures of no masks, and it can distinguish which from which! It is very precise at tracking objects given enough training data. This is perfect for what we need!

By combining both OpenCV and TensorFlow, we can get achieve what we need to do. All we need is a picture to process, which is where a Raspberry Pi with a PiCamera comes in handy. But there's a catch: RaspberryPi3BIsSlow

TensorFlow has very high requirements hardware requirements, needing a strong CPU (Computer Processing Unit), GPU (Graphics Processing Unit), and lots of RAM (Random Access Memory). A Raspberry Pi 3B will be very slow trying torun TensorFlow on it. Our solution was creating a server using Flask, sending pictures from our Raspberry Pi 3B to our main computers that met those requirements. We would then process them using TensorFlow, then sending back False (for NO MASK), or True (for YES MASK). That way, we can skip around this hardware requirement!

Step 2: Raspberry Pi Package Installation

This tutorial is going to focus on installing packages on Raspberry Pi, not installing Raspberry Pi. Click here for an in-depth tutorial! We need to install the basic software we will be using to code both OpenCV and TensorFlow. This includes Python3.7+, which is the programming language used, and Pip19.0+, which stands for Pip Installs Packages. This will act as a package manager, making it very easy to install any python package that we want to use. Linux makes it very easy to install software online, and is a very powerful tool. Get used to using these commands, because they are very important! (don't copy anything after the #)

sudo apt-get install python3        # Installing programming language used!
sudo apt-get install python3-pip    # Installing package manager
sudo apt install python3-opencv     # Install OpenCV for Python3
sudo apt-get update                 # Update the packages to make them up-to-date
sudo apt-get upgrade                # Get any upgrades needed for those packages
pip3 install requests               # Install requests for client

Using these commands, we have imported the framework of what we will need to get our code working!

Step 3: Raspberry Pi Code

It's time! I will be taking pictures with my Raspberry Pi and sending them over to the server for processing.

First, taking pictures and storing them with my Raspberry Pi Cam and Raspberry Pi. Implementation is pretty easy:

# camera.py
from pycamera import PiCamera        # Imports PiCamera library
from time import sleep               # Imports sleep library
import os                            # import OS library

def takePicture():
    # Setup camera object in termms of PiCamera to constastly call 
    camera = PiCamera()
    # Allow camera to get enough light for aperture
    sleep(0.5)
    # Capture the image
    camera.capture("image.jpeg")
    # Don't forget to close the camera every time!
    camera.close()

Second, getting the Facial Detection working with OpenCV. I will link the.xml file here for your use:

# face.py

import cv2             # Imports the OpenCV library 
import sys             # sys is a basic library from python

def detectFace():      # Defining function for facial detection
    # Set the cascade path with the features to recognize faces
    cascadePath = "haarcascade_frontalface_default.xml"  
    # Load the image and returns it to store in image
    image = cv2.imread(image.jpeg) 
    # Classifies the cascade class for facial image processing
    cascade = cv2.CascadeClassifier(cascadePath)
    # Detect all faces in the pictures -- more on arguments here 
    faces = cascade.detectMultiScale(image, 
        scaleFactor=1.1, 
        minNeighbors=5, 
        minSize=(50, 50))
    # Show faces (for testing)
    for (x, y, w, h) in faces:
        cv2.rectangle(image, (x, y), (x+w, y+h), (255, 0, 0), 1)
    cv2.imshow("test", image)
    cv2.waitKey(0)
    # Return the number of faces that are detected in the picture
    return len(faces)

Here's the code! The return statement then returns the number of faces. If we use the test code (not recommended for final product because it will not run), we can test if it works! This is how it looks for m.

We know that if the number of faces is greater than one, we can say there is a face in the picture! Perfect. This is all we need to detect a face in a picture. Now, we just need to send this information to the server. Hopefully, you are port forwarded (to allow the world to view your IP through a port) or use ngrok, a free tool to open your IP to the world. If not, tutorials for both are here: Port Forwarding Ngrok

# client.py

from __future__ import print_function     # Importing print_function
import requests                           # Importing requests to send requests 
import json                               # Importing json file format
import cv2                                # Import cv2

def sendRequest():
    ip = 'whatever your ip is'            # Specify your IP
    port = 'whatever port you opened'     # Specify which port you opened
    address = 'http://'+ip+':'+port       # Entire address 
    content_type = "image/jpeg"           # Content Type
    headers = {'content-type' : content type} # Sending content type over (image/jpeg)    
    img = cv2.imread('image.jpeg')            # Reads the image.jpeg
    _, img_encoded = cv2.imencode('.jpg',img) # Encodes image.jpeg into pyArray
    # Sends a POST request to server and waits for a response...
    response = requests.post(addr, data=img_encoded.tobytes(),headers=headers)
    # Returns what the server responds with. In this case, a 0 (for NO MASK) or 
    # a 1 (for YES MASK)
    return json.loads(response.text)

The client is all set up now! All we need to do is have a main function that uses all this code for the Pi to keep running!

# main.py
import face
import camera
import client

def main():
    print ("Facial Detection V. 1")
    # Keep on repeating
    while True:
        camera.takePicture() # Takes a picture and saves it to image.jpeg
        # If there is more than one face detected in the picture...
        if face.detectDetect() > 0:
            # Send request of picture to server, if server returns 1, then mask exists
            if client.sendRequest() == 1:
                print ("Mask detected. Thank you.")
            # If response is anything other than 1, no mask exists            
            else:
                print ("No mask detected. Please put one on.")
        # If no face detected, then skip checking for facial mask
        else:
            print ("No face")
        
# Automatically start the main function
if __name__ == "__main__":
    main()

There we go! All client code for the Raspberry Pi has been written. If you want everything to be done on the Raspberry Pi, you can also try implementing TensorFlow Lite on Raspberry Pi. All you have to do is assemble it together (putting on camera) and it'll look something like this:

Step 4: Computer Installation (Linux)

These steps will exist if you have a Linux computer. If not, you can try to use Chocolatey to download packages on Windows here. You can also use a setup Keras using a virtual environment with this tutorial, which gives you many benefits for working on multiple Python projects. Since I used a virtual machine for this project I am not that concerned about that, so I skipped this. These are the packages you will have to install for the server.

sudo apt-get install python3        # Installing programming language used!
sudo apt-get install python3-pip    # Installing package manager
pip3 install numpy                  # Install numpy - managing lists
pip3 install flask                  # Install flask - managing server
pip3 install jsonpickle             # Install jsonpickle - formatting
pip3 install tensorflow             # Install TensorFlow
# Install dependences for keras
pip3 install numpy scipy
pip3 install scikit-learn
pip3 install pillow
pip3 install h5py
pip3 install keras                  # Installing Keras, our machine learning wrapper
pip3 install image
sudo apt-get update                 # Update the packages to make them up-to-date
sudo apt-get upgrade                # Get any upgrades needed for those packages

That should be it for setting up! Now, time to write the server code! It is very long. I will link the keras_model.h5 that we made in the files, but also link the website we used to produce the model. Remember, the keras_model.h5 is the file that contains the machine learning algorithm that can tell whether or not a face has a mask or not.

# server.py

# A lot of imports for a lot of use
from flask import Flask, request, Response
import jsonpickle
import numpy as np
import cv2
import tensorflow.keras
import tensorflow
from tensorflow import keras
from PIL import Image as im, ImageOps
# Initialize the Flask application
app = Flask(__name__)
def detectMask():
    # Disable scientific notation for clarity
    np.set_printoptions(suppress=True)
    # Load the model
    model = keras.models.load_model("keras_model.h5")
    data = np.ndarray(shape=(1, 224, 224, 3), dtype=np.float32)
    image = im.open(image_received.jpeg)
    # resize the image to a 224x224 with the same strategy as in TM2:
    # resizing the image to be at least 224x224 and then cropping from the center
    size = (224, 224)
    image = ImageOps.fit(image, size, im.ANTIALIAS)
    # turn the image into a numpy array
    image_array = np.asarray(image)
    # display the resized image
    image.show()
    # Normalize the image
    normalized_image_array = (image_array.astype(np.float32) / 127.0) - 1
    # Load the image into the array
    data[0] = normalized_image_array
    # run the inference
    prediction = model.predict_proba(data)
    if a[0][0] > 0.5:
        return 0
    else:
    return 1
# route http posts to this method
@app.route("", methods=["POST"])
def test():
    r = request
    # convert string of image data to uint8
    nparr = np.fromstring(r.data, np.uint8)
    # decode image
    img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
    new = im.fromarray(img)
    new.save("image_received.jpeg")
    prediction = detectMask()
    # build a response dict to send back to client
    response = prediction
    # encode response using jsonpickle
    response_pickled = jsonpickle.encode(response)
    return Response(response=response_pickled, 
    status=200, 
    mimetype="application/json")
# start flask app
app.run(host="127.0.0.1", port=11111) # Make your port whichever one you opened

So what's happening in this server prompt? To explain is as simply as possible, there are two functions. One is the test() function, which tests for a request and waits until it receives an image. Upon receiving the image, it takes it and decodes it into a real image. Afterwards, it runs the detectMask() function. All this function does is uses TensorFlow and the model that we created to check if there is a mask or not. It uses percentages to show how close it is to having a mask, and not having a mask. Depending on the results, if we're more than 50% sure they're wearing a mask, we send back a 1 to the server. The server then returns whatever number it gets back to the client.

Step 5: Conclusion

There you go! A fully functioning facial recognition application in real time. I hope that you are able to use this for whatever project you have, and that I made it a little bit simpler. Feel free to ask any questions/make fun of how bad the code is. It was really rushed and I hope I could teach you as much as I knew! Thank you guys for reading.