artik.io • Posted by meilily Li

Published June 26, 2018

Facial Recognition with ARTIK Eagleye 530s

To ensure the family security, the ARTIK team developed a security system prototype for boards using the Kairos face recognition service.

IntermediateFull instructions provided1,479

Facial Recognition with ARTIK Eagleye 530s

Things used in this project

Hardware components

Seeed Studio Eagleye 530s

OV5640 5M Auto Focus USB Camera

10.1” PCAP Touch Screen

Software apps and online services

Microsoft Windows 10

Python Qt (pyqt)

Hand tools and fabrication machines

hexagon plastic box

Story

Samsung ARTIK team had made a Face/Identity Detection System built with Eagleye 530s in April, 2018 (more details about the demo, check out here). Last month, the team made a new upgraded version of the demo for the IoT World in Santa Clara Convention Center which took place on May 14-17. They used an Eagleye 530s, an OV5640 5M Auto Focus USB Camera and a touchscreen to build up a security system to ensure family security.

When a person comes to the touchscreen, he/she could follow the instruction on the screen to take a selfies, and the photo will be uploaded to the Face Recognition cloud – Kairos. The system will detect if the person matches with the already enrolled faces in the system.

The system could be connected to door latches to unlock the door automatically only when a family member is detected. It would be a end-user solutions for smart home/ building.

Here comes the full instruction from the Samsung ARTIK team:

We introduced a prototype of Face/Identity Detection system when Eagleye 530s was just released in April, 2018. https://developer.artik.io/documentation/artik/projects/facial-recog.html

We then improved that project and added a touchscreen in the system and made it more appealing.

In this project, we have 3 main components:

Eagleye 530s;

OV5640 5M Auto Focus USB Camera;

10.1” PCAP Touch Screen or other touch screens available from Samsung ARTIK IO marketplace.

We remove the GrovePi+ accessories from the previous project and there is no Pi HAT/Shield being used. We also replace the MIPI camera with a USB camera coming with the same type of sensor.

The final assembly would look like this. It is mounted to a specially designed hexagon plastic box for events purpose.

Hardware Connections

Connect OV5640 USB camera to Eagleye 530S top USB port. (There are 2 USB ports on Eagleye. Use the top one for the camera; use the bottom one for the touch screen.)

Connect Touch Screen mini USB to Eagleye 530S bottom USB port.

Connect Touch Screen HDMI to Eagleye 530S HDMI port.

Connect the power cables of Eagleye 530S and Touch Screen.

Connect internet cable (CAT 5) to Eagleye 530S if you want to use wired internet.

Software Prerequisites

Install x-windows. The main application will run in x-windows environment. Please refer the following instructions.

https://developer.artik.io/documentation/developer-guide/multimedia/display.html

Install Python Qt (pyqt). The application is developed with Qt.

apt-cache search pyqt
sudo apt-get install python-qt4

Install OpenCV. The video streaming and picture taking are based on OpenCV.

sudo apt-get install libopencv-dev python-opencv

Install Watchdog. There is a ‘watchdog’ program watch_for_changes.py watching if the jpg file changed. That means if the camera takes your picture and saves it to the current folder as a jpg file, watch_for_changes.py will call validate.py and do the picture validation process.

pip install watchdog

Software Workflow

Create a bash file watch.sh. In the file there are two lines. (See the attached python watch_for_changes.py for the content)

startx & python watch_for_changes.py

Create a bash file start.sh which has one line. (See the attached videowindow.py for the content)

python videowindow.py

You need to run ./watch.sh & from the console; when x-window starts up from the touch window use keyboard shortcut Ctrl+Alt+t to start a LX terminal there.

From the LX terminal run ./start.sh & to start the main program videowindow.py. You will see the application pops up as the following.

Pull out the keyboard USB cable from the USB port and push in the USB cable from the touch screen to the port. From now on the rest of the jobs would be done through the touch screen.

Press Start button from the application and start the video streaming. You will see yourself in the left window.

Press Take Picture button and your picture will be taken and transferred to the Face Recognition cloud – Kairos. Your picture will in the right window. When the validation result comes back you will see from the right screen if it is “Successful” or “Failed”.

You could Enroll the last taken picture to the Cloud for the next validation.

Code Explanations

Videowindow.py

This python code is implementing the core functionalities of the system. It is for setting up the application, streaming video, taking a picture, calling the validation function and checking the validation result.

1. Import necessary PyQt4, OpenCV, watchdog and other libraries.

2. Capture the video from Video port 0. (If you use MIPI camera with Eagleye 530S the Video port should be 6.)

3. Create a Qt application and an instance of Window which hosts the application.

4. Define the Window. Give the name of the window as Control Panel. Define the 4 buttons (Start, Take Picture, Enroll and Quit) in the window. Set up the background image.

From 5 to 8 are the definitions of the four buttons.

5. When click Start button, the watchdog starts to observe the txt files defined in MyHandler function.

6. Thereafter, starts the video streaming in Video window.

7. When click Take Picture button, a frame of the current video streaming will be saved as original.jpg; the Validating Face wording will be put on the bottom left of the image; the image will be shown in the Photo window.

8. When click Enroll button, the enroll.py will be called and the previously taken image will be enrolled to the gallery on the Cloud side. This part will be explained in part d.

9. Define the detection pattern for the watchdog. In this case, when the validation result from Cloud comes back, it will be saved to a txt file – Detection.txt. Thus, we detect this file.

10. When Detection.txt gets modified, it gets opened and parsed.

11. Put either “Successful!” or “Failed!” wording on the image taken previously.

watch_for_changes.py

This code checks if jpg pictures get changed. Specifically, when a jpg picture is taken and saved to the current folder, this python code will call validate.py.

1. Set up the watchdog pattern as jpg files. All the jpg files creation or modifications will trigger an action – call validate.py.

2. Create an observer and start to monitor the changes.

validate.py

This code sends the encoded picture to Face Recognition platform – Kairos for the validation. The whole returned result will be saved to Result.txt. Then it will be parsed to the Detection.txt.

1. Encode the original.jpg to base64 format and give a gallery name to the “values” field.

2. Create headers – include app_id/app_key pair created at Kairos.com (https://developer.kairos.com/admin). Send request to Kairos web service.

3. Get the validation result and save it to Result.txt.

4. Open Result.txt and parse it.

5. Save the parsed result to Detection.txt. It could be either success or failure.

enroll.py

Of course, initially there is no image in your gallery. You need to enroll some images.

1. Encode the picture just taken to base64 and give it a subject_id. Also indicate which gallery you want to enroll the image to.

2. Include the app_id/app_key pair created before and send the request to Kairos web service.

3. Read the result from the web service and put it to a txt file for later reference.

import base64
import sys
from urllib2 import Request, urlopen

import json
from pprint import pprint

with open('./original.jpg', "rb") as image_file:
    encoded_string = base64.b64encode(image_file.read())

    values = """ {
      "image":" """+encoded_string+""" " ,
      "subject_id": "abc123",
      "gallery_name": "KP"
    }
    """
    headers = {
      'Content-Type': 'application/json',
      'app_id': 'c7cxxxx0a',
      'app_key': 'ac4705bdacxxff406xxxx32b9453'
    }
    request = Request('https://api.kairos.com/enroll', data=values, headers=headers)

    response_body = urlopen(request).read()

    target = open('./Enroll.txt', 'w')
    target.truncate()
    target.write(response_body)
    target.write("\n")
    target.close()

    print(str(response_body))

import base64
import sys
from urllib2 import Request, urlopen
import json
from pprint import pprint

detected = "..."

with open('./original.jpg', "rb") as image_file:
    encoded_string = base64.b64encode(image_file.read())
  #  print(encoded_string)
    values = """ {
      "image":" """+encoded_string+""" " ,
      "gallery_name": "KP"
    }
    """
    headers = {
      'Content-Type': 'application/json',
      'app_id': 'c7cxxxx0a',
      'app_key': 'ac4705bdacxxff406xxxx32b9453'
    }
    request = Request('https://api.kairos.com/recognize', data=values, headers=headers)
    response_body = urlopen(request).read()

    target = open('/root/Result.txt', 'w')
    target.truncate()
    target.write(response_body)
    target.write("\n")
    target.close()

with open('/root/Result.txt') as data_file:
    data = json.load(data_file)

    target = open('/root/Detection.txt', 'w')
    target.truncate()
    try:
        if "images" in data:
           target.write(data["images"][0]["transaction"]["status"])
           detected = data["images"][0]["transaction"]["status"]
        else:
           target.write("failure")
           detected = "failure"
    except:
        detected = "failure"
        target.write("failure")
    target.write("\n")
    target.close()
print(detected)

from PyQt4 import QtGui, QtCore
import cv2
import numpy as np
import os
import sys
import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler

class MyHandler(PatternMatchingEventHandler):
    patterns = ["*.txt"]

    def process(self, event):
        """
        event.event_type
            'modified' | 'created' | 'moved' | 'deleted'
        event.is_directory
            True | False
        event.src_path
            path/to/observed/file
        """
        # the file will be processed there
        print event.src_path, event.event_type  # print now only for degug

    def on_modified(self, event):
       with open('/root/Detection.txt') as data_file:
          for line in data_file:
             data = line
       if "success" in data:
             data = "Successful!"
       else:
             data = "Failed!"
       img = cv2.imread('original.jpg')
       font2 = cv2.FONT_HERSHEY_SIMPLEX
       bottomLeftText2 = (10,220)
       fontScale2 = 1.5
       fontColor2 = (255, 166, 47)
       lineType2 = 2
       cv2.putText(img,data,bottomLeftText2,font2,fontScale2,fontColor2,lineType2)
       cv2.imshow('Photo',img)

    def on_created(self, event):
        os.system('ls')

class Capture():
    def __init__(self):
        self.capturing = False

    def startCapture(self):
        observer = Observer()
        observer.schedule(MyHandler(), path='.')
        observer.start()

        self.capturing = True
        while(self.capturing):
            ret, frame = cap.read()
            cv2.imshow("Video", frame)
            font = cv2.FONT_HERSHEY_SIMPLEX
            bottomLeftText = (10,220)
            fontScale = 1.5
            fontColor = (255,0,0)
            lineType = 2
        observer.join()

    def takepicture(self):
            ret, frame = cap.read()
            font = cv2.FONT_HERSHEY_SIMPLEX
            bottomLeftText = (10,220)
            fontScale = 1
            fontColor = (255,0,0)
            lineType = 2
    
            cv2.imwrite('original.jpg', frame)
            cv2.putText(frame,'Validating Face',bottomLeftText,font,fontScale,fontColor,lineType)
            cv2.imshow('Photo', frame)

    def enroll(self):
       # self.capturing = False
       # cap.release()
        os.system('python ./enroll.py')

    def quitCapture(self):
        cv2.destroyAllWindows()
        cap.release()
        QtCore.QCoreApplication.quit()


class Window(QtGui.QWidget):
    def __init__(self):

        QtGui.QWidget.__init__(self)
        self.setWindowTitle('Control Panel')

        self.capture = Capture()

        self.start_button = QtGui.QPushButton('Start',self)
        self.start_button.setGeometry(200,140,80,40)
        self.start_button.clicked.connect(self.capture.startCapture)

        self.takepicture_button = QtGui.QPushButton('Take Picture',self)
        self.takepicture_button.setGeometry(300,140,80,40)
        self.takepicture_button.clicked.connect(self.capture.takepicture)

        self.end_button = QtGui.QPushButton('Enroll',self)
        self.end_button.setGeometry(400,140,80,40)
        self.end_button.clicked.connect(self.capture.enroll)

        self.quit_button = QtGui.QPushButton('Quit',self)
        self.quit_button.setGeometry(500,140,80,40)
        self.quit_button.clicked.connect(self.capture.quitCapture)
      
        w = QtGui.QWidget()
        p = w.palette()
        p.setColor(w.backgroundRole(), QtGui.QColor(255, 204, 204))
        w.setPalette(p)

        pic = QtGui.QLabel(self)
        pic.setGeometry(5, 5, 790, 145)
        pic.setPixmap(QtGui.QPixmap(os.getcwd() + "/background_a.gif"))

        self.setGeometry(0,0,800,190)
        self.show()


if __name__ == '__main__':
    cap = cv2.VideoCapture(0) 
    cap.set(3,640)
    cap.set(4,400)
    app = QtGui.QApplication(sys.argv)
    window = Window()
    window.show()
    sys.exit(app.exec_())

import os
import sys
import time  
from watchdog.observers import Observer  
from watchdog.events import PatternMatchingEventHandler  

class MyHandler(PatternMatchingEventHandler):
    patterns = ["*.jpg"]

    def process(self, event):
        """
        event.event_type 
            'modified' | 'created' | 'moved' | 'deleted'
        event.is_directory
            True | False
        event.src_path
            path/to/observed/file
        """
        # the file will be processed there
        print event.src_path, event.event_type  # print now only for degug

    def on_modified(self, event):
        os.system('python /root/validate.py')
    def on_created(self, event):
        os.system('python /root/validate.py')

if __name__ == '__main__':
    args = sys.argv[1:]
    observer = Observer()
    observer.schedule(MyHandler(), path=args[0] if args else '.')
    observer.start()

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()

    observer.join()