Scanning Vaccination Cards to Ensure Safety

I recently saw the Core2 and was very interesting in it. I wanted to make something (as to the contest's overview) in safety regarding Covid. Unfortunately, no matter how hard I tried, a lot of things failed and I was only able to create an app to upload images to s3.

To start off, lets look at the project architecture:

Step 1 - Create the expo app

The first step is to create an app to take a picture and store in the s3 bucket.

For this, I used react-native expo, as I do not own MacOS to program in XCode (and I have an iPhone).

• For the actual sending photos to the s3 bucket I followed this amazon doc
• For the style of the app, I used react-navigation-screens and followed the docs there
• For image capturing, I used expo-camera and followed the docs there.

All together, the app was working and the video of it in use is here:

Step 2 - The upload triggers Step 3 (Lambda Function)

Step 3 + 4 - The hearts of it (Lambda function) + sending to Rekognition

Once the image is uploaded, it triggers a Lambda function. This function is the hearts of this whole project (written in Python)

Now that we have the photo, we need to verify that the image is an actual vaccination card. To do this we use Amazon's Rekognition.

• Imports:

import boto3 #for connecting with the s3 bucket
from botocore.config import Config #for configuring the service

The imports are explained by the comments.

• Configuring boto3

my_config = Config(
    region_name = 'us-east-2',
    signature_version = 'v4',
    retries = {
        'max_attempts': 10,
        'mode': 'standard'
    }
)

Standard configuring, it is important to have this in order to configure the same region as your s3 bucket

• Gaining data from Rekognition and storing it in a list

def main():

    bucket='vax-scan'
    photo='image1827600.png'
    text_count=detect_text(photo,bucket)
    print("Text detected: " + str(text_count))
    print(text1)

    check_card()


if __name__ == "__main__":
    main()

in the main function, we have to define the function's parameters for the detect_text function. In this case, we use our bucket and image

def detect_text(photo, bucket):

    client=boto3.client('rekognition', config=my_config)

    response=client.detect_text(Image={'S3Object':{'Bucket':'vax-scan','Name':'image1827600.png'}})
                        
    textDetections=response['TextDetections']
    print ('Detected text\n----------')
    for text in textDetections:
            print ('Detected text:' + text['DetectedText'])
            text1.append(text['DetectedText'])
            print ('Confidence: ' + "{:.2f}".format(text['Confidence']) + "%")
            print ('Id: {}'.format(text['Id']))
            if 'ParentId' in text:
                print ('Parent Id: {}'.format(text['ParentId']))
            print ('Type:' + text['Type'])
            print()
    return len(textDetections)

This is the function taken from the Amazon docs. We assign the discovered text to the list text1 hence the line text1.append(text['DetectedText'])

Step 5 - Receive Rekognition's outputs and scan the card using test words

Now that we the text recognized in a single list, let's look at the function check_card() called in main()

def check_card():
#    text1 = ['information', 'Covid', 'hello', 'test']
    test_words = ['information', 'mm', 'incluye', 'de', 'vaccines', 'Date']
    if(set(test_words).issubset(text1)):
        print("yes!")
        nameIndex = text1.index('Last')
        nameIndex = nameIndex - 1
        print("Name = " + text1[nameIndex] + " passed!")
        blink()

    else:
        print("NOT ALLOWED!")

Here, we use test_words to test if the card is a vaccination card. This is not a very secure method, but can be used for now. If the words are contained in the scanned words, we print the name of the card owner and call the function blink()

Lets look at blink():

def blink():
    # Spin up resources
    event_loop_group = io.EventLoopGroup(1)
    host_resolver = io.DefaultHostResolver(event_loop_group)
    client_bootstrap = io.ClientBootstrap(event_loop_group, host_resolver)
    mqtt_connection = mqtt_connection_builder.mtls_from_path(
                endpoint=ENDPOINT,
                cert_filepath=PATH_TO_CERT,
                pri_key_filepath=PATH_TO_KEY,
                client_bootstrap=client_bootstrap,
                ca_filepath=PATH_TO_ROOT,
                client_id=CLIENT_ID,
                clean_session=False,
                keep_alive_secs=6
                )
    print("Connecting to {} with client ID '{}'...".format(
            ENDPOINT, CLIENT_ID))
    # Make the connect() call
    connect_future = mqtt_connection.connect()
    # Future.result() waits until a result is available
    connect_future.result()
    print("Connected!")
    # Publish message to server desired number of times.
    print('Begin Publish')
    for i in range (RANGE):
        data = "{} [{}]".format(MESSAGE, i+1)
        message = {"message" : data}
        mqtt_connection.publish(topic=TOPIC, payload=json.dumps(message), qos=mqtt.QoS.AT_LEAST_ONCE)
        print("Blinks on!")
        t.sleep(3)
        mqtt_connection.publish(topic=TOPIC, payload=json.dumps(message), qos=mqtt.QoS.AT_LEAST_ONCE)
        print("Blinks off!")
        t.sleep(0.1)
        
    print('Publish End')
    disconnect_future = mqtt_connection.disconnect()
    disconnect_future.result()

Here, I followed the docs for sending MQTT messages by Python from Amazon's docs. Essentially, following the correct certificates, the message myIoT/blink is sent once, then after 3 seconds, sent again. This turns on the blink function in the EduKit's code and after 3 seconds turns it off. This signals that the user's card was accepted and may continue.

That's pretty much it for the lambda function.

Step 6 - EduKit

For this project I used Cloud Connected Blinky Example, since it already has everything needed to take MQTT messages.

There is one line I changed, all that was the line that displays "Subscribing to" to the UI to

ui_textarea_add("Waiting for vaccination card. If the side LEDS blink for 3 seconds, and then stop you are good to go. If not, please contact the owners of this establishment", subscribe_topic, SUBSCRIBE_TOPIC_LEN) ;

That was pretty much it for the IoT.

Further Discussions + Problems ran into

I ran into a lot of problems, but I'll list the main ones

• Creating the app. I ran into problems simply creating the app, since I had to learn react-native from scratch.
• When writing the lambda function, I originally used Java. There were just too many errors, so I moved to python. There, I had many issues regarding the regions of my buckets (even made a discussion post). I fixed it thankfully
• I could not get the .zip to work for my lambda function. After much much deep searching I eventually discovered that I needed to use Ubuntu to zip, asthe python runtime is run on Amazon's LINUX not WINDOWS.
• I would have liked to show the vaccination card on the EduKit core. I followed this Amazon tutorial, however, not hard I tried it would not work. I would run into the same error. I even sent an issue on Github and nothing would work. I had to move past this.

Improving my project

This was a base project since I did not have much time to finish. There are several ways to improve it to make it into a successful one.

• In the lambda function, figure out a way to input the latest image taken as the key for rekogntion. Right now, I use a default image to scan.
• When searching the photo, I would like to make a more secure way to verify the card. This would entail looking for things such as "First Dose" and "Second Dose" as well as a date and place attached with those two data sets.
• Cleaning up the EduKit core. I would have liked to have made the UI look a lot cleaner from the standard "Cloud Connected Blinky" I would have liked a better, more user friendly interface to display the status of the card verification.