Published September 18, 2021 © MIT

CaneCam - helping the blind cross the street with CV/TinyML

A TinyML-powered CV cane to solve an everyday challenge for the blind - crossing the street.

BeginnerFull instructions provided3 hours694

CaneCam - helping the blind cross the street with CV/TinyML

Things used in this project

Hardware components

Walking Cane (Generic)

Any walking cane will work

Raspberry Pi 4 Model B

USB Webcam (Generic)

Any type of USB Webcam / Pi Camera works

USB Power Bank (Generic)

5V relay (Generic)

DC motor (generic)

9V battery (generic)

Jumper wires (generic)

Mini Breadboard (Generic)

Small Box (Generic)

Tupperware works fantastic

2x Bolt, nut and washer (Generic)

Software apps and online services

Edge Impulse Studio

Raspberry Pi Raspbian

Hand tools and fabrication machines

Drill / Driver, Cordless

Story

Background

Vision impairment is a problem faced by ~250 million people worldwide. To the visually impaired, computer vision offers an elegant solution to everyday problems - by letting computers see for them. Until now, computer vision has been limited to mostly stationary applications, requiring large processors and large power supplies. Embedded CV systems exist, but they boast steep learning curves and subpar power consumption.

But with Edge Impulse, Hackers can now launch low-power ML/CV models on embedded systems with ease, bringing TinyML to the masses.

This project intends to take advantage of Edge Impulse's technology and some simple components to solve a problem encountered daily by the blind - crossing the street. Without the ability to see Pedestrian Crossing Lights (PCLs), the blind face problems with travelling on foot, which is often the only mode of transportation available when traveling alone. Traditionally, the blind make use of sound cues or canine aids to know when they can safely cross. But as cars get quieter (looking at you, Elon) and CV tech improves, there is a need for a more elegant solution.

Enter CaneCam.

Using just a Raspberry Pi 4, a webcam, and some circuitry, CaneCam detects when pedestrian crossing lights turn green, and alerts the user with a vibration on the handle from a small motor. Thanks to its optimised duty cycle, the CaneCam can run over 4 hours on a single charge, plenty for day trips on the town.

And the best part is - you too can make a CaneCam with these simple parts and Edge Impulse! Here's how to do it:

Step 1: Assemble the Cane

The first step in making any smart cane is of course the cane itself. For this project, I knew I wanted to contain the electronics entirely within the cane, because no one likes wearing a backpack, belt, or hat with fancy wires that get in the way. So - I opted to use a small clear box from an Arduino kit to self-contain all the wiring, making the CaneCam something you can just pick up and go. (As it should.)

To start, use a drill to make two holes 3/4 up the cane's shaft, crossing the diameter from front to back. Then match those holes with two identical holes in the plastic box. These holes are used to accommodate two toggle bolts and washers which are fastened to the back of the box.

When done, your box should be secured to the cane with next to zero wobble.

Note: The size of your holes depends only on what bolts and nuts you use.

Step 2: Build the electronics

At the heart of the CaneCam is the ubiquitous Raspberry Pi 4B, which I chose because it has the processing power and RAM needed to run our model. It's connected to a USB Webcam for video input, as well as a 5V relay for controlling the vibrating motor. The relay is used to route power from a separate 9V battery to the motor, because powering the motor off the Pi's power bank results in issues with the Pi's operation (low voltage). The wiring is pretty simple, only using 2 GPIO pins, some jumper wires, the relay, a mini breadboard, and the motor. If you choose a large enough box, you should be able to fit everything, including the USB cable of the Webcam, inside the box.

Arduino is replaced with RPi in practice, and the orange wire goes to pin 18.

To wire the Pi, relay, battery, and motor, follow the schematic shown. You can choose small jumper cables for the interior electronics because they are so tightly packed.

Note: The jumper cables that go to the motor will be much longer than the others, see next section for the reason. You may need to solder two wires together to get the desired length.

Once the electronics are connected, simply plug in your webcam to any of the USB ports on the pi, and connect your power bank to power the Pi. Your wiring is done!

Step 3: Install Electronics

Now that we have our guts and our shell, it's time to combine the two. Begin by placing the power bank first, followed by the Pi, battery, relay, and breadboard on top - making sure to keep all pins connected and secure.

You can distribute the components in the box in any way you choose, just make sure to leave enough room for the webcam cable.

Then, take your webcam and attach it to the front/top of the box. I used some duct tape because the shape of my webcam worked well with it, but depending on your box and webcam, your needs may be different. The important part is that the webcam has minimal movement once installed.

Remove the motor and long cables from the box (don't tear its connections though), then connect your webcam to the Pi before laying (stuffing) the rest of the webcam cable inside your box. You may find that you need to drill a hole in your box lid in order to feed the USB cable through.

Finally, route your long jumper cables to your motor up the shaft of the cane to the top, where the motor is attached with some duct tape. Make sure to place the motor flush so it's vibration is transferred to the cane - the closer to the handle the better. Apply tape liberally for cable management, and you're done building!

Note: I suggest future designs to take the minimalism further and conceal the motor and cables inside the cane body, which is often hollow. For this proof of concept however, I opted to leave the motor and cables on the exterior of the cane.

Step 4: Building the model

Gathering Data

Here's the part where Edge Impulse comes in. But before we open the studio, let's talk data. Specifically, what pictures will you use to build your model? Past research has successfully made models that can detect PCLs, but, as you may recognize, notalllightsarethesame. The world is filled with PCLs, but you will scarcely find a city with the same lights as one 30 miles over - not to mention different countries.

As a result, it's difficult to make a generalized model of these lights that look so different. So, I opted for a more specific approach for my proof of concept: I skateboarded to a 4-way intersection near my house, and took 100 pictures of the 8 PCLs there, with different orientations, modes, and magnifications. (While avoiding weird looks of people wondering why I was so interested in Pedestrian Crossing Lights)

An example picture from the dataset, and a rendering of the intersection

Of course, this means that my model will only feasibly work at crosswalks with this specific type of PCL, and it works best at the exact intersection where I gathered my data. But for a proof of concept, that's all I need.

Edge Impulse

Edge Impulse is a relatively new platform that allows hackers to easily deploy ML models on embedded devices, regardless of skill level. That's why it's perfect for this project, which was designed for novices.

We will be using Edge Impulse's studio to create an Image Classification model that can detect PCLs. Normally, a dataset of only 100 pictures would not be nearly enough to train an entire model for classification. But thanks to Edge Impulse, we can leverage transfer learning to get a fairly good model with a small amount of data.

First, create your project

Once prompted, select Images->Classify a single object, since we are only working with one object per picture (the Pedestrian Crossing Light).

Once created, your project dashboard will have a checklist for you at the top, which is what we'll be following. First we'll collect our data - the two main ways to do this are connecting a phone, and uploading pictures from your computer - you can choose either.

Note: The 3 classes I'm using are "Go", "Stop", and "Unknown". Future projects can expand upon these and include classes like "Counting Down" for more functionality

For this example I'll be uploading pictures I've taken previously that are saved on my computer. Simply select all the pictures of lights showing "Go", and assign them the correct label, as shown.

Do this again for the "Stop" and "Unknown" classes. Once you've uploaded all your pictures (try to get at least 50), the rest of our work is contained entirely in the Edge Impulse Studio!

Creating the Impulse

Our next task is to make the actual neural network. In Edge Impulse, models are called impulses, and making them is a breeze. Click on the ImpulseDesign menu. This is where we tell Edge Impulse what kind of model we are trying to make, and with what settings.

We're going to use the recommended settings for image classification in the impulse design - meaning 96x96 Images and Fit shortest axis. We also want to click Add a processing block and Add a learning block to add the recommended Image block and Transfer Learning block.

The impulse after adding the blocks

Finally, click Save Impulse. Navigate the new Images menu item, and select the RGB Color depth and save it. This tells our model that we want to use color images, which of course we need when a big part of the difference between "Go" and "Stop" is the color of the light. (White v Red)

Saving will take you to a "Generate Features" page where you should click, you guessed it, GenerateFeatures.

This gives us a feature graph of our images, which we can use to observe patterns in our data.

We also get a performance estimate for speed and ram usage on our device.

Next, click TransferLearning. Here's where we are actually creating/training the network to respond to our data (in this case, the Pedestrian Crossing Light pictures). Change the Number of training cycles to 100, and check the DataAugmentation box.

Next, click Choosea different model and select the MobileNetV2 96x96 0.35 model.

Almost done (with the model)! Click the Edit button on the model box and change the numberof neurons to 20. All these changes will make the model fit better with your data and enable CaneCam to detect your crossing lights more effectively.

Here's how your settings should look before beginning training.

Finally, click the StartTraining button. Training may take a minute because we've increased the number of epochs, so take this time to grab some water.

Evaluating the Impulse

After training, my model returned a so-so accuracy of 63.6%. Clicking on the ModelTesting menu item shows the test set of data, where you can run your model on your separated data. Click ClassifyAll to test the data.

This evaluation shows an (albeit altruistic) accuracy of 100% on 10 samples of Stop and Go. Not too shabby for <100 images.

Step 4: The Codening

Connect to Edge Impulse

The engineers among us have had their fun, now it's time for the coders to have their field day. After all, what's electronics without the code behind them?

While you can SSH into your Pi for many projects, in this one we want to see the video feedback - so we're going to hook up a USB Keyboard and Mouse for control, as well as a monitor VIA the Pi's Mini-HDMI ports. This setup will be the basis of our coding portion, but don't worry, it's not final.

Note: The following parts follow the Raspberry Pi 4 Guide from Edge Impulse, if you'd like to setup without a screen or with a PiCam, you can find instructions there.

Boot up your Pi (mine uses Raspbian Buster for CaneCam) and open up the terminal. To set up the device with Edge Impulse, run the following commands:

curl -sL https://deb.nodesource.com/setup_12.x | sudo bash -

sudo apt install -y gcc g++ make build-essential nodejs sox gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps

npm config set user root && sudo npm install edge-impulse-linux -g --unsafe-perm

With the software taken care of, connect your USB Webcam to the Pi and run

edge-impulse-linux

This will start a wizard which will ask you to log in, and choose an Edge Impulse project. If you want to switch projects run the command with --clean.

That's it! Navigate back to Edge Impulse to verify that your Pi has been connected. Your screen should show like this:

To run your impulse without a script, simply call

edge-impulse-linux-runner

which will automatically download and run your model, opening the terminal as a text output of the classification. Pretty cool!

Downloading the Python SDK

But we want some power behind our model, and to have more than just a window and terminal as our output. To do this, we need to install Edge Impulse's Linux Python SDK, which we will use to construct a Python script that controls the CaneCam using our impulse/model we've built.

To do this, install a recent version of Python 3, or check if you already have a Python3 version >=3.7 by issuing

python3 --version

I have Python 3.7.3 installed already, so I can move onto the next step. Issue:

sudo apt-get install libatlas-base-dev libportaudio0 libportaudio2 libportaudiocpp0 portaudio19-dev

pip3 install edge_impulse_linux -i https://pypi.python.org/simple

Then, download the model file via:

edge-impulse-linux-runner --download modelfile.eim

With the model finally on your Pi, create a Python file in the same directory as the modelfile.eim you just downloaded. Do this by navigating to Thonny, the Pi's easy Python script editor. Create a new file and copy the code from CaneCam.py (in this guide's attachment section), or find this guide on your Pi and download it there. Name the file CaneCam.py The file is a modified version of Edge Impulse's webcam detection script, with the addition of label detection, duty cycle, and motor control.

In a nutshell, the script takes a picture every 5 seconds and waits until it detects a pedestrian light (either on or off), then it wakes its video feed and runs at ~5 frames PER second (pic every.2s). Once in the video mode, the script continuously looks for a "Go" pedestrian light. One it finds one, the script turns on the motor until the next frame where the script decides what to do yet again. This process repeats until there is no PCL in sight, after which the system goes back to "sleep", aka. taking one picture every 5 seconds. This system allows the Pi to conserve power when it's not actively processing a video feed, and extends the uptime on the battery bank from 2 to 4+ hours!

Note: See the comments in the code for a more in-depth look at the function of each line.

After saving the two files (CaneCam.py, modelfile.eim) in the same directory, move into that directory using the cd command from your terminal. Finally, issue:

python3 CaneCam.py modelfile.eim

Your terminal will start outputting confidence values and a suggested action. You can adjust the threshold for the suggestion by changing min_conf_threshold in CaneCam.py. A 96x96 video feed will pop up to give you a view of what the model sees, and you can have fun pointing it at yourself and making your best Pedestrian Crossing Light face.

The finished product

With your code uploaded, switch your power supply to the battery pack, start the code, and place the Pi inside the cane box with all the other electronics. Close the lid and - voila! Your very own CaneCam. Take it out into the world and enjoy the peace of mind that you can cross the street without needing to see the crossing light.

Video

Areas for Improvement

As shown in the video, CaneCam works quite well at the intersection at which I gathered my data, only taking a few seconds to recognize a "Go" light. However, this is mostly due to overfitting my model to these examples of PCLs. In essence, it is the opposite of generalized. If others are going to replicate this project, I suggest gathering a larger dataset of PCLs in the area to see if the model can generalize to different surroundings.

Future Applications

The CV and TinyML industries are booming, and companies like Edge Impulse are doing good by being right in the middle of them. The great thing about CaneCam is that it's not very specialized to one task - a camera can be used to detect just about anything with CV. This means that with some more work, curious Hackers could add functionality like text reading, location tracking, and obstacle avoidance - all of which are problems that could drastically improve the life of the blind. If a business were to adopt the CaneCam, the implementation is clear - make an attachment for existing canes or develop a whole smart cane system - with functions starting with crossing the street, and dropping software updates that add functionality as time goes on.

Conclusions

I created this project to prove a concept I believe is tech of the future - assistive machine learning. Specifically, Computer Vision brings so many possibilities to the table for those who are vision impaired, immobile, or otherwise disabled. By finding an everyday problem encountered by the blind, I was able to use some simple tools to prove a small idea with huge implications.

In the coming decades, I imagine a world where disabilities like blindness are no longer ailments that prevent people from living their lives. Assistive tech like CV glasses for reading, transportation tech like CaneCam, and more will be able to change the way 250 million vision impaired people around the world live their lives and get around. People are the industry of the world, and if we don't invest in them now, who knows what we're leaving behind.

THANKS FOR READING AND HAPPY HACKING!

Code

CaneCam.py

#Import all needed libraries
import cv2
import os
import sys, getopt
import signal
import time
from edge_impulse_linux.image import ImageImpulseRunner

#Servo imports to use GPI/O pins
import RPi.GPIO as GPIO

motorpin = 18 #Sets the pin to control the relay
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(motorpin, GPIO.OUT) #Makes the motorpin an output

runner = None
show_camera = True #whether you want to see the vid box or not, set T/F
min_conf_threshold = 0.90 #minimum confidence threshhold for making a recommendation, edit for suggestions

def now(): #returns time
    return round(time.time() * 1000)

def get_webcams(): #gets the port IDs of the connected webcam(s)
    port_ids = []
    for port in range(5):
        print("Looking for a camera in port %s:" %port)
        camera = cv2.VideoCapture(port)
        if camera.isOpened():
            ret = camera.read()[0]
            if ret:
                backendName =camera.getBackendName()
                w = camera.get(3)
                h = camera.get(4)
                print("Camera %s (%s x %s) found in port %s " %(backendName,h,w, port))
                port_ids.append(port)
            camera.release()
    return port_ids

def sigint_handler(sig, frame): #interrupt handler
    print('Interrupted')
    if (runner):
        runner.stop()
    sys.exit(0)

signal.signal(signal.SIGINT, sigint_handler)

def help(): #help
    print('python classify.py <path_to_model.eim> <Camera port ID, only required when more than 1 camera is present>')

def main(argv):
    try:
        opts, args = getopt.getopt(argv, "h", ["--help"])
    except getopt.GetoptError:
        help()
        sys.exit(2)

    for opt, arg in opts:
        if opt in ('-h', '--help'):
            help()
            sys.exit()

    if len(args) == 0:
        help()
        sys.exit(2)

    model = args[0]
    
    #Creates paths to the directory and modelfile.eim that we downloaded
    dir_path = os.path.dirname(os.path.realpath(__file__))
    modelfile = os.path.join(dir_path, model)

    print('MODEL: ' + modelfile)

    with ImageImpulseRunner(modelfile) as runner:
        try:
            model_info = runner.init()
            print('Loaded runner for "' + model_info['project']['owner'] + ' / ' + model_info['project']['name'] + '"')
            labels = model_info['model_parameters']['labels']
            if len(args)>= 2:
                videoCaptureDeviceId = int(args[1]) #Sets the webcam to capture
            else:
                port_ids = get_webcams()
                if len(port_ids) == 0:
                    raise Exception('Cannot find any webcams')
                if len(args)<= 1 and len(port_ids)> 1:
                    raise Exception("Multiple cameras found. Add the camera port ID as a second argument to use to this script")
                videoCaptureDeviceId = int(port_ids[0])
            
            #Sets up the camera
            camera = cv2.VideoCapture(videoCaptureDeviceId)
            ret = camera.read()[0]
            if ret:
                backendName = camera.getBackendName()
                w = camera.get(3)
                h = camera.get(4)
                print("Camera %s (%s x %s) in port %s selected." %(backendName,h,w, videoCaptureDeviceId))
                camera.release()
            else:
                raise Exception("Couldn't initialize selected camera.")

            next_frame = 0 # limit to ~10 fps here
            
            rn = now()

            #Workhorse algorithm, this is where the actual classification happens
            for res, img in runner.classifier(videoCaptureDeviceId): #for each frame

                #If there's a valid class
                if "classification" in res["result"].keys():
                    decision = ''
                    for label in labels: #Get the score of each label
                        score = res['result']['classification'][label]
                        if score > min_conf_threshold: #and check if it's > min_conf_threshold
                            decision = label #if it is, set it as a decision
                        print('%s: %.2f\t' % (label, score), end='')
                    print('You should: '+decision) #make the suggestion. "You should: " + "Go/Stop" etc.
                    if decision == 'Go': #If there's a go light
                        GPIO.output(motorpin, 1) #Turn on the motor to vibrate
                    elif decision == 'Stop' or decision == '': #If not
                        GPIO.output(motorpin, 0) #Turn off the motor
                
                #Print the bounding boxes
                elif "bounding_boxes" in res["result"].keys():
                    print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))
                    for bb in res["result"]["bounding_boxes"]:
                        print('\t%s (%.2f): x=%d y=%d w=%d h=%d' % (bb['label'], bb['value'], bb['x'], bb['y'], bb['width'], bb['height']))
                        img = cv2.rectangle(img, (bb['x'], bb['y']), (bb['x'] + bb['width'], bb['y'] + bb['height']), (255, 0, 0), 1)
                
                #Show camera, also listen for q to exit
                if (show_camera):
                    cv2.imshow('edgeimpulse', cv2.cvtColor(img, cv2.COLOR_RGB2BGR))
                    if cv2.waitKey(1) == ord('q'):
                        break
                
                #This is important, editing the 100 here changes your framerate. The max is about 10, but
                #we don't need more than 5 for this project
                next_frame = now() + 100
                print(now()-rn)
                if (next_frame > now()):
                    time.sleep((next_frame - now()) / 1000)
        finally:
            if (runner):
                runner.stop()

if __name__ == "__main__":
   main(sys.argv[1:])

Credits

Hudson Gould

1 project • 0 followers

CaneCam - helping the blind cross the street with CV/TinyML

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

Background

Step 1: Assemble the Cane

Step 2: Build the electronics

Step 3: Install Electronics

Step 4: Building the model

Gathering Data

Edge Impulse

Creating the Impulse

Evaluating the Impulse

Step 4: The Codening

Connect to Edge Impulse

Downloading the Python SDK

Video

Areas for Improvement

Future Applications

Conclusions

Schematics

Schematic (Ardu for RPi)

Code

CaneCam.py

Credits

Hudson Gould

Comments

Embed the widget on your own site

CaneCam - helping the blind cross the street with CV/TinyML

CaneCam - helping the blind cross the street with CV/TinyML

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

Background

Step 1: Assemble the Cane

Step 2: Build the electronics

Step 3: Install Electronics

Step 4: Building the model

Gathering Data

Edge Impulse

Creating the Impulse

Evaluating the Impulse

Step 4: The Codening

Connect to Edge Impulse

Downloading the Python SDK

Video

Areas for Improvement

Future Applications

Conclusions

Schematics

Schematic (Ardu for RPi)

Code

CaneCam.py

Credits

Hudson Gould

Comments

Related channels and tags