RTSP Feed Camera Source for Edge Impulse on Kria KR260

For the past few years, I've been using Edge Impulse to create and train ML models for various projects of mine, the most popular of which was my Lego Land Rover Defender robot. For that project, I installed Edge Impulse on my KR260 and ran a custom ML model I created for detecting and avoiding a target obstacle. After getting the Lego Defender robot up and running, I realized I needed a wireless solution for a couple different aspects of the system, including the video source for the KR260.

To migrate from the USB webcam as the video source for Edge Impulse on the KR260, I chose to utilize a camera capable of streaming its video feed via an RTSP stream. RTSP or real-time streaming protocol, is a network protocol designed to control streaming media in entertainment and communications systems with low latency and no external internet connection required. It functions very similarly to HTTP and is commonly used for security cameras, smart home integration applications, and live video streaming services.

I'm using the official Ubuntu 22.04 desktop image for AMD devices on my KR260 that I installed Edge Impulse on as outlined in my past project post here. And I'm using the Edge Impulse model I created for my Lego Land Rover Defender robot.

Picking an RTSP Capable Camera

Picking the specific camera capable of using an RTSP connection turned out to be the most difficult part of this project. All are technically security cameras that I found, but the difficult part was that a dwindling number of them will allow you to connect with a generalized RTSP link vs forcing you to use the manufacturer's proprietary app with them.

I ended up spending quite a bit of time searching "RTSP security cameras" on Amazon.

Amazon Q&A and reviews ended up being immensely helpful in this venture as I found RTSP-related questions were one of the most common questions asked. I was able to verify the TP-Link Tapo camera I chose could definitely be accessed via a general RTSP link outside of the TP-Link app when I saw the answer to an RTSP question that someone was able to access its steam with VLC media player.

I'm pretty sure my Amazon account will forever suggest security cameras on my homepage from now on though...

Setup the Camera on Target Network

After patiently waiting for my next day shipping for my Amazon Prime order of the Tapo security camera, the first step was to get it connected to a wireless network. I setup the camera on the same LAN/network that the KR260 with Edge Impulse will be operating on using the TP-Link app (the same network as the KR260 is a requirement, but this network doesn't have to be connected to the internet).

Download the TP-Link Tapo application:

Create an account and log in:

Add the security camera from the home page, following the directions to connect to the camera's access point (AP) network first:

Then connect the camera to the same network as the Kria KR260 and name the camera:

Skip cloud storage and local storage since it's not needed in this case:

Finally, skip through setup services until you get to the home page in the app.

Next, we need to figure out what IP address was assigned to the camera for the RTSP link. Open the camera in the app and go into its settings by tapping the gear icon.

Select Advanced Settings > Device Account:

Create an account for camera login via 3rd party apps, this will be the username and password used in the final RTSP link:

Then you'll be able to see the IP address assigned to the camera:

I used VLC player to figure out the specific format of the RTSP feed for the Tapo camera. After from trial and error using the instructions from TP-Link's website here, I found that the link is a combination of the account name and password created for 3rd party apps an the assigned camera IP address:

rtsp://<account name>:<account password><camera IP address>/stream1

So for me, I used the login I created in the TP-Link Security Camera app with the username "legodefenderv2":

rtsp://legodefenderv2:****@192.168.1.106/stream1

Once I was able to open the RTSP feed in VLC player, that was the confirmation to be that the link could be opened with the OpenCV driver in Python.

RTSP Camera Test with OpenCV in Python

Since the main function in Python for running the Edge Impulse ML model already has quite a bit going on, I decided to test the process of capturing an image from the security camera via the RTSP link in a separate simple Python script first.

On the KR260, I created a directory in the home folder to store the test script and test images in:

ubuntu@kria:~$ mkdir -p rtsp_cam
ubuntu@kria:~$ mkdir -p ./rtsp_cam/captured_images
ubuntu@kria:~$ cd ./rtsp_cam

Then I created the test script:

ubuntu@kria:~/rtsp_cam$ nano rtsp_cv2_test.py

And added the following code to it:

import os
import sys
import cv2
import time

capture = cv2.VideoCapture("rtsp://legodefenderv2:****@192.168.1.106/stream1")
time.sleep(0.1)
(success, reference) = capture.read()
cv2.imwrite('/home/ubuntu/rtsp_cam/captured_images/live_image.jpg',reference)
capture.release()

I then pulled the image back onto my host PC to validate it was successfully captured/saved:

whitney@MacBook-Pro % scp ubuntu@192.168.1.145:/home/ubuntu/rtsp_cam/captured_images/live_image.jpg ./

RTSP Camera Integration in Edge Impulse

Now that I know the Tapo security camera can be accessed via the RTSP link, and it works with OpenCV in Python, the final step is to point the Edge Impulse image classification code to use it as the image source.

I simply replaced the default iteration through available video source device numbers with the hardcoded RTSP link for the time being. However, I do plan to come back and better integrate it such that the code will look for USB webcams first then check for the RTSP link.

So I changed the following in my image-classify.py where the Edge Impulse Runner service looks for webcams:

model_info = runner.init()
print('Loaded runner for "' + model_info['project']['owner'] + ' / ' + model_info['project']['name'] + '"')
labels = model_info['model_parameters']['labels']
if len(args)>= 2:
    videoCaptureDeviceId = int(args[1])
else:
    port_ids = get_webcams()
    if len(port_ids) == 0:
        raise Exception('Cannot find any webcams')
    if len(args)<= 1 and len(port_ids)> 1:
        raise Exception("Multiple cameras found. Add the camera port ID as a second argument to use to this script")

    videoCaptureDeviceId = int(port_ids[0])

camera = cv2.VideoCapture(videoCaptureDeviceId)
ret = camera.read()[0]

To just use the hardcoded RTSP link:

model_info = runner.init()
print('Loaded runner for "' + model_info['project']['owner'] + ' / ' + model_info['project']['name'] + '"')
labels = model_info['model_parameters']['labels']

videoCaptureDeviceId = "rtsp://legodefenderv2:*******@192.168.1.106/stream1"
camera = cv2.VideoCapture(videoCaptureDeviceId)
ret = camera.read()[0]

Then I ran my updated version of my classify-image Runner script for my jack-o-lantern obstacle avoidance ML model from my first gen Lego Land Rover Defender robot:

ubuntu@kria:~/rtsp_cam$ python3 pumpkin_stopv2_rtsp_cam.py /home/ubuntu/kr260-pumpkin-obstacle/modelfile.eim

And I was successfully able to get the my jack-o-lantern obstacle avoidance ML model to work with the RTSP feed from the TP-Link Tapo security camera!

The use of a wireless video source was an import upgrade to my first gen Lego Land Rover Defender robot as I'll explain in a future post, but since the use of ML models for security cameras is a pretty popular use case I thought this warranted it's own project write-up for Edge Impulse users at large.

Stay tuned for Lego Land Rover Defender v2 though!