KV260 line follower car

Introduction

When the idea came up to me to make a line follower car, I didn't have a sensor like photoresistor or Infrared receiver but I did have KV260. This project try to make line follower car using KV260.

Hardware

• KV260
• Display which has DisplayPort
• USB camera
• Geared motor x 2
• Motor controller (General Purpose Transistor PN2222, Resistor 1kΩ, Resistor 10kΩ) x 2
• LED (Red, Green), Resistor 330Ω x 2
• Power supply

Software

• Ubuntu Desktop 20.04.3 LTS
• PYNQ
• OpenCV

Hardware Setup

• Motor controller

I used 2 GPIO ports as output. Each port connect to each transistor which on/off the motor to turn the wheel.

To go straight, both motor run.

To go left, only right side motor run.

To go right, only left side motor run.

• Line sensor

I used USB camera to see the line on the floor.

Software Setup

1. Install Ubuntu on SD Card then boot.

2. Install SSH on ubuntu. In this project, I used KV260 through ssh from other computer.

sudo apt install ssh

3. Install PYNQ on Ubuntu.

git clone https://github.com/Xilinx/Kria-PYNQ.git
cd Kria-PYNQ/
sudo bash install.sh

4. Access to KV260 by Jupyter from other computer.

http://<ip address>:9090/lab/

Since PYNQ's DisplayPort library will draw anything fullscreen, I use Jupyter from remote computer instead of Ubuntu desktop on KV260.

Software Development

• DisplayPort

As a car, no need display but used for development.

This is how I used the DisplayPort. Create DisplayPort object, configure the dimension, get frame, whatever put data to frame then write it back.

from pynq.lib.video import *
import numpy as np

frame_out_w = 640
frame_out_h = 480

mode = VideoMode(frame_out_w, frame_out_h, 24)
displayport = DisplayPort()
displayport.configure(mode, PIXEL_RGB)

outframe = displayport.newframe()
# fill the green plane with incremental values
outframe[:, :, 1] = np.arange(frame_out_w, dtype=np.uint8)
displayport.writeframe(outframe)

• GPIO

Pmod connector can be used as GPIO which has 8 I/Os, 2 Vcc, and 2 GND. To control 2 motors, I used 2 ports as output.

This is the how I used the GPIO.

1. Load the overlay. I used the overlay come with PYNQ.

from kv260 import BaseOverlay
base = BaseOverlay('base.bit')

2. Define the functions to access GPIOs.

%%microblaze base.PMODA
#include <gpio.h>
#include <pmod_grove.h>

gpio l;
gpio r;

void init() {
    l = gpio_open(PMOD_G1_A);
    r = gpio_open(PMOD_G2_A);
    gpio_set_direction(l, GPIO_OUT);
    gpio_set_direction(r, GPIO_OUT);    
}

void run(int rstate, int lstate) {
    gpio_write(l, lstate);
    gpio_write(r, rstate);
}

void close() {
    gpio_write(l, 0);
    gpio_write(r, 0);
    gpio_close(l);
    gpio_close(r);
}

3. Call the function to make it run the motors.

init() # initialize the GPIO as output port
run(0, 1) # left: ON right: OFF
run(1, 0) # left: OFF right: ON
run(1, 1) # left: ON right: ON
run(0, 0) # left: OFF right: OFF
close() # close the GPIO

• USB camera (Logitech HD Pro Webcam C910)

USB camera connect to USB port. I used OpenCV for video capture and some video processing.

1. Initialize the VideoCapture device using OpenCV.

import cv2

frame_in_w = 640
frame_in_h = 480

videoIn = cv2.VideoCapture(0)
videoIn.set(cv2.CAP_PROP_FRAME_WIDTH, frame_in_w);
videoIn.set(cv2.CAP_PROP_FRAME_HEIGHT, frame_in_h);

2. Capture the image and process it then show on Display. Depend on the processing result, on/off each motors.

init() # GPIO
number_frames = 30 * 120 # run 120sec
for _ in range(number_frames):
    ret, frame_vga = videoIn.read()
    if (ret):      
        frame, direction = proc(frame_vga)
        if direction == 0:
            run(1, 1) # GPIO
        elif direction == -1:
            run(0, 1) # GPIO
        elif direction == 1:
            run(1, 0) # GPIO
        outframe = displayport.newframe()
        outframe[:] = frame
        displayport.writeframe(outframe)
    else:
        raise RuntimeError("Failed to read from camera.")
close() # GPIO

3. This is the function to find the line and decide the direction to go.

def proc(frame):
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)    
    l = gray[:, 0:gray.shape[1]//2]
    r = gray[:, gray.shape[1]//2:]
    lh, _ = np.histogram(l.ravel(), bins=[0, 128, 256])
    rh, _ = np.histogram(r.ravel(), bins=[0, 128, 256])    
    if abs(lh[0] - rh[0]) < (lh[0] + rh[0]) * 0.1:
        direction = 0
    elif lh[0] > rh[0]:
        frame[frame.shape[0]//8*7:, 0:frame.shape[1]//2, :] = [1]
        direction = -1
    else:
        frame[frame.shape[0]//8*7:, frame.shape[1]//2:, :] = [1]
        direction = 1
    return (frame, direction)

Demo

This is testing with LED instead of motor.

In the display, gray vertical bar is the line to follow. Black bar which right bottom of the screen indicate sensor found the line at right side of the car. So left side motor start (Green LED ON) and right side motor stop (Red LED OFF).

Limitations

This project is not completed yet. As a car, I need to make it work with battery but I didn't get it. Also I integrated the sensor and motor control parts, it is not stable yet.

References

• Kria-PYNQ
• Getting Started with Kria KV260 Vision AI Starter Kit
• Ubuntu Desktop 20.04.3 LTS
• OpenCV Histgrams