The following hackster project uses a solely vision-based approach to implement, tune, and test elementary "autonomous" functions on a Radio-Controlled (RC) car platform.
A Linux-based BeagleBone AI 64 accepts a near-real-time video feed from a USB Webcam to actively steer the RC car platform to stay within lane lines with minimal deviation from the center. Furthermore, the RC car platform must come to a complete stop on detecting a red-colored stop box and resume operation after a predetermined duration. Lastly, a machine learning (ML) STOP sign recognition model performs the same stop-and-go behavior on encountering said "STOP" sign.
The COVID Debuff Team executed the project as a requirement for the ELEC 553 – Mobile and Embedded Systems course at Rice University, Houston, TX, during the Fall 2022 semester. Through this project, students demonstrated their knowledge and execution of PID Controller and tuning, implementing ML models and Linux subsystems/modules.
Due to the time constraints of any given semester, the RC Car platform was predesigned and provided to students. Additionally, a detailed starting point for the code design and software control was provided in the form of a popular Instructables.com post (https://www.instructables.com/Autonomous-Lane-Keeping-Car-Using-Raspberry-Pi-and/) by user raja_961. Additionally, students were provided with designs created by students in previous semesters/years (https://www.hackster.io/elec424/projects?sort=published). Please review these materials to ensure a complete understanding of the hardware platform and the software solution used as a jumping-off point for this project.
The BeagleBone AI 64 is relatively power limited (both in TDP and computation power) compared to visual navigation solutions in modern-day “autonomous” vehicles. Additionally, the initial code design “links” the capturing of the input video to the steering of the car, i.e., both are processed in the same conditional branch. If the video is processed too slowly, the steering of the RC Car platform is negatively affected since it must wait for a longer duration to obtain a new steering angle. Due to this inherent design philosophy, the video processing portion of the conditional statement must be reduced to supply more frequent steering updates, even at the expense of steering angle accuracy. The 1080p webcam mounted on the front of the RC car platform looks ahead at the area in front of the car. To match the computational power of the BeagleBone AI 64 and reduce vision computation time, only a 160x120 resolution sample of the 1080p webcam input is taken. This sampling is done “natively, ” as the camera provides the opportunity to obtain a lower-resolution video feed directly from the sensor. An alternative solution would be to accept a 1080p input and resize/downsample using an interpolation technique. The latter technique may improve clarity/sharpness/detail at the expense of BeagleBone AI 64 computation time/power/cycles. The team implemented the first approach improving the refresh rate of steering/heading angle updates.
Solution #1: Native Low-Resolution Visual Input
video = cv2.VideoCapture(2)
video.set(cv2.CAP_PROP_FRAME_WIDTH, 160)
video.set(cv2.CAP_PROP_FRAME_HEIGHT, 120)
Solution #2: Resized/Interpolated Visual Input
video = cv2.VideoCapture(2)
ret,frame = video.read()
frame = cv2.resize(frame,None,fx=0.2,fy=0.2,interpolation=cv2.INTER_LINEAR)
For post-input processing to differentiate blue lane lines and red stop boxes from the surroundings, the openCV stack provided in the reference materials was modified slightly. Each 120p frame is transformed into HSV (Hue – Saturation – Value) color space for easier masking. Blue and red masks are applied to the frame separately for the lane lines and stop box detection, respectively. The red-masked frames are passed through the region of interest (ROI) cutoff to compensate for early stopping due to a deeper camera angle. Edges are detected using the canny edge detector, and lines are drawn using Hough transform. All function parameters were modified and tuned to provide a more stable lane line detection when the RC Car platform is in motion.
Since the RC Car platform is an unstable system, a closed-loop controller must be utilized to consistently follow the lane lines (i.e., stay between the two lanes and close to the center of the “road”). Most reference materials use only a PD controller (Integral set to zero). Still, our team has also implemented an Integral term for more robust control around the setpoint (center of lane). The controller monitors the error (deviation of target value from the current value) and the time difference between discretized steps. The PID Controller applies three controls, namely Proportional, Integral, and Derivative terms, to stabilize the system. Since the system samples inputs-outputs discretely, we can roughly approximate the PID control equations using the following technique,
e(t) = Target Value – Current Value
dT = Current Time – Previous Time
Kp = Proportional Gain
Ki = Integral Gain
Kd = Derivative Gain
The overall system after applying the controller is represented as,
U(t) = Kpe(t) + Kie(t)*dT + Kde(t)/dT
Due to time constraints and the unstable input target value (shaky camera inputs), the PID controller was tuned manually using the Ziegler-Nichols method. The process was initialized by setting the integral (Ki), and derivative (Kd) gains to zero (0). The proportional gain (Kp) for the completed RC car platform was set to zero, and the car was then deployed on a parallel, straight lane. The proportional gain is then incremented till the RC Car platform has a constant oscillation around the required target value (constant error, e(t)). This overall ultimate gain (Ku) and the time period (Tu) of oscillations are recorded. The individual Kp, Ki, and Kd parameters can be calculated easily using the following equations.
Kp = 0.6Ku
Ki = 1.2Ku/Tu
Kd = 3KuTu/40
The PID parameters are input into the code and almost perfectly stabilize the RC Car platform. With the RC Car platform mostly stable, parameters must be graphed to tune the controller further and improve stability, especially around curves. The process was repeated with the optical encoder to maintain a consistent speed around all portions of the track. However, problems were encountered due to vibrations where incorrect data was reported from the optical encoder. A combination of hardware debouncing and better mechanical connection between the RC Car platform and optical encoder rectified these issues. A Linux module is modified and used to calculate the speed from data generated by the encoder.
As discussed in earlier paragraphs, stop box detection is achieved in a relatively primitive manner due to an elementary requirement scope - stop the RC Car platform when it encounters a red square on the track. Upon encountering the red square a second time, the controller permanently stops the RC Car platform. The process of stop box detection involves utilizing a calibrated red HSV mask to the openCV frames used as inputs for steering, followed by a Region-of-Interest (ROI) mask. This mask adjusts for the camera angle and eliminates false stops, such as premature stops or multiple erroneous stops. In contrast to the red stop box detection, a Machine Learning approach was used to implement STOP sign detection. The team used a combination of YoloV4 custom libraries for traffic sign detection in conjunction with openCV. YoloV4 is an ML solution pre-trained on a dataset of nearly 900 images. This system was given every third openCV frame to allow for faster data processing at the expense of worsened response time due to the BeagleBone’s relatively low compute power.
References:
D. Rothfusz, L. Ivory, R. Bose, I. Scott, and B. Stanley, “Autonomous Path following car, ” Hackster.io, 13-Dec-2021. [Online]. Available: https://www.hackster.io/really-bad-idea/autonomous-path-following-car-6c4992. [Accessed: 09-Dec-2022].
Fredotran, “Fredotran/traffic-sign-detector-yolov4: This repository contains my upgraded version of using Yolov4 with opencv DNN to detect 4 classes of traffic road signs: Traffic lights, speed limit signs, crosswalk and stop signs., ” GitHub. [Online]. Available: https://github.com/fredotran/traffic-sign-detector-yolov4. [Accessed: 09-Dec-2022].
J. G. Ziegler and N. B. Nichols, “Optimum settings for automatic controllers, ” Journal of Dynamic Systems, Measurement, and Control, vol. 115, no. 2B, pp. 220–222, 1993.
“OpenCV modules, ” OpenCV. [Online]. Available: https://docs.opencv.org/4.x/. [Accessed: 09-Dec-2022].
raja_961 and Instructables, “Autonomous Lane-keeping car using Raspberry Pi and opencv, ” Instructables, 07-Oct-2022. [Online]. Available: https://www.instructables.com/Autonomous-Lane-Keeping-Car-Using-Raspberry-Pi-and/. [Accessed: 09-Dec-2022].
W. Kim, Y. Wu, and reddot313, “Gradudes, ” Hackster.io, 14-Dec-2021. [Online]. Available: https://www.hackster.io/gradudes/gradudes-890fe4. [Accessed: 09-Dec-2022].
Comments