FPGA Camera Edge Detection

This project will show how to implement a complete end-to-end image processing pipeline to implement an edge detection vision algorithm capable at running at 30fps in order to detection road lanes.

1. Introduction

To implement real-time lane detection the idea is to take the image and pass it through an edge detector IP. The most well known one is the Sobel edge detector. The Canny edge detector however, has the advantage that it has programmable thresholds. Modifying the thresholds on the fly allows for adaptation under different lighting conditions.

https://en.wikipedia.org/wiki/Canny_edge_detector

The vision processing pipeline can be logically split into three sections. The image source, an image processing pipeline and the image sink.

From a high-level point of view, one needs an image source which can be a camera sensor or an image pattern generator and a sink which can be a display.

2. Hardware blocks

First, all the prototyping hardware parts have to be put together. In this case the system hardware is composed of only five components.

a) MiniZed ZYNQ development board

b) OV7670 Camera

c) Arduino shield perfboard

d) 10pin 0.1inch Female header

e) VGA PMOD

TheOV7670 camera was mounted on a 10pin female 0.1 inch header which itself was soldered to a prototyping board. The VGA PMOD was connected to the MiniZed dual PMOD header. The last step is to connect a VGA cable to the monitor.

Once the hardware system is assembled together the next step is to design the hardware on the FPGA (PL) using Vivado.

3. FPGA hardware

1. Image source

There are a number of camera interfaces but the simplest ones are CMOS sensors that use a parallel bus like the venerable OV7670 camera sensor. To use such a sensor however one has to implement a camera capture module that converts the byte stream piped from the sensor into the proper protocol which in our case is the AXIS bus.

In addition the camera has to be configured by either implementing the configuration module in the PL as a I2C module that reads the configuration from a BRAM or in the PS as a generic camera driver. The PS configuration is obviously more flexible so I picked this option.

The image source on the Vivado design is a custom IP that interfaces with the OV7690 CMOS cameras. The camera uses a parallel eight-bit interface. In addition, there are two synchronization signals HSYNC and VSYNC whose strobing pattern denote row and frames. The camera uses an output pixel clock PCLK. The pixel rate is synchronized to PCLK. The OV7690 camera also requires a 24 MHz (XCLK) camera clock as an input.

The OV7690 outputs 2 bytes per pixel. Depending on the configuration code the camera can be configured to use different color spaces. The IP is configured to use the RGB565 color space. The control of the camera is done via the AXI interface. The camera IP is assigned a location in memory. To start the camera, one has to set the bit in order to enable the camera output data. The data itself is packed in using the AXIS (AXI Stream) protocol.

2. Crossing clock domains

The output of the AXIS from the camera is sent to an asynchronous FIFO. This is one method that is used whenever one needs to cross two different clock domains.

The slave side of the FIFO is connected to PCLK while the master side of the FIFO is connected to the AXI master clock domain that operates at 50MHz. The master clock must always be higher than the slave clock otherwise the FIFO will overflow resulting in missed pixels.

After the pixel stream crosses the clock domain it is sent through a subset converter. This is an AXIS IP block which remaps the RGB565 data to a 24 bit packet. The 24 bit is used by the image processing IP since each color is assigned 8 bits.

The next element in line is the AXIS switch. This IP operates as a simple multiplexer for the AXIS stream. It can be configured either via an AXI lite interface or automatically by leveraging the AXIS strobe signals. In this particular application the AXI Lite interface of the AXIS Switch is enables. This however requires implementing the SDK drivers to configure the switch mutliplexer.

3. Image filtering

The Canny edge detector IP was built using Vivado HLS. It was coded in C. The main advantage of HLS, is that allows for rapid deployment of IP. The disadvantage is that the generated logic is obfuscated and usually not as efficient as hand crafted Verilog code although it can approach it. In any case generating the Canny IP involved making use of Vivado HLS 2013 and using the 2018.3 version of xfopencv. This is a set of C++ libraries that replicates the well known OpenCV library for FPGA logic.

Xilinx has recently released Vivado 2019.1 as well as HLS 2019.1. This required using SDSOC with the revision framework which is not free.

The main change to the revision code that was implemented was to modify the function to support AXIS protocol for input and output. In addition the AXI Lite bus was used to bundle all the programmable variables in a set and expose them to the PS memeory map via an AXI interconnect connected to the GP0 ZYNQ bus.

Finally to switch between the two different IP, an AXI switch was used. This IP can basically be used as a programmable multiplexer or demultiplexer as was meantioned. The IP allows the original image frame pixels to be routed either to the Canny edge detector, the RGB to grayscale detector or the pass-through channel which contains co image processing IP’s.

4.VDMA

The output of the Canny edge detector is a 24 bit pixel stream so it is sent to another subset converter which converts the pixels back to 16 bit. The pixels are then piped to a VDMA IP. The VDMA is configured in both read and write mode. The purpose of the VDMA is to write the pixel stream in a contiguous area in DRAM so that the PS can access the images. The pixel stream is then read and written to AXIS video out.

The VDMA is configured in triple buffering mode. The VDMA needs to be configured prior to operation via the AXI lite interface which connects it to the main AXI Interconnect.

The code below shows the configuration of the VDMA for a VGA resolution.

	Xil_DCacheFlush();    
	Xil_ICacheInvalidate();    
	/* Start of VDMA Configuration */    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0x30, 0x8B);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0xAC, 0x10000000);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0xB0, 0x100F0000);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0xB4, 0x101E0000);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0xA8, WIDTH*2);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0xA4, WIDTH*2);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0xA0, HEIGHT);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0x00, 0x8B);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0x5C, 0x10000000);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0x60, 0x100F0000);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0x64, 0x101E0000);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0x58, WIDTH*2);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0x54, WIDTH*2);    
   Xil_Out32(XPAR_VDMA_S2MM_BASEADDR + 0x50, HEIGHT);     

The output of the VDMA is sent to DDR3 RAM via the High performance interface HP0 while the VDMA MM2S interface is sent to the video sink block.

5. Video sink (VGA interface)

The MiniZed makes use of an external VGA break-out board. The Video AXIS out IP interfaces with the extrenal VGA peripheral. This IP takes the video data from the VDMA and the timing information from the VTC (Video Timing controller).

The VTC itself is configured for a VGA resolution.

This is basically an R2RDAC which takes a digital data bus together with row and frame synchronization signals HSYNCS and VSYNC and sends the data to the monitor.

As you can see below the application switches continuously between normal VGA mode, gray mode (which looks more like bluish) and edge detection mode.

Summary

This project showed how to implement a complete end to end vision processing pipeline on a ZYNQ FPGA SoC. The hardware is capable of real-time edge detection at 30 fps.

Additional ideas include:

• Add more filter types such as median, Guassian, Sobel, Prewit, etc.
• Build a PYNQ distro for MiniZed to access the IP from Linux in order to build a wireless WiFi camera.
• Upgrade to higher definition camera sensors.