Creating an FPGA-Based Low-Cost Imaging System

Introduction

Developing an embedded vision system does not require the use of an expensive FPGA or SoC, large frame buffer memories and a external camera.

We can develop a very powerful image processing system using a cost optimized FPGA / SoC interfacing directly with a CMOS sensor. This allows a solution to be created which is not only achieves cost targets but also one which is compact and power efficient.

Interfacing directly with a sensor is different to interfacing with a camera as we have done previously. When we interface with a camera we receive a video signal over HDMI, CameraLink etc which is fairly straight forward.

When we interface with the image sensor we receive the images typically in a different format e.g. MIPI or Parallel and before we can receive the video we need to first configure the imager to operate as we desire.

Typically imagers require configuration over I2C or SPI, and the number of commands sent over interface can be significant.

To demonstrate how we can integrate a sensor with a cost optimized FPGA for this project we are going to look at integrating the

• TDNext 1.26 Mega Pixel Pmod

• Arty S7-50

As the Arty S7 does not provide a HDMI or other video output directly on board, this example will use a Avnet 10 inch touch screen. However, this is optional another option for outputting the final image is the Digilent Pmod VGA. This Pmod could also be used to implement a very low cost solution.

The interfacing with the TDNext Pmod is pretty simple and can be split into two elements the Video and Configuration.

• Video Interface consisting of 10 bit pixel (split 8 bit and 2 bit LSB) Frame and Line valid, Pixel clock and a Reference clock (24 MHz).

• Configuration interface consisting of I2C connected to both the imaging device and a I2C IO expander to generate a reset to the imager.

The architecture of the solution will be as follows, a softcore processor will be used to configure the imager over I2C. While the image processing path will be implemented within the FPGA however, as a this is a low cost application the solution will not implement a external frame buffer in DDR memory instead the image processing pipeline will be implemented entirely in the FPGA.

The design will also use the softcore processor to control the Video Timing and other associated configuration tasks of the image processing path.

Background

The TDNext is a color imager, this means that the imager has a Bayer pattern applied which filters the wavelength for each pixel. This means each pixel only accumulates photons in either the Red, Green or Blue wavelengths, during the integration period.

At completion of the integration time each pixel is read out as either a 8 bit or 10 bit pixel. This pixel value is referred to as a RAW8 or RAW10 pixel. To recreate a color image, the values of surrounding pixels which contain pixels of different wavelengths are combined using a de-Bayering algorithm.

Vivado Build

The first thing we need to do is create the Vivado platform, this will receive the images from the TDNext Pmod.

To create the block diagram we will be using mostly IP cores from the Vivado library however we will use a camera interface block and a output block from the Avnet library which is available here.

The first step is to install the board definition files, this enables Vivado to understand the configuration of Arty S7. You can download the board definition files from here.

Once downloaded these files should be installed in your Vivado directory under the following path:

<Install path>/Vivado/<version>/data/boards/board_files/

This will allow you to then select the Arty S7 board as your target board for the creation of a new Vivado project.

With the board installed, the next step is to create a new project, block diagram and create the MicroBlaze system.

The fastest way to create a MicroBlaze system is to follow the steps in the video below

With the MicroBlaze system up and running, the next step is to add in the video processing pipeline. The processing chain will use the following IP blocks

• CAM Interface - This interfaces with the TDNext video interface

• Video to AXIS - This converts parallel video to AXI Streaming format

• Sensor Demosaic - This converts the RAW pixel values which represent either R, G or B into a 24 bit RGB format

• Video Timing Generator - This generates the video timing signals for the output format

• AXI Stream to Video Out - Converts AXI Stream to parallel video

• ZED ALI3 Controller - IP module to drive the 10 inch touch display

• AXI IIC - Attached to MicroBlaze this will be used to configure the imager

• AXI UART - Attached to MicroBlaze used to report system status to user

If we are using the Pmod VGA we do not need to use the ZED ALI3 Controller IP block.

Before we can add in the Zed ALI3 and CAM Interface we need to reconfigure the IP core to be able be included in Spartan 7 designs. We do this from the IP catalog view, selecting the desired IP cores and clicking edit IP in packager.

This will open a new project and enable you to select the comparability tab and add in support for the Spartan 7 device. Repackage the design and update the IP lib in the Vivado project.

Once the IP have been upgraded to support the Spartan 7 we are then in a position to complete the design. The complete bock diagram should look as below. It is available on my GitHub Account for a closer look.

Unlike previous examples based upon heterogeneous SoC where used an external frame buffer. This example will not use VDMA to read and write from an external frame buffer, such a approach requires a different configuration of the AXIS to Video and VTC.

Normally the AXIS to video is configured as the master and the VTC is not controlled. However, in this approach the AXIS to video is configured as a slave and the VTC generator clock enable is controlled.

This approach allows the AXIS to Video IP module to control the timing of syncs by enabling and disabling the VTC, so they match the timing of the Syncs in the processing pipeline.

Within the AXI Stream the start of frame is indicated by the use of TUser and the end of line is indicated by the use of TLast.

Key customization for the IP blocks are :-

Video In to AXI 4 Stream

Sensor Demosaic Settings

AXI IIC Settings

In the design I have also added in several Integrated Logic Analyzer (ILA) to enable internal monitoring of the status of the system and debugging.

The total utilization of the Arty S7-50 when the project is completed is shown below.

We can use the additional resources to implement image processing algorithms using HLS is necessary. If we want to save resources, we can use the minimum footprint for the MicroBlaze and remove the ILAs.

Writing the Software in SDK

Once the Vivado hardware has been generated the next step is to write application software which will configure the imager and the IP cores on the video processing core.

As such the software will perform the following

• Initialize the AXI IIC, VTC and Interrupt controller

• Set up the interrupt controller to generate AXI related interrupts - this include three interrupt service routines. One each for IIC transmit, receive and status.

• Configure the timing on the VTC for the 10 inch display

• Reset the camera over I2C and illuminate the lED on the PMOD

• Detect the camera is present over I2C we are looking to detect a MT9M114

• Initialize the camera over the I2C link - This will take a few seconds to write all of the commands

To initialize the imager, I have converted the Zynq based libraries as provided with the TDM114 example design into a format which can be used with the AXI IIC.

Once the camera has been initialized we will be able to see video on the ILA which is connected to the video stream to AXI Stream component.

Monitoring the I2C communication on the back of the TDNext Pmod shows the communication between the Arty S7 and the TDNext.

Once the camera has been detected, the application will download several I2C camera configuration settings.

Progress will be reported using the AXI UART

Once the camera is initialized we can use the ILA to verify, video is being generated by the imager and it is at the resolution we configured.

We do this by using a ILA and examining the video directly as it is received within the FPGA.

The image above shows a line width of 1280 pixel, which is exactly what we are expecting.

Received pixels are converted from parallel format into an AXI Stream.

AXI Stream is a unidirectional bus used to transfer data from a master to a slave, as a stream of data it does not contain an address channel. To control flow and communicate video timing information over the AXI Stream the following signals are used

• TReady - Asserted bythe downstream peripheral when ready to receive data

• TValid - Asserted bytransmitting peripheral when output data is valid

• TUser - Issued for start of Frame

• TLast - Issued for end of line

The second ILA can be used to ensure the AXI Stream is being correctly generated.

As we have no VDMA it is important the video output on the AXIS stream is a contiguus block and that TValid does not assert and deassert during the active pixel period.

We can ensure Tvalid is contiguous by using the pixel clock for the image processing chain.

The Library API used in this project are below, with the exception of camera_initial.h which contains IIC configuration data. All other header files are provided by Xilinx based upon the hardware configuration.

#include <stdio.h>
#include "platform.h"
#include "xiic.h"
#include "xintc.h"
#include "xil_exception.h"
#include "camera_initial.h"
#include "xvtc.h"
#include "xv_demosaic.h"

Devices addresses and identifiers

#define IIC_dev 			XPAR_IIC_0_DEVICE_ID
#define int_dev 			XPAR_INTC_0_DEVICE_ID
#define IIC_SLAVE_ADDR		        0x48
#define PCA9534_IIC_ADDR                0x20
#define INTC_DEVICE_INT_ID	        XPAR_INTC_0_IIC_0_VEC_ID
#define BUFFER_SIZE		        6

The main loop of the application can be seen below

int main()
{
   u32  Status;
   XIic_Config *iic_conf;
   XVtc	VtcInst;
   XVtc_Config *vtc_config;
   XVtc_Timing vtcTiming;
   XVtc_SourceSelect SourceSelect;
   XV_demosaic_Config *mosaic_config;
   init_platform();
   printf("www.adiuvoengineering.com S7 Imager example\n\r");
   mosaic_config = XV_demosaic_LookupConfig(XPAR_XV_DEMOSAIC_0_DEVICE_ID);
   XV_demosaic_CfgInitialize(&mosaic,mosaic_config,mosaic_config->BaseAddress);
   XIntc_Initialize(&InterruptController, int_dev);
   SetUpInterruptSystem();
   iic_conf = XIic_LookupConfig(IIC_dev);
   Status =  XIic_CfgInitialize(&iic, iic_conf, iic_conf->BaseAddress);
   if (Status != XST_SUCCESS) {
	 printf("XIic initial is fail \n \r") ;
	return XST_FAILURE;
   }
   XIic_SetSendHandler(&iic, &iic,	(XIic_Handler) SendHandler);
   XIic_SetRecvHandler(&iic, &iic, (XIic_Handler) ReceiveHandler);
   XIic_SetStatusHandler(&iic, &iic,(XIic_StatusHandler) StatusHandler);
   vtc_config = XVtc_LookupConfig(XPAR_VTC_0_DEVICE_ID);
   XVtc_CfgInitialize(&VtcInst, vtc_config, vtc_config->BaseAddress);
   vtcTiming.HActiveVideo = 1280;
   vtcTiming.HFrontPorch = 65;
   vtcTiming.HSyncWidth = 55;
   vtcTiming.HBackPorch = 40;
   vtcTiming.HSyncPolarity = 0;
   vtcTiming.VActiveVideo = 800;
   vtcTiming.V0FrontPorch = 7;//8;
   vtcTiming.V0SyncWidth = 4;
   vtcTiming.V0BackPorch = 12;
   vtcTiming.V1FrontPorch = 7;
   vtcTiming.V1SyncWidth = 4;
   vtcTiming.V1BackPorch = 12;
   vtcTiming.VSyncPolarity = 0;
   vtcTiming.Interlaced = 0;
   memset((void *)&SourceSelect, 0, sizeof(SourceSelect));
   SourceSelect.VBlankPolSrc = 1;
   SourceSelect.VSyncPolSrc = 1;
   SourceSelect.HBlankPolSrc = 1;
   SourceSelect.HSyncPolSrc = 1;
   SourceSelect.ActiveVideoPolSrc = 1;
   SourceSelect.ActiveChromaPolSrc= 1;
   SourceSelect.VChromaSrc = 1;
   SourceSelect.VActiveSrc = 1;
   SourceSelect.VBackPorchSrc = 1;
   SourceSelect.VSyncSrc = 1;
   SourceSelect.VFrontPorchSrc = 1;
   SourceSelect.VTotalSrc = 1;
   SourceSelect.HActiveSrc = 1;
   SourceSelect.HBackPorchSrc = 1;
   SourceSelect.HSyncSrc = 1;
   SourceSelect.HFrontPorchSrc = 1;
   SourceSelect.HTotalSrc = 1;
   XVtc_RegUpdateEnable(&VtcInst);
   XVtc_SetGeneratorTiming(&VtcInst,&vtcTiming);
   XVtc_SetSource(&VtcInst, &SourceSelect);
   XVtc_EnableGenerator(&VtcInst);
   XIic_Reset(&iic);
   PCA9534_CTRL ();
   Detect_Camera();
   Soft_Reset_Camera();
   Initial_Camera();
   
   XV_demosaic_Set_HwReg_width(&mosaic,0x500);
   XV_demosaic_Set_HwReg_height(&mosaic,0x31f);
   XV_demosaic_Set_HwReg_bayer_phase(&mosaic,0x1);
   XV_demosaic_EnableAutoRestart(&mosaic);
   XV_demosaic_Start(&mosaic);
   while(1){
   }
   cleanup_platform();
   return 0;
} 

Running the Entire software application resulted in me capturing the image below of the my conference badge collection.

I need to adjust some of the settings to provide to increase the integration time however, the basic image processing pipeline is working as we would expect.

Conclusion

It is easy to create a vision processing system which works directly with the imager in place of a camera. This often allows for a more cost efficient and potentially a more responsive solution as processing chain is significantly reduced.

Of course, it is also possible to include in HLS image processing blocks within the image processing chain if desired as we have done previously.

You can find the files associated with this project here:

https://github.com/ATaylorCEngFIET/Hackster

See previous projects here.

More on on Xilinx using FPGA development weekly at MicroZed Chronicles.