Hardware components | ||||||
 |
| × | 1 | |||
| × | 1 | ||||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
FPGA are great for creating embedded image processing applications and several times I have created projects which are implemented using Zynq, Zynq MPSoC or MicroBlaze processors.
One of the benefit of using an FPGA for image processing is the ability to implement a image processing pipeline which provides a high frame rate, low latency and deterministic path.
Such a pipeline can be implemented easily in small FPGA's like the Spartan Seven, what makes this very exciting is the IO structures can support HMDI, MIPI CSI-2 and DSI interfaces.
Of course to configure and control the image processing pipeline we need to need either a custom IP block or a small processor. This is where the Arm Cortex-M1 processor can come into play.
The Arm Cortext-M1 has a very small foot print to implement in a Spartan 7 FPGA and can control the image processing chain.
In this project I am going to demonstrate how we can create such an image processing chain using the SP701 as a target board. I selected the board as it conveniently provides a MIPI CSI input and HDMI output.
So I will note straight off that yes MIPI cores can be expensive however, we can obtain trial licenses easily for the Xilinx CSI Receiver Sub System from the Xilinx License manager. Down the line I understand there are some exciting developments relating to the accessibility of these cores.
For this project I am going to be using Vivado 2019.2 and Vitis along with Keil MDK.
Hardware DesignLike any project we will be getting started with a Vivado design which includes the image processing chain and the Arm Cortex-M1 processor.
To complete this design we will need the following IP blocks
- MIPI CSI-2 Rx Subsystem - this will receive the MIPI image from the camera and output it using a AXI Stream
- DeMosiac - This will convert the RAW output from the sensor into RGB Pixel
- VDMA - This will store the image in a frame buffer in external memory
- MIG - Memory interface generator, this will provide a interface to external DDR memory.
- Video Timing Controller - This will generate the parallel video timing signals for the output video.
- RGB2CYRB - Color space converter from RGB to YUV suitable for output.
- AXI IIC - To provide the IIC configuration of the MIPI Imager and the HDMI output
- AXI GPIO - Provides the enable signal to the MIPI Camera
- AXI Uart - Provides a simple serial out
- ILA -provides information on debugging
- Arm Cortex-M1 - processor configured with no debug and 128KB of instruction and data memory
To add in the Arm Cortex-M1 processor we need to down the processor from the Arm DesignStart FPGA website.
We can then add in the processor which is available under the Vivado directory in the downloaded M1 example.
The IPI repository can be added into the project using the project settings, once this has been added you are able to use the processor in designs.
The architecture of the design looks like below when completed.
Once the constraint file below is entered we can implement the design and generate the bit stream.
create_clock -period 5.000 [get_ports sys_diff_clock_clk_p]
set_property PACKAGE_PIN C13 [get_ports "Pmod_out_0_pin1_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L10N_T1_16
set_property IOSTANDARD LVCMOS33 [get_ports "Pmod_out_0_pin1_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L10N_T1_16
set_property PACKAGE_PIN D14 [get_ports "Pmod_out_0_pin2_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L15N_T2_DQS_16
set_property IOSTANDARD LVCMOS33 [get_ports "Pmod_out_0_pin2_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L15N_T2_DQS_16
set_property PACKAGE_PIN B14 [get_ports "Pmod_out_0_pin3_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L6P_T0_16
set_property IOSTANDARD LVCMOS33 [get_ports "Pmod_out_0_pin3_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L6P_T0_16
set_property PACKAGE_PIN B15 [get_ports "Pmod_out_0_pin4_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L6N_T0_VREF_16
set_property IOSTANDARD LVCMOS33 [get_ports "Pmod_out_0_pin4_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L6N_T0_VREF_16
set_property PACKAGE_PIN A13 [get_ports "Pmod_out_0_pin7_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_0_16
set_property IOSTANDARD LVCMOS33 [get_ports "Pmod_out_0_pin7_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_0_16
set_property PACKAGE_PIN C14 [get_ports "Pmod_out_0_pin8_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L11P_T1_SRCC_16
set_property IOSTANDARD LVCMOS33 [get_ports "Pmod_out_0_pin8_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L11P_T1_SRCC_16
set_property PACKAGE_PIN A14 [get_ports "Pmod_out_0_pin9_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L1P_T0_16
set_property IOSTANDARD LVCMOS33 [get_ports "Pmod_out_0_pin9_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L1P_T0_16
set_property PACKAGE_PIN A15 [get_ports "Pmod_out_0_pin10_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L1N_T0_16
set_property IOSTANDARD LVCMOS33 [get_ports "Pmod_out_0_pin10_io"] ;# Bank 16 VCCO - VCCO_3V3 - IO_L1N_T0_16
set_property IOSTANDARD LVDS_25 [get_ports sys_diff_clock_clk_n]
set_property PACKAGE_PIN AE8 [get_ports sys_diff_clock_clk_p]
set_property PACKAGE_PIN AE7 [get_ports sys_diff_clock_clk_n]
set_property IOSTANDARD LVDS_25 [get_ports sys_diff_clock_clk_p]
set_property PACKAGE_PIN AA20 [get_ports reset_rtl_0]
set_property IOSTANDARD LVCMOS18 [get_ports reset_rtl_0]
set_property PACKAGE_PIN AE15 [get_ports reset]
set_property IOSTANDARD LVCMOS18 [get_ports reset]
set_property INTERNAL_VREF 0.675 [get_iobanks 34]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[0]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[10]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[11]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[12]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[13]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[14]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[1]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[2]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[3]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[4]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[5]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[6]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[7]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[8]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_addr[9]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_ba[0]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_ba[1]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_ba[2]}]
set_property IOSTANDARD SSTL135 [get_ports ddr3_sram_cas_n]
set_property IOSTANDARD DIFF_SSTL135 [get_ports {ddr3_sram_ck_n[0]}]
set_property IOSTANDARD DIFF_SSTL135 [get_ports {ddr3_sram_ck_p[0]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_cke[0]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dm[0]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dm[1]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[0]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[10]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[11]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[12]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[13]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[14]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[15]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[1]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[2]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[3]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[4]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[5]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[6]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[7]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[8]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_dq[9]}]
set_property IOSTANDARD DIFF_SSTL135 [get_ports {ddr3_sram_dqs_n[0]}]
set_property IOSTANDARD DIFF_SSTL135 [get_ports {ddr3_sram_dqs_n[1]}]
set_property IOSTANDARD DIFF_SSTL135 [get_ports {ddr3_sram_dqs_p[0]}]
set_property IOSTANDARD DIFF_SSTL135 [get_ports {ddr3_sram_dqs_p[1]}]
set_property IOSTANDARD SSTL135 [get_ports {ddr3_sram_odt[0]}]
set_property IOSTANDARD SSTL135 [get_ports ddr3_sram_ras_n]
set_property IOSTANDARD SSTL135 [get_ports ddr3_sram_reset_n]
set_property IOSTANDARD SSTL135 [get_ports ddr3_sram_we_n]
##############################################################################
set_property PACKAGE_PIN Y5 [get_ports {ddr3_sram_addr[0]}]
set_property PACKAGE_PIN W4 [get_ports {ddr3_sram_addr[10]}]
set_property PACKAGE_PIN V6 [get_ports {ddr3_sram_addr[11]}]
set_property PACKAGE_PIN W5 [get_ports {ddr3_sram_addr[12]}]
set_property PACKAGE_PIN AB5 [get_ports {ddr3_sram_addr[13]}]
set_property PACKAGE_PIN V7 [get_ports {ddr3_sram_addr[14]}]
set_property PACKAGE_PIN U5 [get_ports {ddr3_sram_addr[1]}]
set_property PACKAGE_PIN AA3 [get_ports {ddr3_sram_addr[2]}]
set_property PACKAGE_PIN U6 [get_ports {ddr3_sram_addr[3]}]
set_property PACKAGE_PIN AA5 [get_ports {ddr3_sram_addr[4]}]
set_property PACKAGE_PIN U7 [get_ports {ddr3_sram_addr[5]}]
set_property PACKAGE_PIN AA7 [get_ports {ddr3_sram_addr[6]}]
set_property PACKAGE_PIN T7 [get_ports {ddr3_sram_addr[7]}]
set_property PACKAGE_PIN Y8 [get_ports {ddr3_sram_addr[8]}]
set_property PACKAGE_PIN AB4 [get_ports {ddr3_sram_addr[9]}]
set_property PACKAGE_PIN T4 [get_ports {ddr3_sram_ba[0]}]
set_property PACKAGE_PIN AA4 [get_ports {ddr3_sram_ba[1]}]
set_property PACKAGE_PIN W3 [get_ports {ddr3_sram_ba[2]}]
set_property PACKAGE_PIN U4 [get_ports ddr3_sram_cas_n]
set_property PACKAGE_PIN W6 [get_ports {ddr3_sram_ck_p[0]}]
set_property PACKAGE_PIN Y6 [get_ports {ddr3_sram_ck_n[0]}]
set_property PACKAGE_PIN Y7 [get_ports {ddr3_sram_cke[0]}]
set_property PACKAGE_PIN R5 [get_ports {ddr3_sram_dm[0]}]
set_property PACKAGE_PIN Y3 [get_ports {ddr3_sram_dm[1]}]
set_property PACKAGE_PIN R2 [get_ports {ddr3_sram_dq[0]}]
set_property PACKAGE_PIN Y1 [get_ports {ddr3_sram_dq[10]}]
set_property PACKAGE_PIN U1 [get_ports {ddr3_sram_dq[11]}]
set_property PACKAGE_PIN AB2 [get_ports {ddr3_sram_dq[12]}]
set_property PACKAGE_PIN V1 [get_ports {ddr3_sram_dq[13]}]
set_property PACKAGE_PIN AC1 [get_ports {ddr3_sram_dq[14]}]
set_property PACKAGE_PIN V3 [get_ports {ddr3_sram_dq[15]}]
set_property PACKAGE_PIN R3 [get_ports {ddr3_sram_dq[1]}]
set_property PACKAGE_PIN P1 [get_ports {ddr3_sram_dq[2]}]
set_property PACKAGE_PIN T5 [get_ports {ddr3_sram_dq[3]}]
set_property PACKAGE_PIN R1 [get_ports {ddr3_sram_dq[4]}]
set_property PACKAGE_PIN R7 [get_ports {ddr3_sram_dq[5]}]
set_property PACKAGE_PIN P3 [get_ports {ddr3_sram_dq[6]}]
set_property PACKAGE_PIN T2 [get_ports {ddr3_sram_dq[7]}]
set_property PACKAGE_PIN Y2 [get_ports {ddr3_sram_dq[8]}]
set_property PACKAGE_PIN W1 [get_ports {ddr3_sram_dq[9]}]
set_property PACKAGE_PIN P5 [get_ports {ddr3_sram_dqs_p[0]}]
set_property PACKAGE_PIN P4 [get_ports {ddr3_sram_dqs_n[0]}]
set_property PACKAGE_PIN AA2 [get_ports {ddr3_sram_dqs_p[1]}]
set_property PACKAGE_PIN AB1 [get_ports {ddr3_sram_dqs_n[1]}]
set_property PACKAGE_PIN T3 [get_ports {ddr3_sram_odt[0]}]
set_property PACKAGE_PIN U2 [get_ports ddr3_sram_ras_n]
set_property PACKAGE_PIN P6 [get_ports ddr3_sram_reset_n]
set_property PACKAGE_PIN V4 [get_ports ddr3_sram_we_n]
###################################################################################
#csi-phy
##For SP701 swapped pins
set_property PACKAGE_PIN AC12 [get_ports {mipi_phy_if_0_data_lp_p[1]}]
set_property PACKAGE_PIN AE12 [get_ports {mipi_phy_if_0_data_lp_n[1]}]
set_property PACKAGE_PIN AF10 [get_ports {mipi_phy_if_0_data_lp_p[0]}]
set_property PACKAGE_PIN AF9 [get_ports {mipi_phy_if_0_data_lp_n[0]}]
set_property PACKAGE_PIN AC9 [get_ports mipi_phy_if_0_clk_lp_p]
set_property PACKAGE_PIN AD8 [get_ports mipi_phy_if_0_clk_lp_n]
set_property PACKAGE_PIN AD10 [get_ports {mipi_phy_if_0_data_hs_p[0]}]
set_property PACKAGE_PIN AD9 [get_ports {mipi_phy_if_0_data_hs_n[0]}]
set_property PACKAGE_PIN AA9 [get_ports {mipi_phy_if_0_data_hs_p[1]}]
set_property PACKAGE_PIN AB9 [get_ports {mipi_phy_if_0_data_hs_n[1]}]
set_property PACKAGE_PIN AA10 [get_ports mipi_phy_if_0_clk_hs_p]
set_property PACKAGE_PIN AB10 [get_ports mipi_phy_if_0_clk_hs_n]
##Use for Switching between IO standards
set_property IOSTANDARD HSUL_12 [get_ports {mipi_phy_if_0_data_lp_n[1]}]
set_property IOSTANDARD HSUL_12 [get_ports {mipi_phy_if_0_data_lp_p[1]}]
set_property IOSTANDARD HSUL_12 [get_ports {mipi_phy_if_0_data_lp_n[0]}]
set_property IOSTANDARD HSUL_12 [get_ports {mipi_phy_if_0_data_lp_p[0]}]
set_property IOSTANDARD HSUL_12 [get_ports mipi_phy_if_0_clk_lp_n]
set_property IOSTANDARD HSUL_12 [get_ports mipi_phy_if_0_clk_lp_p]
set_property IOSTANDARD LVDS_25 [get_ports {mipi_phy_if_0_data_hs_n[1]}]
set_property IOSTANDARD LVDS_25 [get_ports mipi_phy_if_0_clk_hs_n]
set_property IOSTANDARD LVDS_25 [get_ports {mipi_phy_if_0_data_hs_p[1]}]
set_property IOSTANDARD LVDS_25 [get_ports mipi_phy_if_0_clk_hs_p]
set_property IOSTANDARD LVDS_25 [get_ports {mipi_phy_if_0_data_hs_n[0]}]
set_property IOSTANDARD LVDS_25 [get_ports {mipi_phy_if_0_data_hs_p[0]}]
#csi-gpio
set_property PACKAGE_PIN AF12 [get_ports {GPIO_sensor_tri_o[0]}]
set_property IOSTANDARD LVCMOS25 [get_ports {GPIO_sensor_tri_o[0]}]
#csi-iic0
set_property PACKAGE_PIN F17 [get_ports IIC_expander_scl_io]
set_property IOSTANDARD LVCMOS33 [get_ports IIC_expander_scl_io]
set_property PULLUP true [get_ports IIC_expander_scl_io]
set_property PACKAGE_PIN F18 [get_ports IIC_expander_sda_io]
set_property IOSTANDARD LVCMOS33 [get_ports IIC_expander_sda_io]
set_property PULLUP true [get_ports IIC_expander_sda_io]
#csi-iic1
set_property PACKAGE_PIN AD13 [get_ports IIC_sensor_scl_io]
set_property IOSTANDARD LVCMOS25 [get_ports IIC_sensor_scl_io]
set_property PULLUP true [get_ports IIC_sensor_scl_io]
set_property PACKAGE_PIN AE13 [get_ports IIC_sensor_sda_io]
set_property IOSTANDARD LVCMOS25 [get_ports IIC_sensor_sda_io]
set_property PULLUP true [get_ports IIC_sensor_sda_io]
#uart
set_property PACKAGE_PIN Y21 [get_ports rs232_uart_txd]
set_property IOSTANDARD LVCMOS33 [get_ports rs232_uart_txd]
set_property PACKAGE_PIN Y22 [get_ports rs232_uart_rxd]
set_property IOSTANDARD LVCMOS33 [get_ports rs232_uart_rxd]
#dsi-phy
set_property PACKAGE_PIN AD15 [get_ports {mipi_phy_if_1_data_hs_p[3]}]
set_property PACKAGE_PIN AE16 [get_ports {mipi_phy_if_1_data_hs_n[3]}]
set_property PACKAGE_PIN AC16 [get_ports {mipi_phy_if_1_data_hs_p[2]}]
set_property PACKAGE_PIN AD16 [get_ports {mipi_phy_if_1_data_hs_n[2]}]
set_property PACKAGE_PIN AB17 [get_ports {mipi_phy_if_1_data_hs_p[1]}]
set_property PACKAGE_PIN AC17 [get_ports {mipi_phy_if_1_data_hs_n[1]}]
set_property PACKAGE_PIN AD20 [get_ports {mipi_phy_if_1_data_hs_p[0]}]
set_property PACKAGE_PIN AE21 [get_ports {mipi_phy_if_1_data_hs_n[0]}]
set_property PACKAGE_PIN AE18 [get_ports mipi_phy_if_1_clk_hs_p]
set_property PACKAGE_PIN AF19 [get_ports mipi_phy_if_1_clk_hs_n]
set_property PACKAGE_PIN AA17 [get_ports {mipi_phy_if_1_data_lp_p[3]}]
set_property PACKAGE_PIN AA18 [get_ports {mipi_phy_if_1_data_lp_n[3]}]
set_property PACKAGE_PIN AF17 [get_ports {mipi_phy_if_1_data_lp_p[2]}]
set_property PACKAGE_PIN AF18 [get_ports {mipi_phy_if_1_data_lp_n[2]}]
set_property PACKAGE_PIN AC18 [get_ports {mipi_phy_if_1_data_lp_p[1]}]
set_property PACKAGE_PIN AD18 [get_ports {mipi_phy_if_1_data_lp_n[1]}]
set_property PACKAGE_PIN AC19 [get_ports {mipi_phy_if_1_data_lp_p[0]}]
set_property PACKAGE_PIN AD19 [get_ports {mipi_phy_if_1_data_lp_n[0]}]
set_property PACKAGE_PIN AE20 [get_ports mipi_phy_if_1_clk_lp_p]
set_property PACKAGE_PIN AF20 [get_ports mipi_phy_if_1_clk_lp_n]
set_property IOSTANDARD DIFF_HSTL_I_18 [get_ports {mipi_phy_if_1_data_hs_p[3]}]
set_property IOSTANDARD DIFF_HSTL_I_18 [get_ports {mipi_phy_if_1_data_hs_n[3]}]
set_property IOSTANDARD DIFF_HSTL_I_18 [get_ports {mipi_phy_if_1_data_hs_p[2]}]
set_property IOSTANDARD DIFF_HSTL_I_18 [get_ports {mipi_phy_if_1_data_hs_n[2]}]
set_property IOSTANDARD DIFF_HSTL_I_18 [get_ports {mipi_phy_if_1_data_hs_p[1]}]
set_property IOSTANDARD DIFF_HSTL_I_18 [get_ports {mipi_phy_if_1_data_hs_n[1]}]
set_property IOSTANDARD DIFF_HSTL_I_18 [get_ports {mipi_phy_if_1_data_hs_p[0]}]
set_property IOSTANDARD DIFF_HSTL_I_18 [get_ports {mipi_phy_if_1_data_hs_n[0]}]
set_property IOSTANDARD DIFF_HSTL_I_18 [get_ports mipi_phy_if_1_clk_hs_p]
set_property IOSTANDARD DIFF_HSTL_I_18 [get_ports mipi_phy_if_1_clk_hs_n]
set_property IOSTANDARD LVCMOS18 [get_ports {mipi_phy_if_1_data_lp_p[3]}]
set_property IOSTANDARD LVCMOS18 [get_ports {mipi_phy_if_1_data_lp_n[3]}]
set_property IOSTANDARD LVCMOS18 [get_ports {mipi_phy_if_1_data_lp_p[2]}]
set_property IOSTANDARD LVCMOS18 [get_ports {mipi_phy_if_1_data_lp_n[2]}]
set_property IOSTANDARD LVCMOS18 [get_ports {mipi_phy_if_1_data_lp_n[1]}]
set_property IOSTANDARD LVCMOS18 [get_ports {mipi_phy_if_1_data_lp_p[1]}]
set_property IOSTANDARD LVCMOS18 [get_ports {mipi_phy_if_1_data_lp_p[0]}]
set_property IOSTANDARD LVCMOS18 [get_ports {mipi_phy_if_1_data_lp_n[0]}]
set_property IOSTANDARD LVCMOS18 [get_ports mipi_phy_if_1_clk_lp_n]
set_property IOSTANDARD LVCMOS18 [get_ports mipi_phy_if_1_clk_lp_p]
#dsi-gpio
#0-LED_EN
#1-PWN
set_property PACKAGE_PIN AF24 [get_ports {GPIO_dsi_tri_o[0]}]
set_property PACKAGE_PIN AB26 [get_ports {GPIO_dsi_tri_o[1]}]
set_property IOSTANDARD LVCMOS18 [get_ports {GPIO_dsi_tri_o[*]}]
set_property INTERNAL_VREF 0.6 [get_iobanks 33]
##HDMI
set_property PACKAGE_PIN J24 [get_ports IIC_0_scl_io]
set_property IOSTANDARD LVCMOS33 [get_ports IIC_0_scl_io]
set_property PACKAGE_PIN K23 [get_ports IIC_0_sda_io]
set_property IOSTANDARD LVCMOS33 [get_ports IIC_0_sda_io]
#R/CR colorspace
set_property PACKAGE_PIN D26 [get_ports "vid_data[23]" ] ; # HDMI_R_D35
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[23]" ] ; # HDMI_R_D35
set_property PACKAGE_PIN H24 [get_ports "vid_data[22]" ] ;#"HDMI_R_D34
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[22]" ] ;#"HDMI_R_D34
set_property PACKAGE_PIN B26 [get_ports "vid_data[21]" ] ;#"HDMI_R_D33
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[21]" ] ;#"HDMI_R_D33
set_property PACKAGE_PIN B25 [get_ports "vid_data[20]" ] ;#"HDMI_R_D32
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[20]" ] ;#"HDMI_R_D32
set_property PACKAGE_PIN H26 [get_ports "vid_data[19]" ] ;#"HDMI_R_D31
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[19]" ] ;#"HDMI_R_D31
set_property PACKAGE_PIN D24 [get_ports "vid_data[18]" ] ;#"HDMI_R_D30
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[18]" ] ;#"HDMI_R_D30
set_property PACKAGE_PIN F25 [get_ports "vid_data[17]" ] ;#"HDMI_R_D29
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[17]" ] ;#"HDMI_R_D29
set_property PACKAGE_PIN D23 [get_ports "vid_data[16]" ] ;#"HDMI_R_D28
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[16]" ] ;#"HDMI_R_D28
#GREEN/Y colorspace
set_property PACKAGE_PIN E25 [get_ports "vid_data[7]" ] ;#"HDMI_R_D23
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[7]" ] ;#"HDMI_R_D23
set_property PACKAGE_PIN E23 [get_ports "vid_data[6]" ] ;#"HDMI_R_D22
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[6]" ] ;#"HDMI_R_D22
set_property PACKAGE_PIN E22 [get_ports "vid_data[5]" ] ;#"HDMI_R_D21
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[5]" ] ;#"HDMI_R_D21
set_property PACKAGE_PIN D25 [get_ports "vid_data[4]" ] ;#"HDMI_R_D20
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[4]" ] ;#"HDMI_R_D20
set_property PACKAGE_PIN J26 [get_ports "vid_data[3]" ] ;#"HDMI_R_D19
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[3]" ] ;#"HDMI_R_D19
set_property PACKAGE_PIN G22 [get_ports "vid_data[2]" ] ;#"HDMI_R_D18
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[2]" ] ;#"HDMI_R_D18
set_property PACKAGE_PIN M21 [get_ports "vid_data[1]" ] ;#"HDMI_R_D17
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[1]" ] ;#"HDMI_R_D17
set_property PACKAGE_PIN M20 [get_ports "vid_data[0]" ] ;#"HDMI_R_D16
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[0]" ] ;#"HDMI_R_D16
#B / CB colorspace
set_property PACKAGE_PIN L20 [get_ports "vid_data[15]" ] ;#"HDMI_R_D11
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[15]" ] ;#"HDMI_R_D11
set_property PACKAGE_PIN F23 [get_ports "vid_data[14]" ] ;#"HDMI_R_D10
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[14]" ] ; #"HDMI_R_D10
set_property PACKAGE_PIN G24 [get_ports "vid_data[13]" ] ; #"HDMI_R_D9
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[13]" ] ; #"HDMI_R_D9
set_property PACKAGE_PIN G25 [get_ports "vid_data[12]" ] ;#"HDMI_R_D8
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[12]" ] ;#"HDMI_R_D8
set_property PACKAGE_PIN G26 [get_ports "vid_data[11]" ] ;#"HDMI_R_D7
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[11]" ] ;#"HDMI_R_D7
set_property PACKAGE_PIN K20 [get_ports "vid_data[10]" ] ;#"HDMI_R_D6
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[10]" ] ;#"HDMI_R_D6
set_property PACKAGE_PIN J23 [get_ports "vid_data[9]" ] ;#"HDMI_R_D5
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[9]" ] ;#"HDMI_R_D5
set_property PACKAGE_PIN F24 [get_ports "vid_data[8]" ] ;#"HDMI_R_D4
set_property IOSTANDARD LVCMOS33 [get_ports "vid_data[8]" ] ;#"HDMI_R_D4
set_property PACKAGE_PIN F22 [get_ports "video_clk_out" ] ;
set_property IOSTANDARD LVCMOS33 [get_ports "video_clk_out" ] ;
set_property PACKAGE_PIN L23 [get_ports "vid_active_video" ];
set_property IOSTANDARD LVCMOS33 [get_ports "vid_active_video" ];
set_property PACKAGE_PIN C26 [get_ports "vid_hsync" ] ;
set_property IOSTANDARD LVCMOS33 [get_ports "vid_hsync" ] ;
set_property PACKAGE_PIN E26 [get_ports "vid_vsync" ];
set_property IOSTANDARD LVCMOS33 [get_ports "vid_vsync" ];The flow we are going to be using for the programming of the Arm Cortex-M1 is going to update the bitstream.
As such from the example project downloaded from the Arm DesignStart FPGA website we also need to following files and batch programs.
- make_mmi_file.tcl - this will generate a MMI file showing the instruction memory block RAM locations
- make _prog_files.tcl - this will merge the generated ELF with the bitstream to generate a new bitstream which includes the program we create
- make_prog_files.bat - batch file to create the updated bit using the above tcl file
- make_hex_s7.bat - converts the elf file into the necessary format.
Once the design is implemented we need to make some changes to these files.
The first of these is the make_mmi_file.tcl, here we need to update the device target part to our selected part and also make sure we search for the right BRAM output this can be either DOADO or DOBDO.
This can be determined by opening the elaboration view, once you know which one is used the MMI file can be updated accordingly.
Setting the part name
# Set MMI output file name
set mmi_file "m1.mmi"
set part "xc7s100fgga676-2" # SET PART HEREUpdating the BRAM output port
# puts $itcm_ram_reordered
# For each entry display the location
foreach ram $new_list {
# Get the RAM location
set loc_val [get_property LOC [get_cells $ram]]
regexp -- {(RAMB36_)([0-9XY]+)} $loc_val full ram_name loc_xy
# Get the nets driven by the D0 pins
set data_bus [get_nets -of_objects [get_pins -filter {REF_PIN_NAME =~ DOBDO*} -of [get_cells $ram]]]
# Check number of bits is the same as that expected
if { [llength $data_bus] != $mem_bits } {
puts "Error - Number of data pins read, [llength $data_bus], does not equal expected memory bits, $mem_bits"
return -1
}
# Number of pins connected to the memory sets the memory depth.
set memory_depth [expr {(32768/[llength $data_bus])-1}]
set idx_list [list]
foreach entry $data_bus {
# Filter the data_bus down to just the two index numbers
set index [regexp -inline -- {[0-9]+} [regexp -inline -- {\[.*} [lindex $entry 0]]]
lappend idx_list $index
}
# Sort the index list from highest to lowest
set idx_list [lsort -decreasing -integer $idx_list]
# Assign the highest and lowest bits for the range variables
set index_low [lindex $idx_list end]
set index_high [lindex $idx_list 0]
# Debug
# puts $data_bus
# puts $idx_list
# puts "$index_high downto $index_low pos $loc_val"
array set mem_array [list $index_high $loc_xy]
}; # foreach
}; # forOnce this is completed we can open the implemented design and run the make_mmi_file.tcl to create the new MMI File.
Run the script using the command below in the Vivado TCL window.
source make_mmi_file.tclWe are now ready to develop our software
Vitis Software DevelopmentWe are going to use Vitis to develop the Board Support Package which defines the Vivado design.
Before we can do this however we need to make a slight update to the Software IPI provided by the Arm DesignStart download.
The release only includes support for a standalone in Xilinx BSP version 6.6 and 6.7 and Vitis is currently on 7.1
To get around this issue, under the download Vivado/ARM_SW_Directory/CortexM/BSP Directory copy and paste the standalone_v6_7 direcotry and rename it standalone_v7_1
Updated structure
Under the Standalone_v7_1/data open file standalone.mld and change line 41 to show 7.1 in place of 6.7
We also need to update the cpu_cortexm1.mdd file under the Arm_sw_repository\CortexM\drivers\cpu_cortexm1_v1_1\data directory
With that we are free to export the XSA from Vivado (you do not need to include the bit stream) and open Vitis.
Once you have selected the necessary workspace for your Vitis project you will be presented with the Vitis welcome screen.
Select the Xilinx option from the file menu and select Repositories, we need to add in the SW repository.
Add in the Vivado/Arm_SW_REPOSITORY as a global repository for Vitis.
The next step is to create a platform targeting the XSA just exported from Vivado. As you step through this platform creation dialog you will see the processor is shown as Cortex-M1.
This will create the Vitis platform, we can then build the platform and we are ready to use the output products in Keil.
With the BSP created by Vitis we are now able to create the image processing application in Keil.
Start by creating a new project in Keil, targeting the Arm Cortex-M1 and then select the core and start up from the run time environment set up.
Once the project is created the next step is to configure the project settings, this includes adding in the internal memory base address and size. Along with the external DDR RAM
We also need to configure the tool to run the make_hex_s7.bat file once the compilation has been completed.
On the C/C++ options tab select the include path to include the BSP/include directoy we just created in Vitis.
We are now ready to create the application.
Our software application is going to do the following
- Power on the MIPI Camera
- Detect the MIPI Camera
- Configure the MIPI Camera
- Configure the HDMI Output
- Setup up the Video Timing Controller
- Configure the DeMosiac
- Configure the RGB2YCRCB
- Start the VDMA
As there is no usleep function in the Arm M1 to provide timing between enabling the MIPI Camera and configuring it and timing between I2C commands I use the SysTick Timer.
To keep the complied software low I also used the low level I2C commands for the AXI IIC which enable me to send and receive data without the use of interrupts.
STRELOAD = 0xFFFFFF;
STCTRL = (1<<SBIT_ENABLE) | (1<<SBIT_TICKINT) | (1<<SBIT_CLKSOURCE);
sample = FALSE;
while( sample != TRUE){
xil_printf("waiting.....\n\r");
}
sample = FALSE;
STCTRL = (0<<SBIT_ENABLE) | (1<<SBIT_TICKINT) | (1<<SBIT_CLKSOURCE);
//detect camera
SendBuffer[0]= 0x31;
SendBuffer[1]= 0x00;
tx_cnt = XIic_Send(IIC_CAM_BASE, (u8) IIC_CAM_ADDR, (u8 *)&SendBuffer, (unsigned) 2, XIIC_REPEATED_START);
if(tx_cnt != (2) ){
xil_printf("SW I2C Write Error Delay, %x\n\r",tx_cnt);
}
else{
xil_printf("SW I2C Write Done Delay, %x\n\r",tx_cnt);
}To determine the correct number of bytes are sent the transmitted number of tx bytes is checked against the expected. If the I2C slave NACKs or does not respond the number of transmitted bytes will not be as expected and an error flagged.
To ensure Keil can correctly compile all of the source code defined in the Vitis generated BSP include files. We need to provide the source C as well, this is available with the BSP as well under the lib source. In Keil we need to add in new project items, to make the project more readable I grouped them together under the appropriate element.
When this was completed I was able to compile the application code and generate the necessary output files.
Programming the hardware with the updated application uses the make_prog_files.bat file to merge the output elf with the Vivado Bitstream.
Once the script has been run we can use Vivado hardware manager to download the newly generated bitstream.
When the bitstream downloads you see the FPGA immediately spring to life and start running the application.
The first thing I noticed in a terminal window was the VDMA address being provided and the camera being detected. The ID of the camera is reported before the camera configuration I2C writes are performed.
The completion of the script sees the completion of the HDMI configuration and the status reporting of the VDMA once it is set up and started.
Looking at the ILA inserted within the design we can observe the timing generator outputting its configured waveform.
You can also see the output of the MIPI CSI-2 Receiver SubSystem as video is output into the DeMosiac and VDMA System.
Of course, finally we can also see the output on the TV screen and see the new pictures I have hanging on my office wall
Hopefully this project has demonstrated two elements, the first how to work with Arm Cortex-M1 cores in Xilinx Vivado and Vitis 2019.2 a question which comes up often when I teach my getting started with Arm M1 Course.
It also demonstrates how a small micro-controller can be used to control a high performance image processing system in a Xilinx FPGA.
It also opens up the path to replacing the Arm Cortex-M1 with a Arm-Cortex-M3 and using a TinyML solution.
See previous projects here.
Additional information on Xilinx FPGA / SoC development can be found weekly onMicroZed Chronicles.
Comments