MIPI Imaging with the Spartan Edge Accelerator Board

Introduction

In several of the project we have demonstrated image processing with MIPI using Zynq based devices. We have even demonstrated designs which have used simpler cameras in low cost FPGAs.

However all of these projects had one thing in common they all had a processor which would perform the high level configuration of the system and the imager. This works well when devices have significant logic resources however, if the device is tight on resources and the inclusion of a processor just to configure the processor over I2C interface is not an efficient utilization of the resources available.

In this project we are going to demonstrate how we are able to develop a MIPI camera system using a Spartan XC7S15 device which provides just 16000 Flip Flops, 8000 LUTs and 10 BRAM.

This project will leave significant logic resource with which we could implement image processing algorithms as desired.

Camera Control

The key to this project is the creation of a FPGA module which is capable of configuring the camera. Typically MIPI cameras use I2C interfaces for configuration, this module is going to be simple in that it will download all of the words required to configure the camera. However, it will not retry if the camera refuses to acknowledge the I2C command, the module will also not be capable of performing reads over the I2C.

Of course the module could be updated to implement these features, pretty simply.

I2C write transactions consist of the following sequences

• The I2C Slave address is transmitted with a write indicated
• The Upper 8 bits of the register address are transmitted
• The lower 8 bits of the register address are transmitted
• The data byte is transmitted

Between each byte of data is a acknowledge bit, this is left high is acknowledged and pulled low if unacknowledged. As I2C drivers only pull the line low leaving the line pulled high by going tristate, this allows the receiver to issue a nack if there is a problem with the data by pulling it low, without causing contention on the bus. The master also terminates a write sequence by the use of a nack to indicate the last byte.

I2C also has start and stop conditions which indicate to the slave if the bus is becoming active or becoming free. The relationship between the SDA and SCL lines indicated the start or stop condition.

Our RTL module is going to do the following approach it will create a shifter register which is 38 bits long, in this shift register will be start condition for the SDA, the four bytes of data (addr and data), 4 ACK/NACK bits and the stop condition.

Into this shift register will be loaded the address and data to be output from a look up table. At the end of each transmission a counter will be incremented, and the next address and data to be output will be loaded into the register from the look up table.

The state machine which controls this can be seen below.

As the clock needs to stable with the data the SCL will be design to provide the SCL pulse at a safe point in the waveform.

The code for the module can be seen

library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
entity i2c_send is port(
rst : in std_logic;
clk : in std_logic;  --100 kHz
scl : out std_logic;
cnt : out std_logic_vector(7 downto 0);
sda : inout std_logic
);
end entity;
architecture rtl of i2c_send is
type fsm is (idle, start, check, delay);
constant i2_addr : std_logic_vector(6 downto 0) := "0110110";
constant r_w : std_logic := '0';
constant ack : std_logic := '1';
constant nack : std_logic := '0';
constant start_bit : std_logic := '0';
constant stop : std_logic := '0';
signal sda_reg : std_logic_vector(37 downto 0); -- 32 data bits and 4 ack / nack plus start and stop
signal sda_cnt : std_logic_vector(37 downto 0); -- timer
signal load_shr : std_logic;
signal counter : integer range 0 to 127;
signal current_state : fsm;
signal initialised : std_logic :='0';
type data_array is array (0 to 87) of std_logic_vector(7 downto 0);
constant addr_h : data_array := (x"01",x"01",x"30",x"30",x"30",x"30",x"30",x"30",x"30",x"30",x"30",x"30",x"30",x"30",x"31",x"36",x"36",x"36",x"36",x"36",x"36",x"36",
x"36",x"36",x"36",x"37",x"37",x"37",x"37",x"37",x"37",x"37",x"37",x"37",x"37",x"38",x"38",x"38",x"38",x"38",x"38",x"38",x"38",x"38",x"38",x"38",x"38",x"38",x"38",
x"38",x"38",x"38",x"38",x"38",x"38",x"38",x"38",x"3a",x"3a",x"3a",x"3a",x"3a",x"3a",x"3a",x"3a",x"3a",x"3a",x"3a",x"3a",x"3a",x"3a",x"3b",x"3c",x"3f",
x"3f",x"3f",x"40",x"40",x"40",x"40",x"40",x"48",x"50",x"50",x"50",x"50",x"5a",x"01");
constant addr_l : data_array := (x"00",x"03",x"00",x"01",x"02",x"16",x"17",x"18",x"1c",x"1d",x"34",x"35",x"36",x"3c",x"06",x"00",x"12",x"18",x"30",x"32",x"33",x"34",
x"36",x"20",x"21",x"03",x"04",x"05",x"08",x"09",x"0b",x"0c",x"15",x"17",x"31",x"01",x"02",x"03",x"04",x"05",x"06",x"07",x"08",x"09",x"0a",x"0b",x"0c",x"0d",x"0e",
x"0f",x"11",x"13",x"14",x"15",x"21",x"20",x"27",x"08",x"09",x"0a",x"0b",x"0d",x"0e",x"0f",x"10",x"1b",x"1e",x"11",x"1f",x"18",x"19",x"07",x"01",x"05",
x"06",x"01",x"01",x"04",x"00",x"50",x"51",x"37",x"00",x"01",x"02",x"03",x"00",x"00");
constant data  : data_array :=  (x"00",x"01",x"00",x"00",x"00",x"08",x"e0",x"44",x"f8",x"f0",x"0a",x"21",x"a8",x"11",x"f5",x"37",x"59",x"00",x"2e",x"e2",x"23",x"44",
x"06",x"64",x"e0",x"5a",x"a0",x"1a",x"64",x"52",x"60",x"0f",x"78",x"01",x"02",x"00",x"00",x"fa",x"0a",x"3f",x"06",x"a9",x"05",x"00",x"02",x"d0",x"0a",x"50",x"02",
x"ee",x"10",x"04",x"31",x"31",x"07",x"41",x"ec",x"01",x"27",x"00",x"f6",x"04",x"03",x"58",x"50",x"58",x"50",x"60",x"28",x"00",x"f8",x"0c",x"80",x"02",
x"10",x"0a",x"02",x"02",x"09",x"6e",x"8f",x"24",x"06",x"01",x"41",x"08",x"08",x"01");
begin
process(clk, rst)
begin
if rst = '0' then
initialised <= '0';
counter <= 0;
current_state <= idle;
elsif rising_edge(clk) then
load_shr <= '0';
case current_state is
when idle =>
if initialised = '0' then
counter <= 0;
load_shr <= '1';
current_state <= start;
end if;
when start =>
if sda_cnt = std_logic_vector(to_unsigned(0,38)) then -- data has run
counter <= counter + 1;
current_state <= check;
end if;
when check =>
if counter = 88 then -- all data written
initialised <= '1';
current_state <= idle;
else
load_shr <= '1';
current_state <= delay;
end if;
when delay => --delay state
current_state <= start;
end case;
end if;
end process;
cnt <= std_logic_vector(to_unsigned(counter,8));
process(clk)
begin
if rising_edge(clk) then
if load_shr = '1' then
sda_reg <= start_bit & i2_addr & r_w & ack & addr_h(counter) & ack & addr_l(counter) & ack & data(counter) & nack & stop;
sda_cnt <= (others => '1');
else
sda_reg <= sda_reg(sda_reg'high-1 downto sda_reg'low) & '1';
sda_cnt <= sda_cnt(sda_cnt'high-1 downto sda_cnt'low) & '0';
end if;
end if;
end process;
sda <= '0' when sda_reg(sda_reg'high) = '0' else '1'; -- only ever pull down bus
scl <= '0' when (clk = '1' and (sda_cnt(0) ='0' and sda_cnt(sda_cnt'high) ='1')) else '1';
end architecture;

The test bench for this is

library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;
entity i2c_send_tb is
end;
architecture bench of i2c_send_tb is
component i2c_send
port (
clk : in std_logic;
scl : out std_logic;
sda : inout std_logic
);
end component;
-- Clock period
constant clk_period : time := 100 ns;
-- Generics
-- Ports
signal clk : std_logic;
signal scl : std_logic;
signal sda : std_logic;
begin
i2c_send_inst : i2c_send
port map (
clk => clk,
scl => scl,
sda => sda
);
scl <= 'H';
sda <= 'H';
uut : process
begin
wait for 10 ms;
report "simualtion complete" severity failure;
end process;
clk_process : process
begin
clk <= '1';
wait for clk_period/2;
clk <= '0';
wait for clk_period/2;
end process clk_process;
end;

In this case we have to send 88 configuration words to configure the Raspberry Pi 1.3 Camera.

The details of one transaction

The I2C address of the camera (7 bit address) is 0x36, the module does not do any clocking conversion, instead it takes a 100 KHz clock. This can be generated by a simple counter in the top level of the design.

We need to be careful to ensure the highest level of the FPGA design when implemented in Vivado contains the tri state SDA and SCL lines.

Vivado Design

The Vivado design is going to include the following IP blocks in addition to the element just generated these IP blocks are provided by the Trenz, Digilent and SEEED these blocks are

• CSI2-DPHY from Trenz
• CSI2-To-AXIS from Trenz
• CSI-DVP From SEEED
• Bayer2RGB From SEEED
• RGB2DVI From Digilent

None of these IP cores require a configuration over AXI Lite which makes them ideal. The CSI2-DPHY is ideal for use on this board due to how the DPHY interface is layed out, as it is not compatible with the Xilinx CSI2 Subsystem.

The completed block diagram looks like the following

Clocking wise the initial 100 MHz clock is multiplied to 200 MHz to provide the reference for the MIPI interface.

The AXI Stream clock is provided by the MIPI CSI2 DPHY module, while the Pixel and serial clocks are provided at 100 MHz and 500 MHz.

The system is designed such that when the FPGA is configured the I2C module will configure the Raspberry PI three camera and the image processing chain will start processing the image.

The implementation resource in the FPGA are

As you can see there are plenty of resource remaining for implementing image processing algorithms.

Performance

We can program the FPGA on the Spartan Edge Accelerator Board using either the ESP32 or JTAG. For this application I used the JTAG port

Setting up the camera to test I used a stand to hold the Raspberry PI camera, and a battery pack to power the Accelerator board.

To measure the power I used a inline USB power monitor the power of the board.

We can capture the image on a HDMI monitor

Connecting the HDMI output to a HDMI to USB capture device enables a cleaner screen shot and simple video to be created of the board running on my office window looking outside.

Wrap Up

his project shows how we can create complex imaging solutions in low cost and small devices. While still providing room for additional image processing applications to be implemented in the remaining logic resources.

I will upload the project to my hackster github https://github.com/ATaylorCEngFIET/Hackster