FPGA Processor Expansion Module

Introduction

Programmable logic is amazing we are able to interface to high performance peripherals many microprocessors and microcontrollers could only dream of such as high speed ADCs and DACs.

In this project we are going to us a Avnet ZU board and implement a high speed ADC sampling pipeline which can be accessed over SPI by an external SPI master.

Before we go to into this demo I want to explain a couple of important things

• Of course we could use the processor cores within the ZU Board to do this and process the data in SW using Bare Metal, PetaLinux or PYNQ etc.
• However, some solutions do not always enable a developer to be able to select a SoC based device. There may be many reasons why an external processor and FPGA are selected as the approach e.g. processing needs, company heritage etc.
• Therefore this is a demonstration of how Seven Series, UltraScale / UltraScale+ FPGA such as ArtixUS+ or up coming Spartan US+ can be used be used to provide peripheral interfacing.
• The ZUBoard makes a great cost effective prototyping environment even for just logic development.

In this project we are going to be creating a system such that the Zmod Scope from Digilent is interfaced over SYZYGY to the ZU Board. Its output data will then be captured and processed by a programmable logic design. This programmable logic design will store the data samples and make them available to a module which can read them out over SPI connected externally via the MicroE Click connectors.

The code which performs the SPI to AXI is based upon a previous project which worked with a UART. However, this time our design has been it has been expanded to provide support for AXI Burst Transfers.

IP Core Development

To start with we need to write an IP core which will receive data as an SPI slave and convert it to AXI Stream.

This AXI Stream will then be processed by our protocol module to convert to either a AXI Lite or AXI4 transaction.

Read responses will be fed back from the protocol module to the SPI slave over AXI Stream. As such several dummy SPI Slave packets will be needed for the a read to be performed.

This is quite a complex task so I will be sharing two of our Adiuvo IP cores, of which derivations are flying on space missions.

There are two main modules the SPI Slave and the Protocol module. The SPI Slave has an architecture as shown below.

While the Protocol Module has an architecture as below

The protocol module has two different interfaces one AXI4 and the other AXI Lite which one is used depends on the protocol sent over the SPI

The AXIS Write transaction is formatted as

While the AXIS read transaction is formatted as

The AXIS data width is 8 bits, the same as a SPI transaction, this means we can send down a sequence of bytes to trigger single reads or write or burst transactions.

The payload length is equal to the number of words to read or write minus 1 (i.e. the length is zero indexed). For example, to write 256 bytes the payload length needs to be set to 255.

For the single read/write the payload length is always set to 0 (i.e. a single word is read/written).

While the address is a standard 32 bit address, sent as four bytes.

To test this module would work within AMD fabric, I created a block diagram for simulation and connected the IP core to a block RAM via two AXI Bram controllers. One set for AXI Lite Interface the other set for AXI4.

A test bench was then created to simulate SPI transactions to ensure the module was working as expected for single and burst read and writes.

With the IP core working as expected the next step is to upgrade the ZMod Scope controller to work with UltraScale+ Fabric.

Upgrade ZMod Scope Controller

The controller for the ZMod scope can be obtained from the Digilent Vivado IP library here. To be able to use the Zmod scope controller we need to first add the downloaded library to the settings IP Repository.

Once this has been added we can edit the Zmod Scope Controller, in the IP Packager.

This will open a new instance of Vivado, which contains the IP source code. Here we need to change the 7 Series IDDR and ODDR primitives for the UltraScale+ ones which are IDDRE1 and ODDRE1

We need to change the output clock generation

------------------------------------------------------------------------------------------
-- ADC CLKIN
------------------------------------------------------------------------------------------
--InstADC_ClkODDR : ODDR
--   generic map(
--      DDR_CLK_EDGE => "OPPOSITE_EDGE", -- "OPPOSITE_EDGE" or "SAME_EDGE"
--      INIT => '0',   -- Initial value for Q port ('1' or '0')
--      SRTYPE => "ASYNC") -- Reset Type ("ASYNC" or "SYNC")
--   port map (
--      Q => OddrClk,   -- 1-bit DDR output
--      C => ADC_InClk,    -- 1-bit clock input
--      CE => '1',  -- 1-bit clock enable input
--      D1 => '1',  -- 1-bit data input (positive edge)
--      D2 => '0',  -- 1-bit data input (negative edge)
--      R => aiRst,    -- 1-bit reset input
--      S => '0'     -- 1-bit set input
--   );
InstADC_ClkODDR : ODDRE1
generic map (
IS_C_INVERTED => '0',            -- Optional inversion for C
IS_D1_INVERTED => '0',           -- Unsupported, do not use
IS_D2_INVERTED => '0',           -- Unsupported, do not use
SIM_DEVICE => "ULTRASCALE_PLUS", -- Set the device version for simulation functionality (ULTRASCALE,
-- ULTRASCALE_PLUS, ULTRASCALE_PLUS_ES1, ULTRASCALE_PLUS_ES2)
SRVAL => '0'                     -- Initializes the ODDRE1 Flip-Flops to the specified value ('0', '1')
)
port map (
Q => OddrClk,   -- 1-bit output: Data output to IOB
C => ADC_InClk,   -- 1-bit input: High-speed clock input
D1 => '1', -- 1-bit input: Parallel data input 1
D2 => '0', -- 1-bit input: Parallel data input 2
SR => aiRst  -- 1-bit input: Active-High Async Reset
);

We also need to change the input data reception

GenerateIDDR : for i in 0 to (kADC_Width-1) generate      
--      InstIDDR : IDDR 
--      generic map (
--         DDR_CLK_EDGE => "SAME_EDGE", -- "OPPOSITE_EDGE", "SAME_EDGE" 
--                                          -- or "SAME_EDGE_PIPELINED" 
--         INIT_Q1 => '0', -- Initial value of Q1: '0' or '1'
--         INIT_Q2 => '0', -- Initial value of Q2: '0' or '1'
--         SRTYPE => "SYNC") -- Set/Reset type: "SYNC" or "ASYNC" 
--      port map (
--         Q1 => dChannelA(i), -- 1-bit output for positive edge of clock  
--         Q2 => dChannelB(i), -- 1-bit output for negative edge of clock
--         C => DcoBufioClk,   -- 1-bit clock input
--         CE => '1', -- 1-bit clock enable input
--         D => dADC_Data(i),   -- 1-bit DDR data input
--         R => '0',   -- 1-bit reset
--         S => '0'    -- 1-bit set
--         );
         
         
            InstIDDR : IDDRE1
   generic map (
      DDR_CLK_EDGE => "SAME_EDGE", -- IDDRE1 mode (OPPOSITE_EDGE, SAME_EDGE, SAME_EDGE_PIPELINED)
      IS_CB_INVERTED => '1',           -- Optional inversion for CB
      IS_C_INVERTED => '0'             -- Optional inversion for C
   )
   port map (
      Q1 => dChannelA(i), -- 1-bit output: Registered parallel output 1
      Q2 => dChannelB(i), -- 1-bit output: Registered parallel output 2
      C => DcoBufioClk,   -- 1-bit input: High-speed clock
      CB => DcoBufioClk, -- 1-bit input: Inversion of High-speed clock C
      D => dADC_Data(i),   -- 1-bit input: Serial Data Input
      R => '0'    -- 1-bit input: Active-High Async Reset
   );

With that we can repackage the IP Core.

To check the modifications are correct I created as I did for the SPI IP Core a simulation block diagram.

This provides the Zmod scope controller with the necessary clocks, running this should show the ADC reference clocks are generated correctly and the initialisation completes without an issue.

The simulation can be seen below

Now with the two major elements of the FPGA design completed we can create the application.

Overall Design

The complete design requires the following

• Zynq MPSOC Processing block configured for the Zu Board - We use this to provide us the 100 MHz reference clock as there is no PL clock.
• Zmod scope controller - Provides the interface to the Zmod
• SPI Module - Custom SPI to AXI Block
• AXI Bram controller - Configured for AXI4
• AXI Bram controller - Configured for AXI Lite
• Two BRAM - Used for testing the SPI to AXI Interface
• AXI Interconnect - creates the AXI4 and AXI Lite networks
• AXI Stream FIFO - Provides ability to read out the ADC samples

The completed block diagram looks as below - I also added in several ILA to see how the system is working.

The addresses for the memory map are

I also needed to determine the IO allocations for the Zmod controller and the SPI Interface.

The XDC file can be seen below

set_property IOSTANDARD LVCMOS18 [get_ports ZmodDcoClk_0]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[13]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[12]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[11]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[10]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[9]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[8]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[7]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[6]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[5]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[4]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[3]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[2]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[1]}]
set_property IOSTANDARD LVCMOS18 [get_ports {dZmodADC_Data_0[0]}]
set_property IOSTANDARD LVCMOS18 [get_ports iZmodSync_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodADC_CS_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodADC_Sclk_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodCh1CouplingH_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodCh1CouplingL_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodCh1GainH_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodCh1GainL_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodCh2CouplingH_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodCh2CouplingL_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodCh2GainH_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodCh2GainL_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodRelayComH_0]
set_property IOSTANDARD LVCMOS18 [get_ports sZmodRelayComL_0]
set_property IOSTANDARD LVDS [get_ports ZmodAdcClkIn_p_0]
set_property IOSTANDARD LVDS [get_ports ZmodAdcClkIn_n_0]
set_property PACKAGE_PIN L4 [get_ports ZmodDcoClk_0]
set_property PACKAGE_PIN D3 [get_ports {dZmodADC_Data_0[0]}]
set_property PACKAGE_PIN D1 [get_ports {dZmodADC_Data_0[1]}]
set_property PACKAGE_PIN H5 [get_ports {dZmodADC_Data_0[2]}]
set_property PACKAGE_PIN G1 [get_ports {dZmodADC_Data_0[3]}]
set_property PACKAGE_PIN F1 [get_ports {dZmodADC_Data_0[4]}]
set_property PACKAGE_PIN E4 [get_ports {dZmodADC_Data_0[5]}]
set_property PACKAGE_PIN E3 [get_ports {dZmodADC_Data_0[6]}]
set_property PACKAGE_PIN E1 [get_ports {dZmodADC_Data_0[7]}]
set_property PACKAGE_PIN L2 [get_ports {dZmodADC_Data_0[8]}]
set_property PACKAGE_PIN J5 [get_ports {dZmodADC_Data_0[9]}]
set_property PACKAGE_PIN L1 [get_ports {dZmodADC_Data_0[10]}]
set_property PACKAGE_PIN F3 [get_ports {dZmodADC_Data_0[11]}]
set_property PACKAGE_PIN F2 [get_ports {dZmodADC_Data_0[12]}]
set_property PACKAGE_PIN N3 [get_ports {dZmodADC_Data_0[13]}]
set_property PACKAGE_PIN H3 [get_ports iZmodSync_0]
set_property PACKAGE_PIN C3 [get_ports sZmodADC_CS_0]
set_property PACKAGE_PIN G4 [get_ports sZmodADC_Sclk_0]
set_property PACKAGE_PIN H4 [get_ports sZmodADC_SDIO_0]
set_property PACKAGE_PIN H2 [get_ports sZmodCh1CouplingH_0]
set_property PACKAGE_PIN G2 [get_ports sZmodCh1CouplingL_0]
set_property PACKAGE_PIN M2 [get_ports sZmodCh1GainH_0]
set_property PACKAGE_PIN M1 [get_ports sZmodCh1GainL_0]
set_property PACKAGE_PIN N2 [get_ports sZmodCh2CouplingH_0]
set_property PACKAGE_PIN P1 [get_ports sZmodCh2CouplingL_0]
set_property PACKAGE_PIN N5 [get_ports sZmodCh2GainH_0]
set_property PACKAGE_PIN N4 [get_ports sZmodCh2GainL_0]
set_property PACKAGE_PIN M5 [get_ports sZmodRelayComH_0]
set_property PACKAGE_PIN M4 [get_ports sZmodRelayComL_0]
set_property PACKAGE_PIN J3 [get_ports ZmodAdcClkIn_p_0]


set_property IOSTANDARD LVCMOS18 [get_ports i_cs_0]
set_property IOSTANDARD LVCMOS18 [get_ports i_sclk_0]
set_property IOSTANDARD LVCMOS18 [get_ports i_sdi_0]
set_property IOSTANDARD LVCMOS18 [get_ports o_sdo_0]
set_property PACKAGE_PIN G7 [get_ports i_cs_0]
set_property PACKAGE_PIN F6 [get_ports i_sclk_0]
set_property PACKAGE_PIN E5 [get_ports i_sdi_0]
set_property PACKAGE_PIN E6 [get_ports o_sdo_0]

Testing

The testing of this is going to be pretty simple, I will be creating another project which is show cases the application which can be used in this configuration.

To spend in the SPI commands we could use a range of tools from a Raspberry PI4/5 to Analog Discovery, SPI Driver, or Glasgow Interface Explorer.

For this testing I will be using a simple SPI Driver, we can script this simply through Python.

To configure the processor clocks we will need to export the XSA and create a simple Vitis project to configure the processors clocks. We need to do this as there is no PL Oscillator on the ZU Board. Of course this would be different on a custom board and we would not need this step.

The first thing to check is the Zmod Scope controller is correctly configured and data is being read out of the ADC.

We can see this using two of the ILAs inserted into the design the ZMod Scope is up and running as we would expect.

The status signals can be seen below for the ADC

The data on both channels of the ADC can be seen over the ILA

We can use the SPI Command to read and write words into the block memory and read out data from the ADC Interface.

We can see the read and write successfully working below

We can also see the data coming out over SPI as well on the SPI Driver I used to test this functionality

No we are able to get started working with the ZMod Scope controlled from a SPI Master.

Conclusion

This project has shown we can create and prototype an FPGA expansion module which can be connected to any processor with a little thought. We can easily prototype this using a ZU Board and retire the technical risks associated with developing such a solution. This show cases how embedded developers can