Hardware components | ||||||
| × | 1 | ||||
| × | 1 | ||||
| × | 1 | ||||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
In my previous posts, I demonstrated the basic steps for utilizing the Digitizer Zmod in a custom hardware/software design on the Eclypse Z7. In the second post covering the C application for controlling the DMA transfers, I showed the process for setting up the write channel of the DMA and kicking off a stream to memory map (S2MM) transfer to read the ADC's samples into DDR memory.
All was fine until I noticed that the DMA would forever report back as busy after that. I also started to see unpredictable behavior the more times that I ran the C application.
Things got even worse when I added the AWG Zmod and enabled the read channel of the DMA to then write out converted samples to its DAC. With the S2MM channel permanently busy, the MM2S channel to write out to the DAC on the AWG Zmod was also stuck busy. The unpredictable behavior on the S2MM transfers got even worse with intermittent dumping of packets halfway through the transfer, sometimes not accepting any data samples in at all, etc.
So I went back to the ILAs with the AXI protocol checker enabled to track down what was going on... And of course, I found that I had once again fallen victim to the "unspoken" rules of the AXI DMA engine in Vivado/Vitis.
Note: this project is done in version 2022.1, but should be fairly universally applicable to all versions.
"Unspoken" Rules of AXI DMA S2MMNow these rules aren't really "unspoken" per say... They are documented, however they are easy to miss until you're having issues with the DMA engine in Vivado/Vitis not behaving as expected. So I'm trying to capture them here in a format that hopefully mitigates this oversight for some.
Starting with the very basics of the physical interface of the AXI Stream (AXIS) on the S2MM channel of the DMA engine, here is a list of golden rules:
- Every clock with
tvalidandtreadyasserted on the S2MM master AXI Stream interface from the upstream AXIS device triggers a transfer oftdatainto memory. - The transfer is then ended by the clock cycle where
tvalid,tready, andtlastfrom the upstream AXIS device are asserted. - The DMA engine must be configured and ready for a transfer to start (see steps below) before
tvalidfrom the upstream AXIS device is asserted, otherwise the S2MM channel will stall. - Once
tlastis asserted by the upstream AXIS device,tvalidmust also be deasserted at the same timetlastis. Otherwise,tvalidstill being asserted aftertlastgoes low will cause the DMA engine to think a new transaction is starting (that it has not been configured for at that point in time, therefore the S2MM channel will stall). - When
tlastis asserted by the upstream AXIS device, it must be during thetdatabyte index that matches the transfer length that was set in the S2MM's buffer length register (offset 0x58).Tlastbeing asserted when the DMA hasn't read in as many or more bytes as it was expecting to will stall the DMA S2MM channel.
To configure the DMA engine for a S2MM transfer so it will assert tready and initiate the transfer when tvalid is asserted by the upstream AXIS device, it is just a matter of four register writes:
1. Reset the DMA by writing a 1 to bit 2 of the S2MM control register (offset 0x30).
2. Write the destination address of the location in DDR memory the S2MM channel is reading data into to the destination address register (offset 0x48).
3. Start the DMA S2MM channel by writing a 1 to bit 0 of the S2MM control register (offset 0x30).
4. Write the length of the buffer for the S2MM channel by writing the value for the total number of bytes to read into memory on the S2MM channel to the S2MM buffer length register (offset 0x58). This kicks off the S2MM transfer such that the DMA is prepared to receive a data stream from a device in the FPGA logic (which doesn't actually start until it is actually fed data and tvalid on the AXI stream bus is asserted by the device in logic).
This these "golden rules" and process for the S2MM established, I'll explain where I went wrong...
Data Stream Packetizer Logic ModuleAfter busting out the ILAs, I remembered that there is no tlast signal coming from the Zmod Controller IP. While I enabled packet mode on the AXIS Data FIFO the Zmod Controller IP outputs to so that it would automatically start transferring samples out into DMA when the FIFO's almost full condition is met, that didn't generate the tlast signal needed to complete a transfer.
To remedy this, I wrote a simple logic module to count out the number of bytes being set in the buffer length register and assert tlast during that byte. I placed this module after the FIFO so its full/almost full logic is left to control the back pressure of the data stream automatically.
module DataStreamPacketizer(
input clk, // ZmodDcoClkOut
input reset_n, // sZmodDcoPLL_Lock
// AXIS slave interface - DataStream from Zmod Digitizer Controller
input [31:0] s_axis_tdata,
output s_axis_tready,
input s_axis_tvalid,
// AXIS master interface
output [31:0] m_axis_tdata,
input m_axis_tready,
output reg m_axis_tvalid,
output reg m_axis_tlast
);
// 32 is written to the DMA transfer length regs 0x28 & 0x58 -> 2^5 = 32 (count 0 - 31)
reg [4:0] txr_length_cnt;
parameter txr_length = 5'd31;
reg [1:0] last_cnt;
// pass-thru signals
assign m_axis_tdata[31:0] = s_axis_tdata[31:0];
assign s_axis_tready = m_axis_tready;
always @ (posedge clk)
begin
if (reset_n == 1'b0)
begin
txr_length_cnt <= 5'd0;
end
else if (m_axis_tvalid == 1'b1 && m_axis_tready == 1'b1)
begin
txr_length_cnt <= txr_length_cnt + 1;
end
else
begin
txr_length_cnt <= txr_length_cnt;
end
end
always @ (s_axis_tdata)
begin
if (txr_length_cnt == txr_length)
begin
m_axis_tlast <= 1'b1;
end
else
begin
m_axis_tlast <= 1'b0;
end
end
always @ (posedge clk)
begin
if (last_cnt != 2'd0)
begin
m_axis_tvalid <= 1'b0;
last_cnt <= last_cnt + 1;
end
else if (m_axis_tlast == 1'b1)
begin
m_axis_tvalid <= 1'b0;
last_cnt <= 2'd1;
end
else
begin
m_axis_tvalid <= s_axis_tvalid;
last_cnt <= 2'd0;
end
end
endmoduleI also added logic to deassert tvalid at the same time since the AXIS Data FIFO isn't aware of tlast and won't set tvalid low at the appropriate time.
As I mentioned, I added this Verilog module to the block design as a middleman between the AXIS data FIFO and the AXI DMA controller S2MM input.
I learned the hard way that tvalid being asserted on the S2MM interface before it has been configured causes all sorts of weird behavior (well, basically anything but the correct/expected behavior), and I found it to be the worst on the first transfer after boot up. I also found that when the DMA is being reset and initialized, it will assert tready randomly for a few clock cycles.
This made it impossible for my DataStreamPacketizer logic module to account for sporadic cycles of tready from the S2MM interface versus normal operation. Therefore I added an AXI GPIO module with a single output bit such that I could hold all upstream IP in reset until the DMA S2MM had been configured and was ready to receive data.
I placed two AND gates on the reset line of the AXIS data FIFO that would hold the FIFO in reset until the DMA S2MM has completed its reset, the Zmod Digitizer Controller asserts the signal indicating the ADC has been initialized, and the signal from the AXI GPIO is also asserted.
I also routed this AXI GPIO signal to the sEnableAcquisition input of the Zmod Digitizer Controller, which ultimately prevents the tvalid signal from being set until the AXI GPIO signal is set.
I set this AXI GPIO in the C application right after kicking off the S2MM transfer and before attempting to kick off a MM2S transfer:
// Setup & kick off S2MM channel first
Status = XAxiDma_S2MMtransfer(&AxiDma,(UINTPTR) RxBufferPtr, MAX_PKT_LEN);
EnaAcqGPIO(GPIO_DIGITAL_HIGH);After adding the AXI GPIO, control logic to hold AXIS data FIFO in reset, and the data packetization logic to assert tlast, I launched another debug run in Vitis to see the result.
The trusty memory monitors in Vitis showed a perfectly aligned data transferred into the DDR location pointed to by the receive buffer:
And my ILAs showed the same at the PHY level of the AXIS signals:
I've attached the TCL script for my updated block design below for the Eclypse Z7 with the Digitizer Zmod on SYZYGY port A, and AWG Zmod on SYZYGY port B, and my data stream packetizer Verilog is above. Hopefully my ramblings about DMA are helpful!
Comments