Perfecting PCIe with AUBoard

Introduction

One of the commonly used interfaces to connect systems with a FPGA is PCIe. PCIe enables high bandwidth transfers and is commonly used in our lap and desk top computers. It is also used significantly in a range of embedded computing applications which is where our FPGA comes into play.

if you are not familiar with PCI Express (PCIe) it is a high-speed interface standard used to connect peripheral devices such as graphics cards, SSDs, and accelerators to a computer’s processor and memory system. Unlike older bus architectures that share bandwidth among devices, PCIe uses a point-to-point topology, giving each device a dedicated connection to the CPU through high-speed serial links called lanes. These lanes can be configured in x1, x4, x8, or x16 arrangements, allowing scalable performance based on application needs.

One of the key advantages of PCIe is its combination of flexibility and bandwidth. Each new generation significantly increases data rates, making PCIe ideal for modern high-performance systems, including embedded platforms and FPGAs.

For makers and developers, PCIe enables low-latency, high-throughput communication with custom hardware, which is perfect for applications like AI acceleration, real-time data acquisition, or high-speed control.

At the heart of a PCIe system is the root complex, which acts as the central controller, typically integrated into the CPU or chipset. It manages communication between the host processor and all connected PCIe devices, known as endpoints.

Endpoints are the devices that perform specific functions, such as storage, networking, or acceleration. Each endpoint communicates directly with the root complex over its dedicated connection.

The physical connection between devices consists of one or more lanes. A lane is a pair of differential signal wires, operating at multiple Giga Bits per second rates. The simplest link is one lane which has one link transmitting and one link for receiving.

PCIe supports various lane configurations such as x1, x4, x8, and x16, with higher lane counts delivering greater bandwidth. In more complex setups, a PCIe switch can be used to connect multiple endpoints to a single root complex, allowing for expanded topologies while maintaining high-speed point-to-point communication.

Data communication in PCIe is handled by a layered protocol stack. The transaction layer manages high-level data transfers, generating packets for memory reads, writes, and I/O operations. Below that, the data link layer ensures reliable delivery through error checking and sequencing. At the bottom, the physical layer handles actual electrical signalling between devices.

Each device also contains a configuration space used during system initialization to identify the device and allocate system resources. Part of this includes Base Address Registers (BARs), which map a device’s memory or I/O regions into the system address space so the processor can access them directly.

Why BARs Matter

Base Address Registers (BARs) enable communication and data transfer between the host and card PCIe devices. BARs play a critical role in defining how a peripheral within a PCIe system presents its memory and I/O space, a concept we must understand to effectively work with the Tria Technologies AUBoard platform or other Programmable Logic PCIe applications.

Decoding the BARs: A Simplified Overview

At the simplest level a BAR defines the contiguous memory allocation or I/O space required by a PCIe device. To communicate the location and size of the BARs required by a PCIe device. The PCIe component specification defines a PCIe register configuration and initialisation sequence enabling the address space requirements to be implemented by the Operating System.

During system initialisation the configuration of all PCIe cards is read and address spaces are assigned. Initially the PCIe device BAR registers define the size required for each BAR with the device. This is communicated to the system by the BAR registers being set by the system to all 1’s and the device clearing the LSB to 0’s to represent the address space needed.

With the address space known the system is able to allocate the address space, writing back the assigned base address to the PCIe device BARs. This means in the system address space we now have several regions allocated for PCIe BARs for both end points and the root complex. This allows not only the root complex to communicate with end points but also, end point to communicate with a end point via the root complex.

When discussing PCIe addressing the term Memory Space is used for high bandwidth, large volumes of data transactions. While the term I/O space is used to define slow speed control, communication and reporting transactions.

What this means for our AUBoard application is that memory space is used for DMA transfers, while IO space is used to configures the DMA and IP modules within the programmable logic design. This ultimately resulting AXI Lite transactions within the programmable logic.

Conceptually how transactions work is that an application running on the CPU, will make a read or write request to the BARs allocated address space which will then be routed to the end point provided the access requested resides within one of the BARs by the PCIe Root.

Hardware Design

To get started on our design we are going to create a design in AMD Vivado™ design suite.

Select the AUBoard, if you do not have the AUBoard installed in Vivado copy the steps shown in this project.

Once the project opens create a new IP integrator design.

Into the new diagram, from the boards tab drag across the PCI Express.

For this example I selected the PCI Express Gen 4 x1 which gives one lane.

This will result in the diagram as below.

Run the block automation, this will allow is to define the number of read and write channels, along with the maximum link speed.

Once the block automation is completed. Reconfigure the XDMA module by double clicking on it, on the MISC tab uncheck the Configuration management interface.

The resultant image should be.

The next thing we need to do is connect downstream components, add the following into the design. Use IP Integrators connection automation if desired.

• AXI BRAM Controller and BRAM
• AXI GPIO connected to the RGB led on the board

The final design should be.

The next step is to build the bitstream for the FPGA.

While the bit stream is building we can put the AUBoard in the development PC.

Installing the AUBoard

The next step in this project is to insert the AUBoard in a machine which has a free PCIe slot. luckily my office has several development machines and one of which is an Ubuntu machine we use for our AMD Alveo™ Adaptable Accelerator Cards developments. As such I took out the Alveo card and inserted the AUBoard.

With the board safely installed, we can power back on the board.

Checking the Installation

Once the development machine is powered up we can program the AUboard with the generated bitfile using with the JTAG USB cable. You might need to reboot the PC after the AUBoard is programmed to ensure the PCIe interface is detected.

To check we can now see the target board in a development machine. We can use a terminal to look for the vendor ID on the PCIe network.

lspci -vd 10EE:

The -vd extension defines the vendor ID which for Xilinx starts 10EE: This can be changed if desired in the XDMA configuration if necessary.

We can see the successful detection of the AUBoard and its PCIe core within the FPGA.

If we want more detailed information we can use sudo in front of the command. This will provide more information on the capabilities which were previously hidden.

Another approach to detect the AUBoard is to pipe the lspci command through grep and look for the Xilinx identifier.

We are now ready to begin with the installation of the XDAM driver on the development machine.

XDMA Installation

To get started with this we are going to first clone the XDMA GitHub repository located here.

Once cloned we will then compile the source to create the necessary drivers for our host machine.

While this might seem challenging it is straight forward.

Change directory to the source file location of the XDMA drivers, these are within the repo previously cloned.

From a terminal within the directory run the command below.

make install

Having created and installed the drivers, the next step is to compile the tools we will be using to test the compilation and installation. Change directory into the tools directory and run the command

make

We now have both the drivers and test / demo applications compiled and available for use.

The next stage is to install the driver (it is a kernel object) and then to use the tests to verify we can communicate with the AUBoard and its PCIe end point.

Loading the drivers into the kernel is straight forward. Change directory into the tests directory and run the script load driver with the command.

sudo ./load_driver.sh

You should see a message indicating the kernel module was installed correctly.

With the kernel driver installed we are now able to run the tests provided with the XDMA installation to verify we can communicate with the AUBoard over PCIe.

sudo ./run_tests.sh

The tests will run and a report showing the performance of the PCIe connection will be generated in the terminal.

SW Development

With the drivers installed and the basic tests running successfully, the next step is to check the XDMA is listed under the drivers. We know it is as the tests were just successful but this is how our applications are going to interact with it.

ls /dev

Under this directory we will see several devices which enable user space access to the AUBoard:

• Xdma0_user – This enables access to the AXI Lite network as we saw in the last blog.
• Xdma0_control – This used to access the DMA subsystem PCIe Registers
• Xdma0_events_* - These are used to recognise user interrupts.
• Xdma0_c2h_* - These are used to transfer data from the card to the host
• Xdma0_h2c_* - These are used to transfer data from the hast to the card

Of main interest to us, when we conduct high-bandwidth transfers, are the control, c2h, and h2c channels.

The process of creating a transfer is very straightforward, as demonstrated by the demo applications.

Of course, as you would expect the transfer leverages elements of the user space application which defines the channels to be used and allocates the buffers of data and transfer size.

Once this has been created the driver in the kernel space takes over and creates the necessary descriptors and then starts the transfer.

As the transfer is started the DMA takes over and reads the descriptor to determine which kind of transfer is being requested.

With the descriptor processed the DMA then performs the transfer which moves the data in the correct direction either from or to the card.

After the transfer is completed the kernel driver looks for interrupts and checks the status before returning these to the user side to checking.

This is one of the benefits of using the XDMA driver, the complexity is abstracted away. This enables the application developer to focus on the actually behaviour of the application not worrying about transferring data.

If you want to try using the demo applications to test out these drivers you can using the format below.

dma_from_device -d <device> -f <file> -s <size> -a <address> -c <transfercount>

We can also use the reg_rw program to perform AXI Lite transactions.

Wrap Up

PCIe provides us with high bandwidth transfers between the host and an FPGA.

As this project has shown this can be easily demonstrated using the Tria Technologies AUBoard which contains the AMD Artix UltraScale+ AU15P device which supports up to PCIe Gen4 x4.