Driving Accelerators From Userspace on Ultra96

Introduction

Writing drivers for your accelerators in kernel space can be cumbersome when you want to get things running quickly or simply test your accelerator in Linux environment. Hence, this tutorial shows how to access and program your hardware accelerators in userspace.

Diving in

When you write a software application on any modern operating system, your application runs in virtual address space. Whereby you live in a sandbox unlimited access to memory and behave as if you are the sole user. The operating system hides the existence of other processes and I/O via appropriate system calls.

However, when you make your hardware accelerators on FPGAs and connect them to the CPU via AXI, you assign the accelerator a physical address.

In order to talk to the hardware module in Linux, we need to map our hardware accelerator into the virtual address space of our application.

To do this run the following code with Sudo to open devmem:

#include <sys/mman.h>
#include <sys/ioctl.h>

int fd = open("/dev/mem", O_RDWR | O_SYNC);
if(fd < 1) {
  perror("Cannot open /dev/mem");
  return (void*)-1;
}

The O_SYNC flag is necessary to turn off the caches and writing straight to memory. This ensures that whatever you write in your software, propogates through the caches to your hardware accelerators.

The following snippet maps the physical address (“phy_addr”) of our module to virtual space (“mmaped_mem”) where we can talk to it.

size_t pagesize = sysconf(_SC_PAGE_SIZE);
volatile int* mmaped_mem = (volatile int*)mmap(NULL, pagesize, 
                                               PROT_READ |  PROT_WRITE, 
                                               MAP_SHARED, fd, phy_addr);
if(mmaped_mem == MAP_FAILED)
{
  perror("Couldn't map hw accelerator in userspace");
  return (volatile int*)-1;
}
return mmaped_mem;

Creating contiguous physical buffers

To create a contiguous physical memory buffer where hw modules can write in the memory use udmabuf:

https://github.com/ikwzm/udmabuf

A library in C++ and Python for udmabuf deployment can be found in FOS Github:

https://github.com/khoapham/fos/tree/master/udmalib

Conclusion

With this you can map any hardware IP into your virtual address space of your software stack. The method is compatible with all type of system running linux from FPGAs to other embedded systems like Raspberry Pi.

Note, this solution is not a permanent replacement for kernel space drivers as it requires sudo access and is not safe in terms of security for general purpose use.