Hardware components | ||||||
 |
| × | 1 | |||
 |
| × | 1 | |||
 |
| × | 1 | |||
| × | 1 | ||||
| × | 1 | ||||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
 |
| |||||
Hand tools and fabrication machines | ||||||
 |
| |||||
FPGAs are great for low latency, high throughput imaging applications. This is best achieved with cameras that interface directly to the FPGA (as opposed to USB/network cameras). However, for hobbyists, tinkerers and small start-ups, there are limited number of camera module options available, and not all available options are cheap either (relatively speaking). On top of that, many have their physical interface so usually you're limited to using the camera with the dev-kit it was made for. Without a considerable effort, that is.
However, thanks to the popularity of low-cost single board computers like Raspberry Pi, which come with a 15 pin FFC camera interface (MIPI-CSI), things have been changing for the better. Over time, the RPi camera interface has become quite popular and today many single board computers are available with the same camera connectors (Jetson kits, RPi alternative boards, etc). Even some FPGA boards now come equipped with the same 15 pin connector (Zybo-Z7, Kria, etc). And due to the popularity, many low-cost image sensors are now available from various vendors (RPi foundation, Arducam, VEYE,...and many more).
Unfortunately though, once we start using one of these image sensors with our FPGAs, we encounter a huge issue: Most image sensors' datasheets are not public!
You see, before you can use any modern day image sensor, you need to configure it as per your use-case. While the high-throughput data is transferred over protocol like MIPI-CSI, the configuration usually happens over I2C. However, knowing what commands to send over the I2C interface requires knowing the register map for the sensor, and function of each bit of those registers. But without publicly available datasheets, this is a huge issue. In some cases, if you're lucky enough, you will be able to find a leaked datasheet of your particular image sensor on the internet, but this not always the case. For example, at time of this writing (Sep 2021), the RPi "High Quality" camera does not have a publicly available datasheet.
And sometimes even with a datasheet things are not too rosy. Modern image sensors are complex pieces of silicon and configuring most image sensors requires correctly setting hundreds of registers before they start spewing out the right image data in the correct format. There are settings for clock/PLL configurations, resolution, binning, exposure, gains, white balance, output interface formatting - to name a few. Wouldn't it be good to at-least get a reference or starting point to build upon?
So...what can we do?"Sniffing" or "eavesdropping" is a technique commonly used in software/hardware hacking. The idea is to listen-in on the communications between two devices, without disturbing the original conversation, and keeping a meticulous record for later use. We might not understand what the meaning of each part of the conversation is, but sometimes we can just replay the same sequence, and manage to fool the target device into giving us what we need.
My SetupIn this particular case, I have a Zybo-z7-20 board that I bought previously, along with the PCAM-5C board camera. I was able to get the two working together very quickly using the demo project provided by Digilent, which runs "baremetal". However, for future projects, I want to have more camera options than just the PCAM-5C. Many of the RPi interface compatible sensors come with their own Linux drivers. I want to be able to use the same sensors while running baremetal.
My plan is as following:
- Choose a camera with matching physical/electrial interface
- Get it working with it's native platform (RPi, Jetson, etc) with the settings I need (resolution, framerate, etc)
- Setup a I2C sniffer using Logic Analyzer and log all the commands from the processor going to the camera.
- Clean-up, appropriately format and re-play the commands using my own FPGA board.
For this particular experiment, I am going to use what I already have:- Raspberry Pi v2.1 camera- RaspberryPi 4B as "native" platform- Saleae Logic 16 pro as my logic analyzer- Zybo-Zy-20 as my "target" FPGA board.
Lets's get going...!
Step 1: Verify interface compatibilityN. From the look at the schematic for the board to check pin configuration and voltage levels.
We can see that the connector will provide 3.3V to the connected camera and the I2C is also working at 3.3V.
Next, we look at the camera module. From the schematic for the RPi v2.1 camera module, we can verify that the pin configurations and voltages indeed match.
You might notice the pull-up voltage of 1.8V instead of 3.3V, but that is ok since the component Q1 is doing the voltage level translation. And it's expecting the same 3.3V power input. I also verified that the ENABLE signal will work with 3.3V signal from the Zybo-Z7. All the positions of MIPI signals and ground match, as expected.
Step 2: Getting camera working with native platformFirst, we need to get the camera working with its natively supported platform, a Raspberry Pi 4 in this case. However, for a different camera, it can be any other embedded system (Jetson Nano, etc); the platform doesn't matter too much. The main objective is to be able to command the camera to the correct settings using the available driver, so we can spy on the I2C lines.
I will not go into the details of how to get the v2 camera working with Raspberry Pi, since this is covered in a lot of places on the internet, including the official documentation. In short, I booted a Raspbian OS image on the RPi 4, enabled the camera, and then opened a terminal to issue the following command:
raspivid -t 0 -w 1280 -h 720 -fps 60 | gst-launch-1.0 -v fdsrc | ximagesinkLive video started showing on the attached HDMI monitor immediately, indicating the camera was working as desired. Ctrl+C to exit. Note the use of gstreamer pipeline in the above, which is a prerequisite for above command to work.
Step 3: Sniffing I2C communicationsNow that we have a working system, next step is to sniff the I2C communications between the RPi and the Image sensor. First, I used a soldering iron to attach three thin jumper wires to Ground, SDA, SCL pins on the FFC, and then covered with Kapton tape to act as mechanical relief for the attached wires. I did this because the FFC cables are cheap and easy to replace if you make a mistake, compared to messing up the RPi or Camera board during soldering. Also, this gave me a "spy cable" that I can re-use for other cameras/platforms as well.
After this, I hooked these wires up with my logic analyzer. For this step, I used a USB "logic analyzer" to sniff the communications on the I2C channel. I used a Saleae Logic 16 pro, since that is what I have, but any cheap logic analyzer will work for this (I2C has a fairly slow data rate). Finally, I configured the logic analyzer for I2C decoding and hit the Start button.
Now twe have some captured data. Next, we need to clean up and prepare the data before we can use it with our FPGA board. Shatince the exact procedure I followed is very specific to the setup I have, I will detail the intent of each step for the reader's understanding.video.
Now that we have some captured data, we need to clean up and prepare the data before we can use it with our FPGA board. Since the exact procedure I followed is very specific to the setup I have, I will instead detail the intent of each step for the reader's understanding.
First thing I noticed is that there was an initial burst of commands (left), then another dense burst of commands (middle) and then after some delay there is an I2C command every ~15ms (right).
The commands every ~15ms correspond to time between frames at 60fps. Also, the commands are being sent to the same I2C addresses and are all write commands.
So I concluded, after some experimentation, that these must be commands from ISP to the camera to adjust for changing lighting conditions (Automatic Gain/Exposure control, and possibly Automatic White Balance) after processing each frame. This also meant that, for now, I could trim away any data data sent after first instance of these commands, as sending exposure command once will be enough to get started. Also, notice that these repeated commands are being sent to I2C address 0x10. Thus we can infer that the I2C address of our camera is 0x10.
Next, I looked at the trimmed data and noticed some read commands in the beginning, as well as some write commands to I2C address 0x71.
Since we will not know what to do with the data returned from read commands, there is no point in keeping them and so I removed them.
The commands addressed to 0x71 are most probably for the EEPROM located on the Rpi V2.1 cam which is used by the Raspberry pi firmware to see if a genuine camera is attached, and so we also don't need to keep them. Hence I removed them as well.
For the dumped data, I will just mention a bit of breakdown. Let's take a randomly selected row of dumped data:
write to 0x10 ack data: 0x01 0x62 0x0D 0xE8In this row, 0x10 is the I2C address of the camera, 0x0162 is the register address and 0x0D, 0xE8 are the data bytes. Usually one register is one byte long so the 2nd byte will actually go to the next register address (0x1063) due to I2C address auto-incrementing feature in most sensors.
Anyway, after this, I copied the remaining trimmed data to an excel sheet, removed all text but the I2C data and formatted it (with some manual effort) so that it will be easy to use it as array in C later, inside the Xilinx SDK. Part of this included adding a column with the number of I2C bytes in each row, since different rows in the dumped data have different number of I2C bytes sent.
Step 5: Replaying commands with FPGANow, to test our data with our FPGA board, I needed a Vivado &SDK project. Luckily, I already had a Vivado project downloaded from the excellent post by Adam Taylor, which gave me a working system taking data from PCAM-5C camera in 720p60 resolution and displaying it over HDMI output.
Next, I went to the SDK project for this and added an array with the dumped and formatted I2C data from previous steps, and set it up to be sent to the v2 camera I2C address (0x10) after boot-up. Rest nothing had to be changed, infact I even left the previous PCAM-5C code in the file, since commands to that address are simply ignored when there is no device responding on that address. I have also configured Adam's vivado project to use the CSI input (and not the HDMI IN) as source. The full project with all the modifications is linked at the end of the post.
Once the software modifications were done, all that was left to do was to connect the RPi v2.1 camera to the MIPI connector on the Zybo-Z7, connect a HDMI cable from the HDMI TX port to a monitor, and upload the code.
After a few seconds later,..violaa!!... the live video appears on the screen!
I am willing to call it a success. However, before we get carried away, I'll mention an issue that I immediately observed: the exposure and white balance stay fixed. To be fair, this is expected since we are not updating exposure or white-balance after the initial configuration. In future, this can be done by building some HLS blocks to compute Exposure, White balance, etc and sending back updates to sensor after every frame. (Perhaps an idea for more blog posts!)
Result and ConclusionsI was able to successfuly get a 720p-60fps video out of RPi Cam v2.1 using the Zybo-Z7-20 FPGA, without even looking at the datasheet (which, btw, is available for the IMX219 sensor on RPi v2.1 cam module). This proves that the concept adopted here can work for other available sensors as well (albeit with some caveats), and should give us more MIPI image sensor options for use in FPGA projects.
There are a few caveats that I've learned-about during this process:
- More and more image sensors today do not have an internal ISP (Image Signal Processor) and rely on the ISP inside the chip receiving the video for functions like Auto Exposure/Gain, White balance, etc. This means that if one of these sensors are used with the FPGA, then we will need to handle these functions ourselves in the FPGA. Finding the relevant registers should be possible by looking at the data sent to camera after every frame under controlled change of lighting conditions. On the other hand, some camera modules also come with onboard ISP, in which case this will not be an issue. (look at some sensors from Arducam, VEYE)
- Not all image sensors support formats like 720p or 1080p natively. And so, in some cases, the driver will set the sensor in a mode that is larger in height/width (or both) compared to what is requested by user through command line. In such cases, the ISP (Image Signal Processor) inside the Raspberry Pi/Jetson, etc is expected to crop the image down to the resolution requested by the user. This means that if we're using a FPGA to receive this stream, then we will need to do this cropping inside our FPGA. Xilinx VDMA can be easily used to do this cropping by setting appropriate values for Image start position, active pixels and stride in both horizontal and vertical directions, without any processing overheads.
Hope you enjoyed reading and following along. Next time we might try a different MIPI sensor (RPi HQ camera?) and see how that goes.
If you found this useful (or not), do share your comments/thoughts in the comments section below!
Comments