1st steps in Computational Photography with Kria KV260

Motivation

Photo or video cameras capture images that are 2-dimensional representations of a 3-dimensional world. In a conventional camera, light rays from the real 3D-scene pass an aperture before they hit the 2D-sensor/ film. At a given shutter speed and ISO-setting, the aperture defines how much light enters the camera, and how large the depth of field of the 2D-photo/frame can finally be.

For certain imaging applications, a 2 dimensional representation is needed that includes a larger depth of field as it can physically be captured by a given combination of aperture and shutter speed. Examples are Macro Photos, Microscopy, and more. For such a situations a solution is needed.

The XILINX KRIA KV260 is a powerful board and it is able to handle multiple intense image streams in parallel. This beginner project elaborates the range of possibilities, and test one of the so-called focus-stacking techniques using XILINX`s KRIA KV260.

Optics

The following optical systems mark opposite sides of the spectrum:

• Pinehole camera (camera obscura): Here the aperture is indefinitely small so that exactly one ray passes the aperture for each part of the object.Therefore everything is in focus, but the system is very light inefficient.
• Lens camera: These consist of a lense and a larger aperture, so that several light rays per area of the object hit the sensor. Here only one plane of the object is in-focus, but the camera is much more light efficient.

In order to create photos that are sharp in every picture plane, it is either possible to close the aperture, or it is possible to computationally combine several photos with varying focus planes into one image (focus stack).

However, with a lens camera a very small aperture leads to light bending and therefore to an overall degradation of sharpness. Therefore the use of the aperture to increase the depth of field is technically limited. The focus stacking on the other hand is limited by the number of frames, which can be taken in a defined time interval before objects in the scene move.

Take a Focus Stack

The following distances define the system, and they depict the available technical possibilities to capture the focal stack:

• The lens is operated at a defined focal length f, and lens aperture f/#. Both parameters define how many light rays enter the camera and hit the sensor.
• The object and the lens distance D defines, how far the object of interest is away from the lens.
• The lens-sensor distance D' defines the distance between the focus plane of the lens, and the sensor plane.

Technically a focus stack is captured either by changing the lens-sensor distance D` (i.e. by rotating the focus ring of the lens), or it is captured by means of an alteration of the lens-object distance D (i.e. by using a focus rail that moves the camera or the lens between shots.

Merge a Focus Stack

The following processing steps merge a focal stack after it is captured:

• Align each image in the stack
• Calculate which parts of the frame are in-focus and assign per-pixel weights that represent these parts.
• Calculate the image average.

Telecentric lenses employ a pinehole at focal length, f, which allow only rays that parallel to the primary ray can pass. As the magnification is constant, a prior alignment is not needed. For other lenses, the following formula is used: Magnification: m = f / (D' - f). 1/D' + 1/D = 1/f.

To identify In-focus pixel information about local sharpness is needed. First the Laplacian operator is used, while the size of the operator indicates sharpness (the higher, the sharper). Gaussian blur is added as the nearby pixels need to have similar weights.

The calculation of the weights depend on the method used for focus measurement. The depth from focus determines the sharpest pixel in a focal stack. Cameras that use the contrast method produce focus stacks that can be used to calculate depth maps, and with this all in-focus photos. For other methods each image and pixel the Laplacian is multiplied with the Gaussian blur. The result is added, and finally divided by the sum of weights.

More Information: In the GitHub page for my project you find a more detailed concept of the mathematical algorithms.

XILINX KRIA KV 260

The XILINX KRIA KV 260 board and accessory pack includes the complete hardware that is needed. In addition you need a host computer in order to recreate this project.

You can provide the inbound photo stream as by the following channels:

• USB Webcam
• Image Sensor (MIPI Interface)
• SD card

To keep things simple and focussed, this project works with example images stored on the SD card. You can download them from WikiPedia as described in the notebook on my GitHub.

Installation of Xilinx Toolchain

This section targets developers that start from scratch on MAC OSX.

• Getting started: After you received your development kit, get your hands around it, by means of a hello world example. Xilinx offers the following starter guide and introduction video https://www.xilinx.com/video/som/out-of-the-box-with-kria-kv260-up-and-running-in-under-an-hour.html. Run the intro from start to end. Remark: The starter guide requires that you have a separate monitor, mouse and keyboard. The following documentation describes how you can configure and use the VLC player instead of a separate monitor in order to present the hello-world-video stream https://www.hackster.io/whitney-knitter/getting-started-with-the-kria-kv260-ai-vision-kit-c6d6af
• Toolchain: In order to develop Xilinx Kria Apps, Xilinx offers several tools. To install and use them you either need a system on Microsoft OS or on Linux. On Apple computers (OSX) you typically run these tools in a Linux or Windows System that runs within a virtualized box. Some virtualization options are Oracle's Virtual Box (https://www.virtualbox.org), or the open source QEMU (https://www.qemu.org/download/). Install your virtualization framework of choice, and create an empty image. Be aware that Xilinx Tools demand a high amount of disk space and compute power (see next section).
• Start the image, and install a clean target OS (for my project, I use Ubuntu 18.04). The following guide leads you through the process to install the complete Xilinx tool-chain on this clean system: https://www.hackster.io/whitney-knitter/installing-vivado-vitis-petalinux-2021-2-on-ubuntu-18-04-0d0fdf (Remark: As by the author, the resource requirements are "I've found you should have at least 300GB of free space available to install all of the Xilinx tools and 32GB - 64GB of RAM with at least 8 CPU cores you can dedicate to them").
• Develop a Kria App: After your system is up and running, you can start to develop your project. The following Guide leads you through the process to develop a Kria App that makes use of the open source GStreamer framework. The GStreamer framework is often used in video projects https://community.element14.com/technologies/fpga-group/b/blog/posts/kv260-vvas-sms-2021-1-part7. I recommend you to recreate the example within your own hardware.

Installation Python Environment

In this project I use a different, less hardware-demanding option: Install a Python environment within the official and new Ubuntu SD Card image for the XILINX KRIA KV260 SOM. (Remark: As by their GitHub, "PYNQ is an open-source project from Xilinx that makes it easy to design embedded systems with Zynq All Programmable Systems on Chips (APSoCs).")

• This GitHub repository describes the installation process: https://github.com/Xilinx/Kria-PYNQ.
• This post documents the installation steps https://discuss.pynq.io/t/pynq-now-available-for-the-kria-kv260-vision-ai-starter-kit/3579.

After installation update your installation using the commands "sudo apt-get update", "sudo apt-get upgrade", and "sudo reboot". After the system is up and running, start the environment using this weblink inside your own network http://kria:9090, and log In to your system.

You can install the extensions of your needs by using the corresponding area in JupyterLab. In particular implement the assignment to Github using the jupyterlab/jupyterlab-github extension. This extension allows you to open GitHub repositories directly from your XILINX KRIA KV260 board.

The JupyterLab installation for the XILINX KRIA KV260 SOM includes several examples that show you the different options to program and work with the board. This is for example code for a webcam that is able to detect faces using openCV http://kria:9090/lab/workspaces/auto-y/tree/kv260/video/opencv_face_detect_webcam.ipynb. Explore all other options.

Focus stacking in Python on Ubuntu

After you have installed and tested the standard Xilinx Kria KV260 and PYNQ you can upload the Jupyter Notebook for focus stacking that I created in my project. You find it in my GitHub https://github.com/produkt-manager/focusstacking.

Use the standard upload features in the Jupyter Lab in order to load it to the board. If you have installed it you can also use the GitHub link from Jupyter lab to my GitHub.

The Jupyter Notebook is self-explanatory. Using example images, it shows you step by step, how to combine these images into one image.

Next Steps

Follow-up projects will extend the proof-of concept for example into these directions:

• Capture stacks by a motorized lense and process them
• Work with lightfields
• Focus stacking and other computational photography in programmable hardware.