A mobile soft growing sustainable agricultural robot

Overview

We have created a renewable energy powered AI-driven soft growing agricultural soft robot arm which uses air pressure to drive the movement of the robotic arm. It not only has the flexibility of a traditional robotic arm, but also has an extremely soft large-area flexible shell to protect the delicate fruits from harm in harvesting. This robot can not only accept remote control, but also automatically identify fruits and guide the robot to pick them with the support of Vitis AI and DPU, saving a lot of manual labour and protecting the health of farmers. In terms of power use, the robot controller uses a variety of power sources as power-battery-solar to ensure that the robot can operate and recover safely in various unexpected situations.

There are lots of fruit plantations in our area. During the harvest season, a large amount of manpower is needed to pick fruits. Fruit picking is a time-consuming and labour-intensive activity that requires human participation, especially for delicate fruits such as strawberries, that only manual picking can ensure the integrity of the skin. Traditional metal robotic arms cannot guarantee the quality of product because their hard surfaces will damage the skin. Some time ago, we discovered the soft robotic arm, a pneumatically driven robotic arm. However, due to its completely different structure from the traditional mechanical part, it is difficult to drive it directly using traditional methods in MCU. It requires more efficient driving methods and more controllers.

Fortunately, KR260 can provide flexible controller deployment methods. Its PL part can add more industrial controllers in various types, which provides us with a low-cost solution for driving complex flexible robotic arms. In a few months, we successfully made the framework of the flexible robotic arm through 3D printing and used ROS to complete various verifications of the robot, and finally realized the control of the flexible robotic arm on KR260. In the remaining space of PL, we also integrated a DPU to realize more complex automated movement, automated decision-making and other matters. In this way, a Robotics AI-driven soft robotic arm is formed, with the goal of fully automatic fruit picking.

Soft robot arm

The robot consists of two parts: a cylinder made of high-strength plastic which is a soft robot arm and a chassis for fixing the air compressor and one endpoint of arm. The chassis is controlled by a servo to rotate on some axes to control the direction of movement of the robot arm. To ensure softness, no metal skeleton is set in the soft robot arm, and the position of the end point of the robot arm is controlled by the shape of cylinder after pressurization. In our setting, the cylinder is always filled with about 2 bars of air pressure, and the length of the arm is controlled by the reduction motor on the chassis. In this process, a group of servos and a group of motors are used, and a controller is needed to control their running.

In the design process for KR260, three tool chains are involved: vivado, vitis, and vitis-AI. In vivado, we pre-place some of PWM controller (AXI Timer) (we strongly recommend that official support for AXI Timer in PYNQ, this is really a very commonly used IP) and GPIO controllers in the blank block design. A VCU is also integrated to support the VVAS in the future.

After exporting the platform, the detail flow is similar to the Smart Camera — Kria™ KV260 2022.1 documentation (xilinx.github.io). The general steps are to package the DPU's.xo file, then use vitis to integrate the dpu into the design and build a complete bitstream file.

Hardware Design

In our test of a single joint, the default pinout is shown in the figure. Due to the power limitation of the development board, we do not recommend using the power supply on the development board as the main power supply to drive the active parts under any circumstances, which may cause some functions to breakdown.

The power monitoring part of this project mainly consists of the following hardware: SSD1306 OLED display, ESP32 microcontroller, automatic power switching circuit and INA226 current/power monitoring circuit. The SSD1306 OLED display and INA226 communicate with the ESP32 using an IIC interface. The ESP32 is equipped with an IIC interface for data transfer with the OLED display and the INA226 chip. At the same time, ESP32 is also equipped with a UART interface for programme burning and debugging.

To realise the automatic burning function of ESP32, We designed a control circuit. By controlling the level of the Boot pin and the EN pin, it can automatically switch the working state of ESP32. The INA226 chip uses a 50mΩ sampling resistor connected to the low side of the load. This connection allows accurate measurement of the load current and thus calculates the power consumption. The automatic power switching circuit consists of three MOS tubes and two resistors. The automatic power switching circuit is composed of three MOS tubes and two resistors. It can realise the function of giving priority to the external power supply in case of external power supply, and automatically switching to the power supply in case of removing the external power supply.

The schematic and design of PCB is attached in our github page

Fast Deployment

The subsequent process requires many files and the is complicated. We used a makefile and a series of tcl scripts to automate all the workflow required for the process. The automatic process step including using v++ instructions to generate xclbin files, creating.bin format bitstreams used for DFX flow from the project, and automatically generating dtbo device tree files according to the design. Through this process, we can generate bitstreams that can be used for both DFX and PYNQ. For the convenience of debugging, we use pynq-dpu (GitHub - Xilinx/DPU-PYNQ: DPU on PYNQ) to deploy PL bitstream. In addition to the driver code of AXI Timer and DPU, we use multi-threading to ensure the smooth display of the images collected from USB Webcam and the DPU inference results. Here are the steps to run the fast deployment.

1. Before using the script, make sure that the vivado and vitis operating environments exist in PATH. Please run the following command to import the environment

source <path-to-xilinx-toolchain>/vitis/settings64.sh

2. Enter the DPUCZDX8G/prj/Vitis/ directory, modify the dpu configuration you need, and execute the pack_xo.sh file to generate the dpu.xo file

sh pack_xo.sh

Place the generated xo file in the project root directory and name it dpu.xo and sfm.xo respectively

3. Place the xsa platform file generated by vivado in src/base_wrapper.xsa. Make sure the file names are consistent

4. Run the command

make all
make pack

The final file will be placed in export and export/src, just choose the file in here. More details can be found in our github page

Manual Synthesis

First, please complete the first three steps in fast deployment to generate the pre-require files.

1. Create a platform in Vitis (different from Vivado!)

Open Vitis, select your work path on the initial interface, and after confirming, you will see the following screen, select Create Platform Project

Enter the platform name. Please keep it simple and short, because Vitis will automatically add a suffix to your name in the subsequent steps.

in the subsequent steps

On the next screen, select Browse under the Create a new platform from hardware (XSA) tab, and then select the xsa file you generated in Vivado. Later, in the OS below, select Linux and uncheck the Generate boot component option further down.

After configuration, you will see the following screen. Select top-level platform in explorer, then click

on the toolbar to build your system platform.

2. Synthesize bitstream

Select File-> New -> Hw Kernel Project, select the built platform in the dialog box, and then name your bitstream.

Right-click the newly created item in Explorer, select Import Source, select the two xo files exported by dpu, and then select the target path in the same folder as the one in the folder.

。

Select the.prj file without the hw_link suffix, select the icon

in the Hardware Function in the middle window, and import the DPU function. Then use the same process to select the file with the hw_link suffix, and then create a container in the same location, then add the two system functions into the container, build this project, and you can complete the bitstream construction.

At this point, the required.xclbin file is built, and the device tree file construction can be found in the previous link. Vitis will also generate a vivado project. The project files can be found in *.build/link/vivado/vpl. You can use vivado to create bitstreams in bin format and generate other files.

Target Bitstream file

For PYNQ-DPU

• bitstream (.bit)
• Xilinx Binary object (.xclbin)
• design description (.hwh)

For DFX

• bitstream (.bin)
• Xilinx Binary object (.xclbin)
• shell.json
• device tree (.dtbo)