Automated system of liquid chromatography and TLC analysis

Intoduction/Background

Being a synthetic organic chemist is not an easy job. Almost every day, I have to stop a reaction, set-up another one and purify some samples. But the situation can be more tedious when you are running out of time. Panic monster appears and you have to do several of those the same day.

Reactions party! As some colleges called it. It’s very nice to see, but for me, it’s only a future hangover: the column purification. In which you have to spend more than 5 hours changing vials, adding solvent and doing thin layer chromatography (TLCs) analysis to check is the compound is already out.

In that case, you only have two options: refuse and continue or try to optimize it. I've been refusing for too long, and my research group doesn't have the money to buy an automatic column machine which cost more than four thousands of dollars. So it's time to be creative, use whatever is available and apply some AI and robotics to automate the repetitive and monotonous task of managing chemical vials. Who doesn't want to have some extra hours to focus on more complex tasks or just to finish on time to go home and rest?

Project development

Modifying a 3D Printer for Vial Management

Just to start with the development of the prototype, we borrowed an old 3D printer with the condition that I get it ready again.

After an intensive cleaning and lubrication, it was time to check the electronics and repair some connections. After doing it, there's even room for the kria. Can you find it?

I also take some discarded material from the laboratory. A broken and unusable chromatographic column, that was already in the glass container trash, a couple of old Pasteur pipettes and some vials.

I started to repurpose the existing 3D printer to serve as the movement mechanism for our vial management system. Firstly, the printer's was adapted to securely hold the column. Don't ask how =)

Installing a Servo to Control a Valve

After this, a small servo motor was installed. Again, don't ask how, that happens for dissassembling the 3D printer... This servo controls the column valve and allows opening and closing after filling each vial with the eluent. It also allows precise filling of the vials, avoiding spillage, drips and compound loss when changing from vial to vial.

WIP: Servo Integration

A VHDL IP has been developed in order to control the servo. The intention was to add it to the same hardware project that contains the DPU. It seems that was able to build Vivado project with the DPU, but it doesn't appear (almost for me) in the block design, so I have to investigate more. The servo can be tested now creating a dedicated Vivado project, adding my ServoController IP and choosing a pin for the output in a constraints file. The Raspberry Pi connector on the board has 5V protected for connecting Vcc. An userspace (via /dev/mem) small application/driver has been developed to control the module.

Setting Up the Logitech Brio 4K Camera

Next step was the installation of a Logitech Brio 4K camera. It was mounted on top of the printer with a rudimentary and cheap system. Whatever is useful, even a broomstick can complete your new project.

Recording Videos in Different Lighting Conditions

To ensure the AI model could accurately recognize vials under various conditions, videos were recorded with the vials in different ambient lighting.

Maybe I need a vacation, but it's seems again a party, more precisely, a disco party. Anyway, this step was important for creating an initial dataset. Positons of vials are fixedin this first run, so they can be annotated easily.

Training a YOLO Model

Using the recorded frames, a YOLO model was trained to recognize the positions of the vials. Pytorch was used for this process due to its flexibility and powerful capabilities in handling computer vision tasks.

Model performed very well in majority of test images. When the vials are concentrated, it has difficulties to find them, but in the future probably the will be positioned using a grid, so that is not problematic.

TODO:Live Recognition with Kria KR260

Unfortunately, I am stuck here. Probably, the selection of YOLOv8 model is not officially supported, and I am finding lot of troubles in the process of quantizing and compiling the model in order to deploy on the Kria. I arrived to quantize a dummy model (with small random data, only to test all the following steps), but compilation is failing due to unsupported operators on the model. I need to study carefully and solve this part.

When I finish, I expect to perform real-time recognition of vial positions during live operation, enabling the system to adjust the movement of the 3D printer accordingly.

TODO:Automating Movement

The final step involved integrating the trained model with the 3D printer's control system. This integration will allow the printer to move automatically to the recognized vial positions and perform tasks such as opening the valve. I intend to connect it through serial port using pyserial or similar, and send gcodes directly.

Future Work

In future iterations of this project, I plan to expand the system's capabilities to include automated pipetting and thin layer chromatography (TLC) analysis. These enhancements will further reduce manual intervention and streamline laboratory workflows.

Directory Files Explanation

1_annotate_from_video.py: Script to create annotations from video.

2_prepare_for_training.py: Script to prepare training and validation sets.

3_train_model.py: Script to train the YOLOv8 model.

4_predict_on_host.py: Script for running predictions on the host machine.

WIP - 5_DUMMY_docker_vitis_ai_quantize.py: Dummy script for Vitis AI model quantization.

WIP - 6_docker_vitis_ai_compile.txt: Instructions for compiling the model with Vitis AI in Docker.

vials_dark.mp4: Video file of vials in dark lighting for annotation.

vials_light.mp4: Video file of vials in light lighting for annotation.

vials_detection.pt: Pre-trained YOLOv8 model file.

test_images.zip: Zip file containing test images for the model.

quantize_result.zip: Zip file containing quantized model results.

ServoController_hw_IP.zip Zip file containing the hardware IP to generate the servo signal.

servoctrl.zip: Zip file containing application to control the servo.