Deployment and Validation of ACT- on the Ventuno Q

Deployment and Validation of ACTon the Qualcomm NPU of the Arduino VENTUNO Q

PyTorch → ONNX → QNN/HTP → Bimanual Robotic Control

Project summary. This project evaluates whether an ACT robotic policy can be deployed and executed effectively on the Qualcomm NPU/HTP of the Arduino VENTUNO Q. The complete path is PyTorch → ONNX → ONNX Runtime → QNNExecutionProvider → Qualcomm QNN → HTP/NPU → action chunk → safe robotic control.

1. Motivation

Transformer-based robotic policies are commonly trained on GPUs, but real-world robot deployment requires low latency, limited memory usage, temporal stability, operator compatibility, and physical safety.

Research question: Can an ACT policy run effectively and stably on the Qualcomm NPU of the Arduino VENTUNO Q?

2. Objectives and Contributions

• Convert a PyTorch-trained ACT policy into an ONNX graph.
• Load and verify the graph using QNNExecutionProvider.
• Execute the model through Qualcomm QNN/HTP on the NPU.
• Integrate multimodal inputs: images + qpos + text_tokens.
• Transform the neural output into safe joint commands.
• Evaluate latency, jitter, action smoothness, resource usage, and joint-tracking RMSE.

3. Overall System Architecture

The system separates perception and control responsibilities between the CPU and the Qualcomm

Perception → NPU → Action Chunk → Robot

4. Deployment Platform: Arduino VENTUNO Q

The VENTUNO Q is used as the physical Edge AI deployment platform. The CPU handles camera acquisition, qpos reading, preprocessing, control logic, and synchronization. The Qualcomm NPU/HTP executes the ACT tensor graph. Physical control remains responsible for postprocessing, safety limits, interpolation, and command transmission.

Key idea: The NPU does not replace the complete robotic system. It accelerates the neural component while the CPU maintains the perception and control pipeline.

5. What the NPU Actually Executes

The NPU does not execute the PyTorch.pt file, the Python forward() method, or the training code. Instead, it executes the exported ONNX graph containing operators such as Conv, MatMul, LayerNorm, Softmax, Reshape, and Transpose.

PyTorch → ONNX → QNN → HTP/NPU

6. PyTorch-to-ONNX Conversion

• Reconstruct the same ACT architecture used for training.
• Load the learned checkpoint parameters.
• Set the model to inference mode.
• Retain only the inference inputs required for deployment: joint state, images, and task context.
• Export the policy as a fixed-shape ONNX inference graph.

7. ONNX Model Interface

Interpretation: the model receives perception + proprioception + task context and returns 50 future steps for 14 joints.

8. Strict NPU Verification

The deployment was verified with QNNExecutionProvider active and CPU fallback disabled. If an ONNX operator is unsupported by QNN/HTP, execution fails rather than silently moving that operator to the CPU. This makes it possible to verify that the accelerated graph is genuinely executing on the NPU.

9. Multimodal Inputs and Inference

• Vision: 240 × 320 × 3 → resize + RGB + normalization + CHW → [1, 3, 3, 128, 128].
• State: 14 joint values from the left and right arms → [1, 14].
• Task: instruction/context → tokenizer → [1, 32].

The CPU prepares images, qpos, and text_tokens. The ACT graph is then evaluated on the NPU through Qualcomm QNN/HTP. The output is a temporal block of future actions: output_0 ∈ R^(1×50×14).10. Postprocessing, Safety, and Closed-Loop Control

The neural output is never sent directly to the actuators. Each predicted action chunk passes through denormalization, sign mapping and offsets, joint safety limits, interpolation/smoothing, and only then becomes servo commands.

Deployment principle: CPU prepares → NPU infers → controller validates.

The closed-loop control cycle is: capture cameras → read qpos → preprocess → NPU inference → action chunk → execute motion → repeat.

11. Experimental Design

Two experimental runs (Test 0 and Test 1) were performed using the same ACT model and the same VENTUNO Q platform, with approximately 2, 000 control steps per run.

• Per-stage latency
• End-to-end latency
• Control-loop jitter
• Action smoothness
• CPU / RSS / temperature
• Per-joint tracking RMSE

12. Key Results

13. Interpretation

• Both tests exhibit globally consistent runtime behavior.
• The NPU completes ACT inference in only a small fraction of the complete cycle.
• Observed jitter is associated mainly with the full system pipeline rather than the NPU alone.
• The policy shows long intervals of smooth motion with rapid corrections in specific states.
• Tracking error is not uniform: the final right-arm joint remains the largest source of error.

Main conclusion: The NPU is fast enough for this ACT configuration. The next optimization target should be the complete control pipeline and, especially, tracking of the final right-arm joint.

14. Limitations

• Only two experimental runs were analyzed.
• More repetitions and statistical analysis are required.
• The tokenizer used during training should be validated for deployment consistency.
• Tensor-copy time should be separated from pure NPU execution time.
• Different HTP performance modes should be compared.
• FP32 / FP16 / INT8 variants should be compared when supported.The final right-arm joint requires dedicated analysis.

15. Future Work

• Run more repetitions and report confidence intervals.
• Evaluate multiple manipulation tasks.
• Compare different action-chunk sizes and control frequencies.
• Optimize the camera pipeline, qpos acquisition, servo communication, and synchronization.
• Improve calibration and tracking of the final right-arm joint.

16. Reproducibility Workflow

• Verify that the ONNX model loads through QNN/HTP.
• Confirm that QNNExecutionProvider is active.
• Keep CPU fallback disabled during verification.
• Run full physical validation with cameras, joint states, NPU inference, and actuator control.

Separating the NPU load test from full physical validation makes it easier to distinguish runtime, sensor, communication, and actuator problems.

17. Final Takeaway

PyTorch → ONNX → QNN → HTP/NPU → Action Chunk → Robot

Edge AI for Transformer-Based Robotic Policies

Vision + state + task → NPU inference → physical motion

Figure 1: NPU inference accounts for only a fraction of the total control-cycle time. Most of the cost comes from the complete pipeline.

Figure 2: Both tests show a stable timing band with occasional drops associated with the overall system pipeline.

Figure 3: Test 1 shows a slight visual improvement in temporal stability, although relevant excursions remain.

Figure 4: The policy maintains long periods of low variation and performs rapid corrections in specific states.

Figure 5: RSS increases slowly, temperature remains around 40 ◦C, and CPU load stays high but reproducible.

Figure 5: Most joints remain between approximately 0.5◦ and 2.5◦. The final right-arm joint remains the main tracking issue.