AMD ECS - Deploying & Profiling AI Models with VitisAI 6.x

AMD ECS also hosted a technical session titled “Quantizing, Deploying, and Profiling AI Models”, focusing on the Vitis AI development toolchain.

Since Vitis AI introduced NPU support in version 5.1, it has continued to gain popularity across AMD's Versal AI Edge platforms. Vitis AI 6.2 is widely used with platforms such as the VEK280, while Vitis AI 6.3 is gaining traction with the next-generation Versal AI Edge Gen 2 platforms, such as the VEK385.

The technical session primarily focused on the Vitis AI 6.2 workflow, providing insights into its key tools and capabilities, including the Quantizer for generating quantized model snapshots, as well as the Profiler and Analyzer for evaluating and optimizing AI model performance.

Session outline:

• AI workflow overview
• Quantization Strategies
• Model Compilation and Optimization
• Deployment Runtimes
• Profiling and performance analysis

In the following section, we summarize some of the key information and highlights from the presentation slides.

Vitis AI Workflow

• Model Quantization
• Model Compilation
• Inference Deployment
• Profiling and Analysis

Vitis AI for Mixed Precision Quantization

Vitis AI also supports Mixed Precision, leveraging INT8 and BF16 for the quantization process. Mixed precision combines BF16 and INT8 operations within a single model to achieve optimal speed-accuracy trade-offs.

While the Vitis AI 6.3 (Gen2) support mixed precision ranges from BF16, FP16 and INT8. Additional document on Vitis AI 6.3 can be found at - https://vitisai.docs.amd.com/projects/advanced-micro-devices-63-gen2/en/latest/docs/model_compilation/mixed_precision_compilation.html

During INT8 quantization, certain operations may experience unacceptable accuracy degradation and therefore need to fall back to CPU execution in FP32 precision. This fallback introduces additional data-transfer overhead between the NPU and CPU, potentially creating performance bottlenecks and increasing overall power consumption.

Mixed-precision compilation enables the AMD Vitis™ AI compiler to execute models containing both quantized (VINT8) and floating-point operations entirely on the NPU. For operations that are excluded from INT8 quantization, the compiler automatically converts FP32 operations to BF16 or FP16 precision. This eliminates the need for CPU fallback, reduces data-transfer overhead between the CPU and NPU, and helps maintain model accuracy.

By keeping all operations on the NPU, mixed-precision execution minimizes data-transfer overhead and reduces latency. Compared to full FP32 execution, it also lowers memory bandwidth requirements and reduces the model's memory footprint, resulting in improved overall power efficiency and performance [Reference - https://vitisai.docs.amd.com].

References on Vitis AI & NPU

• Vitis AI 6.3 documentation - https://vitisai.docs.amd.com/en/6.3/
• INT8 quantization with AMD Quark - model_quantization/quick-start-quark.html#int8-quantization-with-amd-quark
• Tutorial on Mixed precision (Versal AI Edge, VEK280) -vai-tutorial-mixed-precision-main/vai-tutorial-mixed-precision-main#83-run-inference-on-the-target