DOCSEARCHbyAlgolia
Submit Docs

Ryzen AI Software Documentation

website·Indexed 19 days ago

https://ryzenai.docs.amd.com/

Tools and runtime libraries for optimizing and deploying AI inference on AMD Ryzen AI powered PCs. Includes documentation for the AMD AI Analyzer, model quantization, and deployment flows using DirectML and OnnxRuntime GenAI (OGA) for LLMs. Covers installation for Windows and Linux, NPU driver setup, and support for the Ryzen AI CVML library, Stable Diffusion, and whisper.cpp.

Tokens
24K
Snippets
92
Records
173
Agent score
99%

What's inside Ryzen AI Software

  1. Overview of OnnxRuntime GenAI (OGA) Flow for Ryzen AI

    The OnnxRuntime GenAI (OGA) API is the lowest-level API for building LLM applications on Ryzen AI PCs. It supports two primary execution modes:

    • Hybrid execution mode: Utilizes both the NPU and iGPU to optimize Time To First Token (TTFT) and Tokens Per Second (TPS) during prefill and decode phases.
    • NPU-only execution mode: Uses the NPU exclusively for both prefill and decode phases.

    Supported Hardware: Strix and Krackan Point processors. (Phoenix (PHX) and Hawk (HPT) are not supported).

    Compatible API Version: OGA version 0.11.2 (associated with Ryzen AI 1.7).

  2. Overview of Ryzen AI Software capabilities

    Ryzen AI Software provides tools and runtime libraries to optimize and deploy AI inference on AMD Ryzen AI powered PCs. It enables models trained in PyTorch or TensorFlow to run on the integrated GPU or the Neural Processing Unit (NPU) built on the AMD XDNA architecture. Deployment is achieved using the ONNX Runtime and the Vitis AI Execution Provider (EP).

  3. Overview of Ryzen AI CVML library

    The Ryzen AI CVML library provides a C++ API that allows developers to integrate powerful AI capabilities into applications without needing to train specific models or manually integrate them into the Ryzen AI framework. It automatically selects the appropriate hardware (GPU or NPU) and framework for inference, though the preferred backend can be set via the API. If the NPU driver is missing, the library automatically falls back to the GPU backend.

  4. Monitor NPU partitions and context bindings

    The examine --report aie-partitions command reports runtime information about NPU partitions and column occupancy. This should be executed while a model is running on the NPU. Adding the --verbose flag provides additional details regarding the binding of columns to NPU HW contexts.
    # Report partition and column occupancy
xrt-smi examine --report aie-partitions

# Report partition occupancy and detailed HW context bindings
xrt-smi examine --report aie-partitions --verbose

  5. Prepare LLMs for Ryzen AI deployment

    Preparing an LLM for deployment on a Ryzen AI PC involves two primary stages:

    1. Quantization: Reducing the memory footprint of the pretrained model to better map to hardware accelerator compute resources.
    2. Postprocessing: Exporting the model to OGA (ONNX Runtime GenAI) format and applying specific optimizations for either NPU-only or Hybrid execution modes.

    This flow supports fine-tuned versions of already supported models (e.g., Llama 2, Llama 3). Models with architectures not supported by the hybrid flow cannot be used.

  6. Understand Ryzen AI model quantization formats

    Model quantization maps high-precision weights and activations to lower precision formats (BF16/INT8) to improve computational and memory efficiency on NPU devices. The Ryzen AI compiler supports the following input formats:

    • CNN Models: Supports INT8 (quantized) and FP32. FP32 models are automatically converted to BF16 during compilation.
    • Transformer Models: Supports FP32, which is automatically converted to BF16 during compilation.

    For CNN models specifically, AMD Quark can be used to convert models to INT8 for maximum inference performance.

  7. Accelerate whisper.cpp using Ryzen AI NPU

    AMD provides NPU acceleration for whisper.cpp via a specialized AMD-maintained fork. On AMD Ryzen AI 300 Series systems, the encoder can be fully offloaded to the NPU to achieve significant speed increases compared to CPU-only execution. Currently, NPU acceleration is supported exclusively on Windows, with Linux support planned for the future.
    Refer to the official AMD fork for setup and build flags: https://github.com/amd/whisper.cpp

  8. Analyze Model Partitioning and NPU Insights

    The AI Analyzer UI provides three primary analysis sections:

    1. PARTITIONING: Shows how operators are assigned to CPU vs NPU. The 'Summary' page includes a 'CPU Because' table explaining why certain operators were not offloaded. The 'Graph' page provides an interactive diagram of the partitioned ONNX model.
    2. NPU INSIGHTS: Details NPU-specific optimizations. It includes an 'Original Graph' (lowered to NPU primitive operators) and an 'Optimized Graph' (showing final transformations like fusion and chaining, as well as Failsafe CPU fallbacks).
    3. PERFORMANCE: Analyzes inference execution. The 'Summary' page shows GOP and MAC efficiency if ai_analyzer_profiling was enabled. The 'Timeline' page provides a layer-by-layer execution breakdown for individual inferences.

  9. Update LLM OGA Flow for Ryzen AI 1.5

    The OGA (Optimized Graph Architecture) flow for LLMs has been updated to version v0.7.0 in Ryzen AI 1.5.

    Key changes include:

    • Folder Consolidation: The hybrid_llm and npu_llm folders are merged into a single LLM folder. This folder now contains model_benchmark.exe, run_model.py, and the required C++ headers and .lib files for both Hybrid and NPU workflows.
    • NPU Model Configuration: The vaip_llm.json file is no longer required and its path has been removed from genai_config.json. Users must re-download NPU models from Hugging Face when using the Ryzen AI 1.5 installer.

  10. Ryzen AI 1.4 New Features and Hardware Support

    Ryzen AI 1.4 introduces support for Ryzen AI 300 series processors, including support for up to 16 hardware contexts. This release provides unified support for LLMs, INT8, and BF16 models in a single package, and publicly releases the LLM Hybrid OGA flow and BF16 CNN/NLP model compilation on Windows.

  11. Prepare system prerequisites for Ryzen AI installation

    Ryzen AI Software requires AMD processors with a Neural Processing Unit (NPU). The following software dependencies must be installed on Windows 11 before proceeding with the Ryzen AI installation:

    • Windows 11: Build >= 22621.3527
    • Visual Studio 2022: If using AMD Quark for custom op flow, ensure 'Desktop Development with C++' is installed.
    • cmake: Version >= 3.26
    • Python distribution: Latest version (Miniforge is preferred).

    Critical Path Configuration: For Miniforge, ensure one of the following paths is added to the System PATH variable in the System Variables section of the Environment Variables window:

    • path\to\miniforge3\condabin
    • path\to\miniforge3\Scripts\
    • path\to\miniforge3\

  12. Set up SD3.5 Canny ControlNet

    To use Canny ControlNet with SD3.5, you must manually copy the SD3.0 Canny ControlNet files into the SD3.5 model layout within your GenAI-SD tree. Once the files are in place, run the example using the SD3.5 model ID.
    python run.py -C canny --model_id amd/stable-diffusion-3.5-medium-amdnpu