DOCSEARCHbyAlgolia
Submit Docs

ONNX Runtime GenAI

repository·main·Indexed 22 days ago

https://github.com/microsoft/onnxruntime-genai

A high-performance API for running generative AI models (LLMs) on-device. It manages the complete generative loop, including inference, sampling, and KV cache management, across various hardware accelerators. The library provides C/C++, C#, and Python APIs with support for multi-turn conversations, multi-modal models, tool calling via constrained decoding (JSON schema or Lark grammar), and streaming ASR using models like Nemotron Speech Streaming.

Tokens
32.1K
Snippets
82
Records
128
Agent score
78%

What's inside ONNX Runtime GenAI

  1. Overview of ONNX Runtime GenAI capabilities

    main

    ONNX Runtime GenAI provides a high-performance API for running Large Language Models (LLMs) on-device. It manages the entire generative AI loop, including:

    • Pre and post processing
    • Inference via ONNX Runtime
    • Logits processing, search, and sampling
    • KV cache management
    • Grammar specification for tool calling

    Supported Architectures

    Currently supports: AMD OLMo, ChatGLM, DeepSeek, ERNIE 4.5, Fara, Gemma, gpt-oss, Granite, HunYuan Dense V1, InternLM2, Llama, Mistral, Nemotron, Phi (language + vision), Qwen (language + vision), SmolLM3, and Whisper.

    Supported Hardware and OS

    • OS: Linux, Windows, Mac, Android
    • Architecture: x86, x64, arm64
    • Acceleration: CPU, CUDA, DirectML, NvTensorRtRtx (TRT-RTX), OpenVINO, QNN, WebGPU

  2. Run models on Qualcomm Snapdragon NPUs via QNN EP

    main

    ONNX Runtime GenAI supports hardware-accelerated LLM inference on Snapdragon-based devices (Windows ARM64 and Linux ARM64) using the Qualcomm QNN Execution Provider (EP).

    For optimal performance, it is recommended to use the newer QAIRT-targeting pipelines from olive-recipes which optimize models for the Genie runtime. Standard QNN-targeting recipes also work but do not utilize this specific acceleration pathway.

  3. Run LLMs on device using .NET/C#

    main

    ONNX Runtime GenAI provides a high-performance API for running Large Language Models (LLMs) such as Llama, Phi (language and multi-modal), DeepSeek, Gemma, and Mistral on-device using .NET/C#.

    It manages the complete generative AI loop, including:

    • Pre and post-processing for language, vision, and audio.
    • Inference via ONNX Runtime.
    • Logits processing, search, and sampling (greedy, beam search, and random sampling).
    • KV cache management for optimized performance.
    • Multi-target execution supporting CPU and GPU (with NPU support coming).

  4. Understand the Model Builder's implementation abstractions

    main

    The Model Builder relies on several key abstractions to manage complexity:

    • Model Base Class: The central hub that holds model information, auto-determines optimizations (like replacing MultiHeadAttention with GroupQueryAttention), manages attributes, and contains the functions to generate the final ONNX model and associated GenAI config/tokenizer files.
    • Architecture Classes: Subclasses of Model that store architecture-specific information. They use inheritance to reuse code and can override Make functions for specialized architectures.
    • Attrs Dictionaries: Dictionaries defined in the Model class to store operator-specific and scenario-specific variables (e.g., self.layernorm_attrs). This ensures attributes are globally accessible across different layers of the graph, which is critical when components are added or removed during model construction.
    • Make Functions: Functions used to construct specific operators or subgraphs. They typically take minimal required parameters and use **kwargs to handle scenario-specific options, keeping signatures clean and flexible.

  5. How the Model Builder is designed

    main

    The ONNX Runtime GenAI Model Builder is designed to be a lightweight, standalone tool for converting and optimizing models into quantized ONNX formats. It follows four core principles:

    • Simplicity: Uses minimal command-line arguments. Scenario-specific settings are passed via the --extra_options argument as key-value pairs. It has no dependency on the onnxruntime-genai package.
    • Efficiency: Aims to accelerate the traditional pipeline (conversion $\rightarrow$ optimization $\rightarrow$ quantization) to produce optimized models within minutes.
    • Modularity: Uses class inheritance for model architectures. Optimizations and quantizations are defined in base classes and inherited by specific model architectures (e.g., GemmaModel, MistralModel, PhiModel).
    • Compatibility: Produces models that work directly in ONNX Runtime GenAI and other ONNX-based solutions like Hugging Face's Optimum without extra modification.

  6. Perform incremental core and SDK builds

    main

    To iterate quickly on a specific language binding without rebuilding the native core, follow this two-step process:

    1. Build and install the core to a specific directory using --install_dir. Set ENABLE_PYTHON=OFF and ENABLE_TESTS=OFF via --cmake_extra_defines to minimize build time.
    2. Build the SDK (Python, Java, or C#) using the --sdk flag, pointing to the core installation with --prebuilt_genai_home and the ONNX Runtime with --ort_home.

    Core installation layout:

    • <prefix>/include/: Public C/C++ headers
    • <prefix>/lib/: Native GenAI libraries
    • <prefix>/lib/cmake/onnxruntime-genai/: CMake package configuration

    Native Consumer Integration: If consuming the core in a C++ project, use:

    find_package(onnxruntime-genai CONFIG REQUIRED)
target_link_libraries(my_app PRIVATE onnxruntime-genai::onnxruntime-genai)

    Configure with -Donnxruntime-genai_DIR=<prefix>/lib/cmake/onnxruntime-genai.

    # Step 1: Build and install core
python build.py --config RelWithDebInfo --parallel --skip_tests --skip_examples --skip_wheel --ort_home /path/to/ort --install_dir /path/to/core-install --cmake_extra_defines ENABLE_PYTHON=OFF ENABLE_TESTS=OFF

# Step 2: Build a single SDK (e.g., python)
python build.py --sdk python --prebuilt_genai_home /path/to/core-install --ort_home /path/to/ort --config RelWithDebInfo

  7. Understand ONNX Runtime GenAI model packages

    main

    An ONNX Runtime model package is a directory that bundles multiple build variants of the same model. Each variant targets different hardware or execution-provider (EP) configurations. This allows the library to automatically select the best-matching variant for the local hardware at load time.

    Key features include:

    • Automatic Selection: If a package contains multiple variants for the same EP, ONNX Runtime uses compatibility_string to pick the highest-scoring match.
    • Shared Assets: Common files (like tokenizers) can be stored once in a shared_assets/ directory and referenced by multiple variants to save space.
    • Package vs. Flat Directory: A directory is identified as a package if it contains a top-level manifest.json and no top-level genai_config.json.

  8. Supported constrained decoding modes

    main

    ONNX Runtime GenAI integrates LLGuidance to support constrained decoding, which ensures model outputs follow specific formats (e.g., for structured tool calling). Currently, three modes are supported:

    1. Lark Grammar (Recommended): Allows for a mix of regular text output and structured function/tool output in JSON format.
    2. JSON Schema: Forces the output to strictly follow a provided JSON schema corresponding to one of the available functions/tools.
    3. Regex: Constrains the output to match a specific regular expression.

  9. Resolve native runtimes for Mac Catalyst .NET targets

    main
    When using the ONNX Runtime GenAI NuGet package for Mac Catalyst .NET targets, the package includes blank files under build/ and buildTransitive/ for the target framework folder. This is intentional. The Mac Catalyst platform is designed to directly resolve the xcframework from the runtimes/native/ios folder, following the standard .NET RuntimeIdentifierGraph resolution logic.

  10. Quantize MoE expert weights (QMoE)

    main

    MoE expert weights exported as com.microsoft::QMoE are quantized via a separate path.

    • If qmoe_block_size <= 0, the builder uses per-channel quantization (one scale per output channel).
    • For CUDA quant_type="int", the runtime uses unsigned storage with an implicit zero-point offset:
      • 4-bit: Numeric range [-8, 7], scale max(abs(w)) / 8, stored value q + 8.
      • 8-bit: Numeric range [-128, 127], scale max(abs(w)) / 128, stored value q + 128.

    This unsigned-offset contract ensures compatibility with ORT CUDA QMoE prepacking for CUTLASS MoE GEMM.

  11. How OgaShutdown and re-initialization work

    main

    OgaShutdown returns the genai library to a just-loaded state by tearing down all ONNX Runtime-derived global state. This includes the OrtEnv, device interfaces, genai add-on libraries, and trivial-session allocators.

    A subsequent genai call will transparently re-initialize the environment with a fresh OrtEnv. This is particularly useful for host applications that recreate their wrappers in-process (e.g., a Manager class) and want fresh logging and environment configuration without restarting the entire process.

    All environment-scoped state is managed by a single owner, OrtGlobals, which is destroyed during shutdown and rebuilt on demand during the next use.

  12. Lifetime contract for OgaShutdown

    main

    When using OgaShutdown, you must ensure that no objects holding device memory outlive the shutdown call.

    The following objects must be destroyed before calling OgaShutdown:

    • Model
    • Generator
    • Tokenizer
    • OgaTensor
    • Engine
    • Request

    Consequences of violation: Failure to destroy these objects before shutdown results in undefined behavior, typically manifesting as a crash when the buffer attempts to free itself through a now-invalid allocator. In debug builds, genai will report any leaked objects at shutdown.