DOCSEARCHbyAlgolia
Submit Docs

IREE (Intermediate Representation Execution Environment)

repository·main·Indexed 26 days ago

https://github.com/iree-org/iree

An MLIR-based compiler and runtime that transforms Machine Learning models into a unified IR optimized for deployment across hardware ranging from datacenters to edge devices. The project includes developer tools for Bazel and CMake builds, custom LLVM/MLIR stack management via byo_llvm.sh, and regression testing utilities like bisect_packages.py.

Tokens
182.2K
Snippets
375
Records
879
Agent score
87%

What's inside IREE

  1. Overview of IREE TFLite C API Compatibility Shim

    main

    The IREE TFLite C API Compatibility Shim provides a (mostly) TFLite-compatible API. It allows developers to load compiled IREE modules, manage tensors, and invoke functions using conventions similar to the TFLite C API.

    Note: This is currently EXPERIMENTAL. It is intended as a bridge for applications currently using the TFLite API to onboard with IREE quickly. For full feature access, users are encouraged to eventually migrate to native IREE APIs.

  3. Overview of IREE Tokenizer

    main

    IREE Tokenizer is a high-performance, streaming tokenizer library written in C, designed for ML inference and training. It supports HuggingFace tokenizer.json and OpenAI tiktoken (.tiktoken) formats, providing bit-exact compatibility with their respective reference implementations. It supports BPE, WordPiece, and Unigram models.

    Key Features

    • Streaming-first: Uses a pull-based architecture (initialize() -> feed(text) -> finalize()) that handles arbitrarily long input with fixed memory.
    • Bounded memory: Uses a fixed-size state (~1-2KB) and a caller-provided transform buffer, making it suitable for embedded systems and high-throughput servers.
    • Offset tracking: Maps every token back to its exact byte range in the original input, even after normalization (NFC, lowercase, etc.), which is critical for training data provenance and RAG.
    • Thread-safe & Allocation-free: The tokenizer object is immutable after construction, and the hot path performs no hidden allocations as all buffers are provided by the caller.

  4. Overview of IREE Parameters

    main
    Parameters in IREE are externalized, device-aware storage for large resources like machine learning model weights. Instead of embedding large data directly into .vmfb files, parameters allow data to be stored in separate files (like .irpa, .safetensors, or .gguf) and linked via a scope and a unique key. This enables more efficient loading, transmission, and manipulation of large datasets.

  5. Overview of JitEval compiler-in-compiler JIT evaluation

    main

    JitEval provides compiler-in-compiler JIT evaluation tools. These tools allow delegating portions of the compilation process to both the compiler and the runtime.

    Note: This is a fragile layering that depends on most of the system components. Because of this, JitEval capabilities are isolated to the compiler/src/iree/compiler/ConstEval/ directory and must be explicitly configured for use from top-level drivers to ensure they remain isolated and optional.

  6. Overview of the IREE Hardware Abstraction Layer (HAL)

    main

    The IREE Hardware Abstraction Layer (HAL) provides a low-level abstraction over modern compute APIs (such as Vulkan, including for CPU execution). Implementations of the HAL interface enable the following capabilities:

    • Device Management: Enumerate and query devices and their specific capabilities.
    • Code Execution: Define executable code designed to run on the target device.
    • Memory Management: Allocate unified or discrete memory and manage cache control.
    • Work Submission: Organize work into sequences for deferred submission.
    • Synchronization: Provide explicit synchronization primitives to manage the ordering of submissions.

  7. Overview of IREE Common Transformations

    main

    The IREE compiler utilizes a set of common transformations and passes that are applied across various compilation phases, including Global Optimization, HAL (Hardware Abstraction Layer) Transformation, and Codegen.

    These transformations are designed to be:

    1. Self-contained: They minimize dependencies on specific backend dialects (such as the Codegen dialect) to remain versatile.
    2. Backend-agnostic: They often act as wrappers around upstream MLIR passes or implement IREE-specific logic that does not depend on a particular hardware target.

    Developers looking to extend the compiler or understand the optimization pipeline should look for these passes when implementing logic that needs to be reused across different hardware backends.

  9. Overview of IREE Python Runtime Components

    main

    The IREE Python runtime package provides a high-level API designed for two primary tasks:

    1. Running compiled IREE binaries: Execute models or kernels that have been compiled into IREE's intermediate representation format.
    2. Hardware-abstraction-layer interfacing: Interact with the underlying hardware abstraction layer (HAL) to manage device resources and execution environments.

  10. Overview of the VMVX Dialect

    main

    VMVX (Virtual Machine-based Vector eXtensions) is a dialect designed as a virtual Instruction Set Architecture (ISA) extension to the IREE VM. It exposes variable-length vector operations to prevent std and vector dialect operations from lowering to expensive scalar loops in IREE bytecode.

    VMVX operations are modeled as a machine ISA: they do not use shapes, element types are encoded directly within the operations, and memory access is tightly restricted.

  11. Overview of IREE (Intermediate Representation Execution Environment)

    main
    IREE is an MLIR-based end-to-end compiler and runtime designed to lower Machine Learning (ML) models to a unified Intermediate Representation (IR). It is built to scale across diverse hardware, from high-performance datacenters to resource-constrained mobile and edge devices.