DOCSEARCHbyAlgolia
Submit Docs

AutoRound Quantization Toolkit

repository·main·Indexed 23 days ago

https://github.com/intel/auto-round

An advanced quantization toolkit for LLMs, VLMs, and diffusion models using sign-gradient descent to achieve high accuracy at 2–4 bit widths. It features broad hardware compatibility (NVIDIA GPUs, Intel CPUs, Intel GPUs, and Gaudi HPUs) and includes BesTLA, a header-only acceleration library for high-performance GEMM on Intel platforms. Supports weight-only quantization configurations from INT1 to FP8, post-operation fusion, and integration with a specialized vLLM fork.

Tokens
30K
Snippets
65
Records
132
Agent score
79%

What's inside AutoRound

  1. What is a Processor in MLLM quantization?

    main
    A Processor is a callback interface used to call different processors, such as text or image processors, for MLLMs. Users can define their own custom processor and then declare it using the registration function. Detailed implementation details can be found in auto_round/compressors/mllm/processor.py.

  2. What is BesTLA and its core abstractions

    main

    BesTLA is a lightweight, header-only acceleration library for high-performance GEMM and related computations on Intel platforms. It uses a template-based architecture inspired by Cutlass, allowing users to construct flexible kernels by combining several template classes:

    • Launcher: Schedules computation-related template classes (GemmCore, Prologue, and Epilogue).
    • Parallel: Defines the data splitting strategy for task distribution. The default implementation uses an L2-cache-fusion concept where each core attempts to store processed data in its L2 cache during each gemm-tile computation round.
    • GemmCore: The central micro-kernel class for tile GEMM computation. Supported ISAs include:
      • AVX2: sgemm, u8s8 igemm
      • AVX_VNNI: u8s8&s8s8 igemm
      • AVX512F: sgemm
      • AVX512BW: u8s8 igemm
      • AVX512_VNNI: u8s8 igemm
      • AMX_BF16: bf16 hgemm
      • AMX_INT8: s8s8&u8s8&u8u8&s8u8 igemm
      • AVX512_FP16: fp16 hgemm
    • Prologue: Preprocesses input data (e.g., data type conversion or padding) to meet GemmCore requirements.
    • Epilogue: Post-processes results (e.g., eltwiseop-fusion) to expand application scenarios.

  3. Use Rotation (Experimental) for better low-bit quantization

    main

    Rotation redistributes outliers in weights and activations to make them more uniform, which is highly effective for aggressive low-bit schemes like MXFP4, NVFP4, or W4A4.

    Warning: This is an experimental feature. Inference relies on forward hooks and currently only supports the Hugging Face Transformers backend, which may result in slower inference speeds compared to non-rotated models.

    To use rotation, pass a rotation_config to the AutoRound constructor. The recommended preset is "quarot" (deterministic Hadamard rotation), which requires no training or calibration data.

    Quantized models with rotation can be saved and loaded transparently; rotation matrices and hooks are automatically restored during loading.

    from auto_round import AutoRound

model_name = "Qwen/Qwen3-0.6B"

# QuaRot preset: Deterministic Hadamard, no training required
ar = AutoRound(model_name, scheme="MXFP4", rotation_config="quarot")
ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")

  4. Choose a quantization configuration scheme

    main

    AutoRound provides several pre-defined configuration schemes based on your requirements for precision and speed.

    • 4-bits (W4A16): Use the default scheme.
    • 2-bits (W2A16): Use the best scheme for higher precision.
    SchemeBatch SizeIterationsSeq LenCalibration SamplesLearning Ratedisable_opt_rtn
    default82002048128AutoFalse
    best810002048512AutoFalse
    light85020481285e-3False
    opt_rtn802048128AutoFalse
    rtn8020480AutoTrue

  5. Use AutoScheme for automatic mixed-precision quantization

    main

    AutoScheme automatically generates an adaptive mixed-bit/mixed-data-type quantization scheme.

    Key Constraints:

    • It does not support automatic quantization for the Embedding layer; these layers will default to the highest precision configuration in the candidate set.
    • When used with model_free=True, it only supports INT (W2A16, W4A16, W8A16) and MXFP (MXFP4, MXFP8) families. You cannot mix these families in a single AutoScheme call.
    • Mixed data types are supported for tuning but cannot currently be exported to actual models.

  6. Programmatically configure AutoRound settings

    main
    Instead of setting environment variables manually in the shell, you can use the set_config() function to configure multiple variables programmatically within your Python script. This is a convenient way to manage settings for the current process and its children.

  7. Use OPT-RTN mode for fast baseline quantization

    main

    OPT-RTN (optimized Round-To-Nearest) provides a fast baseline quantization without requiring calibration data.

    How to enable:

    • API: Set iters=0 in the AutoRound constructor. It is recommended to use group_size=32 for better results.
    • CLI: Use the dedicated auto-round-opt-rtn shortcut, which is equivalent to auto-round --iters 0 --enable_opt_rtn.

    Note: For GGUF formats, the RTN algorithm is optimized. To use the original (non-optimized) RTN, use the --disable_opt_rtn flag or the auto-round-rtn CLI command.

    # 优化版 RTN（推荐的快速基线方案）
auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"

# 原始 RTN（速度最快、显存占用最低；仅作基线参考）
auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"

  8. Use AutoScheme for adaptive mixed-precision quantization

    main

    AutoScheme (experimental) automatically generates adaptive mixed-precision/data-type quantization recipes to reach a target average bit-width.

    AutoScheme Hyperparameters

    • avg_bits (float): The target average bits for the entire model (calculated only for quantized layers).
    • options (str | list[str] | list[QuantizationScheme]): A set of candidate quantization configurations (e.g., ["W4A16", "W2A16"]).
    • ignore_scale_zp_bits (bool): If True, ignores scale and zero-point bits when calculating the average bit-width.
    • shared_layers (Iterable[Iterable[str]]): Defines groups of layers that share the same quantization configuration.
    • batch_size (int): Batch size for the process. Setting to 1 reduces VRAM usage but increases time.
    from auto_round import AutoRound, AutoScheme

model_name = "Qwen/Qwen3-8B"
avg_bits = 3.0
scheme = AutoScheme(avg_bits=avg_bits, options=("GGUF:Q2_K_S", "GGUF:Q4_K_S"), ignore_scale_zp_bits=True)
layer_config = {"lm_head": "GGUF:Q6_K"}

# For non-GGUF schemes, set iters to 200
ar = AutoRound(model=model_name, scheme=scheme, layer_config=layer_config, iters=0)
ar.quantize_and_save()

  9. How QuaRot and SpinQuant rotation transforms work

    main

    Rotation transforms redistribute outliers in weights and activations before quantization by multiplying tensors with orthogonal (Hadamard) matrices. This makes intermediate distributions flatter and more quantization-friendly, which is particularly useful for aggressive low-bit schemes like MXFP4, NVFP4, and W4A4.

    AutoRound provides two main implementations:

    1. QuaRot / SpinQuant: An architecture-aware, full-model rotation applied at up to four specific positions (R1–R4). This is the recommended approach.
    2. Per-Linear Block Rotation: A simpler implementation that applies a block-diagonal Hadamard uniformly to every nn.Linear. This can be selected via rotation_config="default" or the --algorithm hadamard CLI flag.

    Note: Rotation transforms are currently experimental. Inference relies on forward hooks (supported by Hugging Face Transformers), which may result in slower inference compared to native models.

    # Example of using the recommended QuaRot preset
from auto_round import AutoRound

model_name = "Qwen/Qwen3-0.6B"
# QuaRot preset: R1+R2 deterministic Hadamard, no training
ar = AutoRound(model_name, scheme="MXFP4", rotation_config="quarot")
ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")

  10. How rotation_config is dispatched in AutoRound

    main

    The rotation_config parameter accepted by AutoRound is normalized by the apply_rotation() entry point in auto_round/algorithms/transforms/__init__.py using the following dispatch logic:

    • Direct Instance: If a BaseRotationConfig instance (SpinQuantConfig or RotationConfig) is provided, it is used directly.
    • SpinQuant Shortcuts: The strings "quarot" or "spinquant" map to SpinQuantConfig shortcuts.
    • Dictionary Mapping: A dictionary containing algorithm="spinquant" is converted into a SpinQuantConfig.
    • Default/Hadamard Mapping: Any other string (e.g., "default", "hadamard", "random_hadamard") or dictionary maps to the per-linear RotationConfig.

  11. Use Model-Free Mode for low-memory quantization

    main

    Model-Free Mode allows performing RTN WOQ quantization without loading the full model into memory. It downloads safetensors files and quantizes each Linear weight tensor shard-by-shard. This is ideal for resource-constrained environments.

    Key Features:

    • No model object required: Only needs config.json and safetensors files.
    • Low disk/RAM usage: Processes shards individually.
    • Layer-specific configuration: Supports --layer_config for per-layer bitwidths and --ignore_layers to keep specific layers in full precision.
    • Automatic Routing: In CLI, passing --iters 0 --disable_opt_rtn with a supported INT WOQ or MXFP scheme automatically triggers Model-Free mode.

    Supported Schemes in Model-Free Mode:

    • Integer Weight Quantization (outputs auto_round:auto_gptq format):
      • W2A16, W2A16G32, W2A16G64, W4A16 (default), W4A16_MIXED, W8A16.
    • MXFP (Microscaling Floating Point) (outputs mxfp4-pack-quantized or mxfp8-quantized formats):
      • MXFP4, MXFP8.

    Warning: Schemes like W3A16, FPW8A16, GGUF:*, etc., are not supported in Model-Free mode and will raise a ValueError. Use the standard AutoRound flow for these.

    from auto_round import AutoRound

AutoRound(
    model="meta-llama/Llama-3.2-1B-Instruct",
    scheme="W4A16",  # Also supports QuantizationScheme objects
    layer_config={
        ".*k_proj": {"bits": 8, "group_size": 32},
        ".*v_proj": {"bits": 8, "group_size": 32},
    },
    ignore_layers="mlp",
    model_free=True,
).quantize_and_save("./int4-llama")

  12. Fast tuning LayerNorm and Linear bias via fake quantization

    main

    AutoRound provides a method for tuning LayerNorm and Linear bias parameters to improve low-bit quantization performance (e.g., 2-bit).

    Warning: The author notes that performance is poor in most scenarios, and this method is not recommended for use at this time.

    Instead of using Adam, this approach limits tuned parameters to a quantization space. It introduces a trainable parameter $V$ in the range $[-0.5, 0.5]$, which can be tuned using SignSGD. The formula used is:

    $$W' = s \cdot \text{clip}(W/s + zp + v, N, M)$$

    Where:

    • $s$ is the quantization scale (predefined by $W$ and hyperparameters like bits).
    • $v$ is the trainable parameter.
    • $N, M$ are the quantization bounds.

    A key distinction in this method is the removal of rounding, as LayerNorm and bias weights are typically kept at 16-bit precision in most deployment scenarios. This reduces unnecessary rounding loss.