llm-compressor

AutoRound is an advanced quantization technique that optimizes rounding values and clipping ranges using three trainable parameters (V, α, and β). It processes decoder layers sequentially using block-wise output reconstruction error as the training objective.

Key benefits include:

• High-accuracy, low-bit quantization: Particularly effective for sub-4-bit (INT2/INT3) where it can achieve 10–20% absolute accuracy improvements over standard PTQ.
• Compatibility: Results are fully compatible with compressed-tensors and can be served directly with vLLM.
• Versatility: Performs well for large models (≈30B+), small-to-medium LLMs, and emerging formats like MXFP4 and NVFP4.

The llm-compressor framework is designed to be largely DDP-agnostic, but specific components require awareness to function correctly in a distributed environment:

• Modifiers: Must be DDP-aware to handle distributed operations.
• Observers: Provide an interface that modifiers use to synchronize activation statistics across processes.
• Saving and Data Preparation: These areas include specific DDP-awareness to prevent common errors (foot-guns) like multiple processes attempting to write the same file simultaneously.

Other components, such as the sequential pipeline, remain DDP-blind and do not require modification.

An Observer is a utility class used during calibration to analyze weight and activation tensors. The process follows a two-phase design:

1. Observe: Accumulate statistics from tensors using forward() or update_statistics_from_observed(). Common statistics include min_vals and max_vals.
2. Compute: Derive quantization parameters (such as scale, zero_point, and global_scale) from the accumulated statistics using get_qparams().

This design allows for complex behaviors like Observer Fusion, where multiple observers (e.g., for Q, K, and V projections) can share a single global_scale computed from their combined statistics.

LLM Compressor provides tested quantization configurations and recommended parameters for several key models. You can find specific quantization examples, including precision settings (e.g., FP8, NVFP4) and algorithm-specific setups (e.g., HCA, CSA, mHC), in the dedicated model documentation pages for:

• DeepSeek V4: Supports HCA, CSA, and mHC, quantized to FP8 + NVFP4.
• Qwen3.5: Covers vision-language and sparse MoE models.
• Qwen3.6: Specifically covers the Qwen3.6-35B-A3B sparse MoE model.
• Kimi-K2.6: Moonshot AI's multimodal agentic model.
• Gemma 4: Google's multimodal model.
• Llama 4: Meta's Llama 4 Scout multimodal model.
• Mistral Large 3: Mistral's 675B parameter model.
• HY-V3: Tencent's sparse MoE model, quantized to NVFP4 + FP8.
• GLM-5.2: THUDM's mixed dense/MoE model, quantized to NVFP4 + FP8.

The imatrix_mse observer implements importance-weighted quantization. It uses per-channel activation importance (E[x²]) to weight quantization error during range selection, ensuring channels that carry more signal receive more careful range optimization.

When used as a weight observer, imatrix_mse collects E[x²] per input channel during calibration via forward pre-hooks and applies importance weighting in the MSE grid search using the formula: err = sum(importance * |Q(w) - w|^p).

LLM Compressor provides state-of-the-art quantization and pruning techniques to optimize large language models. Key benefits include:

• Reduced hardware costs: Achieve 50-75% memory reduction, allowing deployment on fewer GPUs.
• Improved inference speed: Lower latency and higher throughput via optimized kernels and specialized hardware (e.g., Tensor Cores).
• Maintained accuracy: Uses advanced algorithms to preserve model quality (e.g., <1% accuracy difference in some studies).
• Broad support: Compatible with standard LLMs, multimodal models, and Mixture of Experts (MoE) architectures.
• Production-ready: Outputs use the compressed-tensors format, ensuring seamless integration with vLLM and Hugging Face.

LLM Compressor is a library designed to optimize large language models (LLMs) for deployment. It provides various quantization techniques to balance model quality, performance, and resource efficiency. The typical workflow involves:

1. Installation: Setting up the library via pip or from source.
2. Compression: Using different algorithms and formats to compress your model.
3. Deployment: Using vLLM to run the compressed model for efficient inference.

To run quantization examples for the Qwen3.6-35B-A3B sparse MoE model, you must ensure transformers >= v5 is installed. You can upgrade your environment using uv pip.

uv pip install --upgrade transformers

To install llm-compressor for using AutoRound quantization, clone the repository and install it in editable mode using pip.

git clone https://github.com/vllm-project/llm-compressor.git
cd llm-compressor
pip install -e .

The basic pipeline is faster than the sequential pipeline but should only be used if the model (plus auxiliary memory like GPTQ hessians) fits entirely within available VRAM. For this pipeline, load the model directly onto GPU devices using device_map="auto" and specify pipeline="basic" when calling oneshot.

model = AutoModelForCausalLM.from_pretrained(model_stub, device_map="auto")  # model is on devices
...
oneshot(model, ..., pipeline="basic")