AutoAWQ Documentation

Fused modules combine multiple layers into a single operation to increase efficiency. They are activated by setting fuse_layers=True during AutoAWQForCausalLM.from_quantized.

Important Constraints when using Fused Modules:

• Linux Only: The primary accelerator (FasterTransformer) is only compatible with Linux.
• Fixed Cache: A custom cache is used that preallocates based on batch_size and max_seq_len. You cannot change the sequence length after the model is created.
• Initialization: Use AutoAWQForCausalLM.from_quantized(max_seq_len=seq_len, batch_size=batch_size) to set these parameters.
• Dummy Values: The past_key_values returned by model.generate() are dummy values and cannot be used for subsequent generation steps.

For specialized models (like Qwen2-VL or MiniCPM3), you can extend the AwqQuantizer class to define custom logic for initialization or weight clipping.

When using a custom quantizer, pass the class instance to the quantizer_cls argument in the .quantize() method. This allows you to override how calibration data is processed or how the best clipping values are computed.

AutoAWQ supports two versions of quantization based on how matrix multiplication is executed:

• GEMV (quantized): Best for batch size 1. It is approximately 20% faster than GEMM but is not suitable for large contexts.
• GEMM (quantized): Best for batch sizes below 8 and large contexts. It is much faster than FP16 in these scenarios.

Note on Throughput: If you require the highest possible throughput, it is recommended to use vLLM with non-quantized FP16 models.

Install the package using pip:

pip install autoawq

Important Compatibility Note: Your torch version must match the version used to build the AutoAWQ wheel. For example, you cannot use torch 2.0.1 with a wheel built with 2.2.0.

Before installing, ensure your hardware meets these requirements:

• NVIDIA GPUs: Must be Compute Capability 7.5 or later (Turing architecture and newer). CUDA version must be 11.8 or later.
• AMD GPUs: ROCm version must be compatible with Triton.
• Intel CPU/GPU:
  • For optimized performance, use torch and intel_extension_for_pytorch version 2.4 or later.
  • Alternatively, use Triton kernels by installing intel-xpu-backend-for-triton along with compatible torch and transformers versions.

To use AWQ-scaled weights in other frameworks like llama.cpp, follow these steps:

1. Quantize with compatibility mode: Call .quantize() with export_compatible=True. This computes AWQ scales and applies them to the weights but skips the actual weight packing, resulting in an FP16 model with AWQ scales applied.
2. Save the model: Use .save_quantized() to save the FP16 model.
3. Convert to GGUF: Use llama.cpp's convert.py to convert the HuggingFace FP16 weights to GGUF FP16.
4. GGUF Quantization: Use llama.cpp's quantize tool to perform the final quantization (e.g., 4-bit) on the GGUF file.

To avoid Out-of-Memory (OOM) errors when quantizing models with long contexts or large datasets, adjust the following parameters in the .quantize() method:

• n_parallel_calib_samples: When set to an integer, samples are offloaded to system RAM to save GPU VRAM. This helps prevent OOM but requires sufficient system memory.
• max_calib_samples: Limits the number of calibration samples used. For AWQ, 128-256 samples are typically sufficient.
• max_calib_seq_len: Limits the sequence length of the calibration samples.

model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data=load_cosmopedia(),
    n_parallel_calib_samples=32,
    max_calib_samples=128,
    max_calib_seq_len=4096
)

To run inference on a CPU, you should install intel_extension_for_pytorch (IPEX).

Requirements:

• Install IPEX: pip install intel_extension_for_pytorch.
• Ensure your torch version matches the version IPEX was built with (e.g., IPEX 2.4 requires torch 2.4).
• If building IPEX from source, ensure torch version consistency.

When loading the model, set use_ipex=True.

model = AutoAWQForCausalLM.from_quantized(
    ...,
    use_ipex=True
)

You can benchmark the speed (prefill and decoding) and memory usage of models using the provided benchmark.py script.

GPU Benchmarking To benchmark on a GPU, use the following command: python examples/benchmark.py --model_path <hf_model> --batch_size 1

CPU Benchmarking To benchmark on a CPU using the Hugging Face generator, use: python examples/benchmark.py --model_path <hf_model> --batch_size 1 --generator hf

Note that performance is highly dependent on GPU memory bandwidth and CPU single-core clock speed.

AutoAWQ can be installed via PyPI using different options depending on your hardware and performance requirements:

1. Default: Uses Triton for inference and includes no external kernels.

   pip install autoawq

2. With Kernels: Installs AutoAWQ_kernels. Requires matching the latest Torch version used during the kernel build.

   pip install autoawq[kernels]

3. Intel CPU/XPU Optimized: Requires at least torch 2.4.0.

   pip install autoawq[cpu]

For AMD GPUs, inference runs through ExLlamaV2 kernels without fused layers. When loading a quantized model using AutoAWQForCausalLM.from_quantized, you must explicitly disable fused layers and enable ExLlamaV2.

model = AutoAWQForCausalLM.from_quantized(
    ...,
    fuse_layers=False,
    use_exllama_v2=True
)