ExLlamaV3 Documentation

Use the convert.py script to quantize unquantized Hugging Face models into the EXL3 format.

Required Arguments:

• -i / --in_dir: Path to the source HF model directory (must contain config.json, tokenizer.json, and .safetensors weights).
• -o / --out_dir: Destination directory for the converted EXL3 model.
• -b / --bits: Target average bits per weight (bpw).

Recommended Arguments:

• -w / --work_dir: Directory for temporary files and checkpoints. Ensure it has enough space for a full copy of the output model.
• -ss / --shard_size: Output shard size in MB (default: 8192). Set to 0 to disable sharding.
• -hq: Increases bitrate for select layers (attention, shared-expert) to improve fidelity with minimal size increase (typically 0.05 - 0.10 bpw for MoE models).
• -hb / --head_bits: Bits per weight for the lm_head layer (integer 1-8, default: 6).

python convert.py -i /mnt/models/llama3.1-70b-instruct \
                  -o /mnt/models/llama3.1-70b-instruct-exl3-3.75bpw \
                  -w /mnt/temp/exl3 \
                  -b 3.75

You can speed up quantization by using multiple GPUs. By default, only the first visible device (device 0) is used.

Options for Multi-GPU:

• -d / --devices: A comma-separated list of GPU device IDs (e.g., 0,1,2).
• -dr / --device_ratios: A comma-separated list of relative ratios to distribute the encoding workload. This is useful for heterogeneous GPU setups to prevent slower GPUs from bottlenecking the process. For example, 1,1,3 distributes workload relative to those values.
• -pm / --parallel_mode: An experimental mode that fully parallelizes quantization across multiple GPUs by distributing entire linear layers to each GPU rather than splitting trellis encoding tiles. This can improve throughput for models with many small tensors (like MoE models).

# Example: Convert using the first three devices
python convert.py -i /mnt/models/llama3.1-70b-instruct \
                  -o /mnt/models/llama3.1-70b-instruct-exl3-3.75bpw \
                  -w /mnt/temp/exl3 \
                  -b 3.75 \
                  -d 0,1,2

# Example: Multi-GPU with specific ratios and high-quality layers
python convert.py -i /mnt/models/qwen3.5-35b-a3b \
                  -o /mnt/models/qwen3.5-35b-a3b-4.00bpw-plus \
                  -w /mnt/temp/exl3 \
                  -b 4.00 \
                  -hq \
                  -d 2,0,1 \
                  -dr 3,4,5

ExLlamaV3 can be installed using three different methods. Before installing, ensure you have an appropriate version of PyTorch installed (CUDA 12.4 or later), as the Torch dependency is not automatically handled by pip.

Method 1: Prebuilt Wheel (Recommended)

If you are unsure about build dependencies, download a wheel from the releases page and install it directly via pip.

Method 2: PyPi

Install via pip. Note that this method does not contain a prebuilt extension and requires the CUDA toolkit and build prerequisites (e.g., VS Build Tools on Windows, gcc on Linux, python-dev headers).

Method 3: Building from Source

1. Install a flash-attn-2 wheel.
2. On Windows, ensure triton-windows is installed for optimal performance.
3. Clone the repository and install requirements.
4. Install the library into your active virtual environment.

# Method 1: Prebuilt wheel example
pip install https://github.com/turboderp-org/exllamav3/releases/download/v0.0.6/exllamav3-0.0.6+cu128.torch2.8.0-cp313-cp313-linux_x86_64.whl

# Method 2: PyPi
pip install exllamav3

# Method 3: Building from source
git clone https://github.com/turboderp-org/exllamav3
cd exllamav3
# (Optional) git checkout dev
pip install -r requirements.txt
pip install .

ExLlamaV3 uses several environment variables to tune attention performance and behavior. These are read at import time (Python) or on first use (C++). Set them before loading a model.

Graph-Captured Decode Attention

• EXL3_BC_ATTN (default: 1): Enables graph-captured C++ decode attention. This runs the entire attention block (projections, RoPE, cache append, etc.) as a single CUDA graph per (bsz, q_len) shape, reducing Python host overhead. Significant for host-bound setups. Set to 0 to disable.
• EXL3_BC_ATTN_TRACE (default: 0): If enabled, prints one line per attention module/cache-layer pair when the graph-captured path is built or declined. Useful for verifying if EXL3_BC_ATTN is actually working.

Quantized K/V Cache Staging

• EXL3_QC_STAGING (default: 1): Controls how quantized K/V caches feed attention kernels.
  • 0: No staging. Packed cache tensors feed kernels directly. Lowest memory usage, but prefill pays a 5–25% performance penalty due to in-kernel expansion.
  • 1: Prefill staging (default). Dequantizes prefill chunks (256+ tokens) into an fp16 scratch. High performance (~1–3% of fp16) but requires a memory reservation at load time.
  • 2: Full staging. Legacy path where whole cache layers are expanded into fp16 temporaries. Use for debugging.
• EXL3_QC_PF_TWO_PASS_MIN_Q (default: 256): The query-length threshold for prefill staging when EXL3_QC_STAGING=1. Chunks shorter than this use the direct path.
• EXL3_QC_PREFILL_NS (default: 0): Pipeline stage count for the direct quantized-cache prefill kernel. 0 triggers automatic measurement of the best value (1 or 2).

# Example: Enabling graph-captured attention and disabling staging to save memory
export EXL3_BC_ATTN=1
export EXL3_QC_STAGING=0