GPTQModel Documentation

EoRA (Eigenspace Low-Rank Approximation) is a training-free method designed to mitigate quantization errors in compressed Large Language Models (LLMs). It uses a calibration dataset to construct low-rank matrices that compensate for the loss of accuracy caused by quantization.

Key characteristics:

• Training-free: It does not require model fine-tuning.
• Calibration-driven: You can use general-purpose data (like C4) to boost overall model quality, or task-specific data (like MMLU validation sets) to improve performance on specific downstream tasks.
• Efficiency: The generation process takes approximately the same amount of time as standard GPTQ quantization.

Machete is a Cutlass-based Mixed Precision GEMM optimized for Hopper architectures. It performs quantized matrix multiplication using the following logic:

scale_type = w_s.dtype
compute_type = a.dtype
out = (w_q.to(scale_type) * w_s - w_z.to(scale_type)) @ a

Important Note on Zeropoints (w_z):
Because w_z is subtracted after the scales are applied to allow for FMA (Fused Multiply-Add) operations, the supplied zeropoints must have the scales pre-applied if they were originally intended to be subtracted before scaling.

The quantization protocol in gptqmodel is a pipeline-based configuration system used to define how a model is quantized. It is designed to be stage-based, where each stage contains a set of rules that match specific model components (modules) and apply operations to them.

The protocol can be authored in two ways:

1. Python DSL: An ergonomic builder API for programmatic configuration.
2. YAML/JSON: A portable serialized format for checked-in configs and non-Python tooling.

The root of the protocol consists of two primary fields:

• version: The protocol version.
• stages: An ordered list of Stage objects defining the execution flow.

version = 2

stages = [
    Stage(
        name="ptq",
        rules=[
            Rule(
                match="*",
                # ... other fields
            ),
        ],
    ),
]

Each tensor target (e.g., weight, input, or output) can define three primary stages of the quantization lifecycle:

1. prepare: Local pre-quantization transformations applied to the target. Use this for target-only modifications like clipping or padding.
   • Example: clip.mad(k=2.75) or pad.columns(multiple=4).
2. quantize: Defines the method used to compute the quantized state.
   • Example: gptq(bits=4, sym=True) or skip() to bypass quantization.
3. export: Defines the final encoded representation (format, variant, and implementation) for the target.

Placement Rule: Use prepare for local target-only modifications. Use actions for modifications that involve cross-target or rule-context logic.

weight={
    "prepare": [...],     # optional
    "quantize": ...,      # optional
    "export": ...,        # optional
}

The protocol distinguishes between operations that affect the module context (actions) and operations that affect specific tensors (prepare).

• actions: Used for rule-scoped or cross-target behavior (e.g., smoothquant, awq_balance, calibrate_router). These run in the context of the rule match but do not rematch the whole model.
• weight.prepare: Used for target-local pre-quantization behavior (e.g., local weight clipping, padding, or smoothing).

Decision Guide:

• If the operation is a balancing step like SmoothQuant: use actions.
• If the operation is a local transformation on a specific tensor: use weight.prepare.

Rules in the protocol act as patches over an accumulated configuration. A broad rule defines defaults, and narrower rules (matched via regex) patch only specific fields. Unchanged fields are inherited from the broader rule.

Example Workflow:

1. A global rule (match: "*") sets method: gptq, bits: 4, and format: gptq.
2. A specific rule (match: ".*down_proj$") sets bits: 3.
3. The effective configuration for down_proj becomes: method: gptq, bits: 3, group_size: 128, and format: gptq.

Advanced: mode: replace By default, rules merge. To bypass merging and explicitly replace the entire target configuration, use mode: replace.

- match: "*"
  weight:
    quantize:
      method: gptq
      bits: 4
      sym: true
      group_size: 128
    export:
      format: gptq

- match: ".*down_proj$"
  weight:
    quantize:
      bits: 3

When working with Huawei Ascend NPUs using the Ascend Extension for PyTorch (7.3.0) and PyTorch (2.9.0), you must replace torch.cuda.* calls with their corresponding torch_npu.npu.* or torch.npu.* equivalents.

Key Mapping Rules:

• For supported APIs, replace the cuda namespace with npu.
• torch_npu.npu.* and torch.npu.* are functionally equivalent.
• Graph Support: Most CUDAGraph equivalents (e.g., torch.npu.NPUGraph) are supported for inference only; training is unsupported.
• Device Properties: torch.npu.get_device_properties only populates name, total_memory, L2_cache_size, cube_core_num, and vector_core_num. Other CUDA properties will be empty.
• Unverified APIs: Any PyTorch CUDA API not explicitly listed in the support tables should be treated as unverified on NPU.