SmoothQuant

To install SmoothQuant, create a new Conda environment with Python 3.8 and install the required PyTorch, Transformers, and Accelerate dependencies. Finally, run the setup script from the repository root.

conda create -n smoothquant python=3.8
conda activate smoothquant
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
pip install transformers==4.36.0 accelerate datasets zstandard

python setup.py install

#!/bin/bash
conda create -n smoothquant python=3.8
conda activate smoothquant
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
pip install transformers==4.36.0 accelerate datasets zstandard

python setup.py install

The quantization process for OPT models involves smoothing, quantizing, and exporting. For developers looking to implement or customize this flow, the following files in the repository contain the implementation details:

• smoothquant/opt.py: Contains the core quantization flow logic for OPT models.
• examples/generate_act_scales.py: Demonstrates how to generate activation scales.
• examples/export_int8_model.py: Demonstrates how to export the final INT8 model.

To run SmoothQuant demonstrations or use the library, ensure the following packages are installed in your environment:

• smoothquant
• PyTorch
• Transformers
• Accelerate

To run SmoothQuant real-INT8 inference for PyTorch, you must install the following dependencies:

• smoothquant
• torch-int (provides CUTLASS INT8 GEMM kernels wrapped as PyTorch modules)
• PyTorch
• Transformers
• Accelerate

Note: For models larger than what can fit on a single A100 GPU (like OPT-30B), it is recommended to use the FasterTransformer implementation of SmoothQuant.

SmoothQuant improves W8A8 quantization performance by smoothing activation outliers. To use it, follow these steps:

1. Load your model in FP16 using Transformers.
2. Load pre-computed activation scales (e.n., .pt files).
3. Call smooth_lm with the model, the scales, and a smoothing factor (e.g., 0.85).
4. Apply W8A8 quantization using quantize_llama_like.

This process balances the quantization difficulty between activations and weights, resulting in perplexity closer to the original FP16 model compared to naive W8A8 quantization.

SmoothQuant provides INT8 inference for PyTorch using CUTLASS INT8 GEMM kernels via the torch-int library.

Prerequisite: You must install torch-int before running the inference.

For OPT models, you can use the pre-quantized models provided by MIT Han Lab on Hugging Face. The Int8OPTForCausalLM class handles the INT8 linear layers and quantization scales.

Available OPT model sizes: 125m, 1.3B, 2.7B, 6.7B, 13B, 30b, and 66b.

from smoothquant.opt import Int8OPTForCausalLM
model = Int8OPTForCausalLM.from_pretrained("mit-han-lab/opt-30b-smoothquant")

SmoothQuant provides pre-smoothed and quantized OPT models on Hugging Face. You can load these models directly using the Int8OPTForCausalLM class from smoothquant.opt.

Available model sizes for the prefix mit-han-lab/opt-[MODEL-SIZE]-smoothquant include:

• 125m
• 1.3B
• 2.7B
• 6.7B
• 13B
• 30b
• 66b

from smoothquant.opt import Int8OPTForCausalLM

# Example loading the 30B quantized model
model = Int8OPTForCausalLM.from_pretrained("mit-han-lab/opt-30b-smoothquant")

SmoothQuant improves W8A8 quantization accuracy by smoothing activation outliers and migrating quantization difficulty from activations to weights.

To apply SmoothQuant, use smooth_lm with a pre-computed activation channel scales tensor and a smoothing factor (e.g., 0.5). After smoothing, use quantize_opt to perform the W8A8 quantization.

Note: Activation scales for models like OPT and BLOOM are available in the ../act_scales/ directory of the repository.

from smoothquant.smooth import smooth_lm
from smoothquant.fake_quant import quantize_opt

# Load model
model = OPTForCausalLM.from_pretrained(
    "facebook/opt-13b", torch_dtype=torch.float16, device_map="auto"
)

# Load pre-computed activation scales
act_scales = torch.load("../act_scales/opt-13b.pt")

# Apply smoothing and then quantize
smooth_lm(model, act_scales, 0.5)
model_smoothquant_w8a8 = quantize_opt(model)

You can evaluate the language modeling perplexity (PPL) of models (OPT, BLOOM, Llama, Falcon, Mistral, Mixtral) using simulated W8A8 quantization via smoothquant/ppl_eval.py.

Arguments:

• --model_path: The name or path of the model.
• --act_scales_path: Path to the pre-generated activation scales file.
• --smooth: Enables the smoothing transformation.
• --alpha: The smoothing factor (alpha).
• --quantize: Enables quantization simulation.

python smoothquant/ppl_eval.py \
    --model_path <model_name_or_path> \
    --act_scales_path <act_scales_file_path> \
    --smooth \
    --alpha <alpha> \
    --quantize