DOCSEARCHbyAlgolia
Submit Docs

Fujitsu One Compression (OneComp)

repository·main·Indexed 19 days ago

https://github.com/fujitsuresearch/onecompression

An automated Python package for LLM compression that optimizes model performance and memory footprint by automating VRAM detection, bit-width selection, quantization, and evaluation. The library includes benchmarks for Llama-3-8B using GPTQ, JointQ, and LPCD+GPTQ, supporting various configurations such as actorder, MSE, and diagonal regularization.

Tokens
71.9K
Snippets
229
Records
342
Agent score
64%

What's inside OneCompression

  1. Overview of OneComp Benchmark Directories

    main

    The benchmark/ directory contains various benchmark runs comparing different quantization methods (GPTQ, JointQ, QEP+GPTQ) across different models (Llama-3-8B, Qwen3-8B, Qwen3-14B). Most benchmarks evaluate 4-bit and 3-bit quantization using gs128 (group size 128) and per-channel settings.

    Available benchmark directories:

    • llama3-8b-gptq/: Llama-3-8B GPTQ
    • llama3-8b-jointq/: Llama-3-8B JointQ
    • llama3-8b-qep-gptq/: Llama-3-8B QEP+GPTQ
    • llama3-8b-various/: Llama-3-8B Various quantizers (default parameters, no QEP)
    • qwen3-8b-gptq/: Qwen3-8B GPTQ
    • qwen3-8b-jointq/: Qwen3-8B JointQ
    • qwen3-14b-gptq/: Qwen3-14B GPTQ
    • qwen3-14b-jointq/: Qwen3-14B JointQ

  2. Overview of OneComp quantization features

    main

    OneComp is an open-source library for post-training quantization (PTQ) of Large Language Models. Key capabilities include:

    • Advanced Algorithms: Implements Quantization Error Propagation (QEP) for error correction and Layer-Projected Coordinate Descent (LPCD) for submodule optimization.
    • Mixed-Precision (AutoBit): Automatically assigns per-layer bitwidths based on available VRAM using ILP-based assignment.
    • JointQ: Optimizes weight assignments and scale parameters simultaneously.
    • Block-wise PTQ: Minimizes intermediate-representation MSE against an FP16 teacher model at the Transformer-block granularity.
    • Rotation Preprocessing: Uses SpinQuant/OstQuant-based rotation to reduce quantization error, supporting Llama and Qwen3 architectures.
    • Post-Processing: Supports LoRA SFT Post-Process for accuracy recovery via fine-tuning.
    • Inference & Evaluation: Integrates with vLLM for serving and includes onecomp-eval for benchmarking with MT-Bench and throughput metrics.

  3. Overview of Model Validation Suite

    main

    The Model Validation suite is an operational validation tool for the OneComp end-to-end workflow: quantize → save → load → inference. It tests various model architectures and sizes across different quantization recipes to ensure the workflow is functional and the resulting models are not broken.

    Sanity Checks Performed:

    • Perplexity (PPL): Measured on wikitext-2-raw-v1. This is used as a check to ensure the quantized model is not broken, but it is not an accuracy benchmark.
    • Greedy Generation: Performed on saved and reloaded models to verify that the load and inference paths execute without errors.

  4. Overview of OneComp Features

    main

    Fujitsu One Compression (OneComp) provides several advanced quantization and optimization techniques:

    • Quantization Error Propagation (QEP): A post-training quantization method that corrects errors by propagating them to subsequent layers.
    • Layer-Projected Coordinate Descent (LPCD): A unified PTQ framework that extends layer-wise quantization to arbitrary submodules.
    • vLLM Plugin Integration: Built-in plugins for DBF and Mixed-GPTQ quantization methods to serve models via vLLM.
    • AutoBit: Mixed-precision quantization that uses ILP-based bitwidth assignment to minimize error within a specific VRAM budget.
    • JointQ: Optimizes weight assignments and scale parameters simultaneously, supporting group-wise quantization (e.g., 4-bit, groupsize=128).
    • Block-wise PTQ: Post-quantization block-wise distillation at Transformer-block granularity using Phase 1 (greedy) and Phase 2 (CBQ) optimization.
    • LoRA SFT Post-Process: Accuracy recovery or domain-specific fine-tuning using LoRA adapters with SFT loss or teacher distillation.
    • Rotation Preprocessing: SpinQuant/OstQuant-based rotation to reduce error, supporting Llama and Qwen3 architectures via online Hadamard hooks.
    • Web Dashboard (HPC): A browser-based interface for launching jobs and validating inference in HPC environments.

  5. Understand Qwen3-14B JointQ Benchmark results

    main

    The Qwen3-14B JointQ Benchmark evaluates model compression performance using several metrics.

    Metrics used for evaluation:

    • PPL: Perplexity on WikiText-2 (lower is better).
    • Accuracy: 0-shot acc_norm where available, or acc (e.g., for Winogrande) (higher is better).
    • Time (s): The elapsed time for the compression process.

    Benchmark results are categorized by compression strategy (e.g., incremental, fixed, diagonal, actorder+mse), bit-width (e.g., 4-bit, 3-bit), and group_size (e.g., 128, per-channel).

  6. Llama-3-8B JointQ Benchmark Overview

    main
    The Llama-3-8B JointQ Benchmark evaluates Meta-Llama-3-8B using OneComp v1.1.0. It tests various combinations of quantization bits and group sizes in a single pass by sharing calibration data accumulation across quantizers to improve efficiency. The benchmark covers 27 distinct configurations, including incremental lambda modes, fixed lambda modes, diagonal regularization modes, and combinations with actorder and gptq_mse enabled. Evaluation is performed using Perplexity (WikiText-2) and Accuracy (lm-eval-harness) for both the original unquantized model and all resulting dequantized models.

  7. What is DBF (Double Binary Factorization)?

    main

    DBF is an extreme compression method that approximates weight matrices using binary factors, achieving approximately 1.5-bit quantization.

    It decomposes each weight matrix $W$ as $W \approx A \cdot \text{diag}(d) \cdot B$, where $A$ and $B$ are binary matrices (${-1, +1}$) and $d$ is a scaling vector. This approach drastically reduces storage requirements because the binary matrices only require 1 bit per element, while the scaling vector $d$ maintains the necessary dynamic range. Optimization is performed using ADMM (Alternating Direction Method of Multipliers) with optional weight balancing.

  8. What is JointQ?

    main

    JointQ is a post-training quantization method that jointly optimizes integer weight assignments and scale parameters to minimize layer-wise reconstruction error. Unlike GPTQ, which quantizes column-by-column, JointQ uses local search to optimize weight assignments and scale/zero-point parameters simultaneously.

    Key characteristics:

    • Optimization Goal: Minimizes $\min_{\hat{W}} | Y - \hat{W} X^T |_F^2$, where $Y$ is the full-precision output.
    • Decomposition: Weights are decomposed as $\hat{W}{i, g} = s{i,g} \cdot (a_{i,g} - z_{i,g})$ (scale, zero-point, and integer assignment).
    • Hardware Requirement: Requires a GPU for CUDA-based local search.
    • Compatibility: Emits quant_method="gptq", making it compatible with standard OneComp save/load APIs and vLLM's GPTQ plugin.

  9. What is LPCD (Layer-Projected Coordinate Descent)?

    main
    LPCD is a unified framework that extends post-training quantization (PTQ) from individual linear layers to larger Transformer submodules. While standard methods like GPTQ optimize one layer at a time, LPCD allows OneComp to refine interactions within entire blocks (like attention or MLP blocks) by lifting the optimization target to a submodule level. This makes it a refinement framework that works alongside existing layer-wise quantizers rather than replacing them.

  10. What is QEP (Quantization Error Propagation)?

    main

    QEP is a meta-algorithm designed to improve layer-wise quantization methods (like GPTQ) by compensating for quantization errors that propagate from one layer to the next.

    Standard Post-Training Quantization (PTQ) quantizes layers independently using original activations. However, in a quantized model, the input to layer $l+1$ is the output of the quantized layer $l$, which contains error. QEP addresses this by adjusting the weights of each layer before quantization to account for the activation error introduced by previously quantized layers. It calculates a weight correction $\Delta W$ using the Hessian matrix and a correction strength parameter, then quantizes the adjusted weight $W + \Delta W$.

  11. What is AutoBit and how does it work?

    main

    AutoBit is a mixed-precision quantization method that uses Integer Linear Programming (ILP) to automatically assign optimal per-layer bit-widths based on a specific memory budget.

    It aims to minimize quantization error while ensuring the total bits-per-weight (bpw) across all layers does not exceed a target average bit-width ($b^*$).

    Key Features:

    • Error Metrics: Supports RTN error (Frobenius norm) and Activation-aware error (using input/output curvature statistics from calibration data).
    • vLLM Compatibility: By default, it uses enable_fused_groups=True to ensure that fused layers (like q/k/v or gate/up projections) receive the same bit-width, which is a requirement for serving with vLLM.
    • DBF Fallback: For ultra-low-bit targets ($\le 2$ bpw), it can inject DBF (Double Binary Factorization) as a fallback for layers where standard GPTQ candidates would cause excessive error.

  12. Manual quantization workflow with `ModelConfig`, `Quantizer`, and `Runner`

    main

    For full control, use the manual configuration approach. This involves three components:

    1. ModelConfig: Specifies the model and device.
    2. Quantizer (e.g., GPTQ): Defines the quantization method and parameters.
    3. Runner: Orchestrates the pipeline using the config and quantizer.
    from onecomp import ModelConfig, Runner, GPTQ, setup_logger

setup_logger()

model_config = ModelConfig(
    model_id="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
    device="cuda:0",
)
gptq = GPTQ(wbits=4, groupsize=128)

runner = Runner(model_config=model_config, quantizer=gptq)
runner.run()