DOCSEARCHbyAlgolia
Submit Docs

AngelSlim Toolkit

repository·main·Indexed 23 days ago

https://github.com/tencent/angelslim

A toolkit for large model compression featuring integrated algorithms for quantization, distillation, and speculative decoding. It supports FP8 quantization for Diffusion Transformers via DynamicDiTQuantizer, distributed quantization-aware training (QAT) via the mcore_qad backend for models like Qwen3-MoE and Hy3, and speculative-decoding training through the AngelSpec framework. The toolkit includes support for various quantization methods (FP8-Static, SmoothQuant, INT4, INT8, DAQ, Sherry, TEQUILA), vision token pruning with VisionZip, and sparse attention algorithms like Stem.

Tokens
74.8K
Snippets
165
Records
267
Agent score
81%

What's inside AngelSlim

  1. Overview of AngelSlim features

    main

    AngelSlim is a toolkit dedicated to making Large Language Model (LLM) compression more usable, comprehensive, and efficient. Its core value propositions include:

    • High Integration: Integrates mainstream compression algorithms into a unified toolkit for one-click usage.
    • Continuous Algorithmic Innovation: Combines industry-standard algorithms with continuously released self-developed compression algorithms.
    • Extreme Performance: Focuses on end-to-end optimization for both the compression process and deployment. For example, it enables quantizing massive models like Qwen3-235B and DeepSeek-R1 on a single GPU.

  2. Overview of AngelSlim mcore QAD backend

    main

    The mcore_qad package is an isolated mcore backend designed for distributed, scale-only quantization-aware training (QAT) and distillation. It is optimized for large-scale model training workflows where quantization is a primary goal.

    Supported Models

    • Qwen3-MoE
    • Hy3 (HunYuan-3)

    Supported Quantization Formats

    The backend supports the following six formats:

    • nvfp4
    • nvfp4a16
    • w4a16
    • w8a8
    • fp8
    • w4afp8

  3. Overview of AngelSlim toolkit

    main

    AngelSlim is a comprehensive and efficient toolkit designed for large model compression. It provides a highly integrated framework that allows developers to access mainstream compression algorithms with one-click ease of use. The toolkit focuses on performance-driven workflows, such as enabling the quantization of massive models like Qwen3-235B and DeepSeek-R1 on a single GPU, and continuously integrates new research into its core offerings.

    Key capabilities include:

    • Quantization: Support for various algorithms including FP8-Static, SmoothQuant, INT4, INT8, and specialized methods like DAQ, Sherry (1.25-bit), and TEQUILA (ternary quantization).
    • Speculative Decoding: Frameworks like DFlare (block-diffusion) and AngelSpec (torch-native training) to accelerate inference.
    • Distillation: Support for full-precision HuggingFace models and quantized QAT-style models.
    • Sparse Attention: Algorithms like Stem to accelerate the prefill stage of long-context LLMs.
    • Hardware Efficiency: Optimized kernels for low-bit quantization (e.g., 1.25-bit and 2-bit models).

  4. Overview of Token Compressor

    main

    The Token Compressor is an efficient module in AngelSlim designed for Multimodal Large Language Models (MLLMs). Its primary goal is to dynamically reduce the number of visual tokens during inference. This process significantly reduces KV Cache memory consumption and accelerates both the Prefill and Decoding stages of model execution.

    Key features include:

    • Metadata-driven architecture: A universal adapter architecture that supports one-click deployment of various pruning and merging algorithms.
    • Decoupled design: The compression algorithm logic is highly decoupled from the underlying model code. This allows researchers to develop pruning or merging strategies without needing to understand the complex internal implementations of specific models, enabling fast migration and deployment across different models.
    • Integrated evaluation: Deep integration with the lmms-eval toolkit allows users to perform multi-dimensional accuracy evaluations on mainstream benchmark tasks with a single command.

  5. Overview of QAT (Quantization Aware Training)

    main

    QAT (Quantization Aware Training) is a compression method that simulates quantization effects during the training process. By inserting Fake Quantization operations during the forward pass, the model learns parameter distributions that are more resilient to the precision loss caused by quantization. This is particularly effective for low-bit quantization scenarios like W4A8 or INT4, where Post-Training Quantization (PTQ) might suffer significant accuracy degradation.

    Key Features of AngelSlim QAT:

    • Multiple Quantization Formats: Supports INT (arbitrary bit-width) and FP8 (E4M3), including W4A8-FP8 mixed precision.
    • Flexible Granularity: Supports per-tensor, per-channel, per-group, and per-token quantization.
    • Symmetric/Asymmetric Support: Supports both symmetric and asymmetric quantization modes.
    • Training Modes: Offers both End-to-End (E2E) training and Blockwise training.
    • Plugin Architecture: Features like Learnable Scale are implemented as plugins for easy extensibility.
    • Built-in Evaluation: Includes PPL (Perplexity) and ACC (Accuracy) metrics for benchmarks like wikitext2, c4, piqa, and arc.

  6. Overview of AngelSlim LLM Compression Toolkit

    main

    AngelSlim is an efficient Large Language Model (LLM) compression toolkit developed by Tencent. It provides an end-to-end workflow from compression to deployment, supporting mainstream cutting-edge models. The toolkit integrates various compression algorithms to improve usability and performance with minimal cost.

    Key Compression Algorithms Supported:

    • Quantization: Reducing precision to save memory and increase speed.
    • Speculative Decoding: Using smaller models to accelerate inference.
    • Pruning/Sparsification: Removing redundant parameters or attention heads.
    • Distillation: Training smaller models to mimic larger ones.
    • Cache/Token Compression: Optimizing KV caches and token sequences.

  7. What is Stem sparse attention?

    main

    Stem is AngelSlim's sparse attention algorithm designed to accelerate the Prefill stage of long-context LLMs (e.g., 32K–128K tokens). It reduces prefill latency by estimating attention importance at a block level and only performing precise attention on the most critical blocks.

    Key Concepts:

    • Block-Level Scoring: Uses a Triton-accelerated strided group GEMM to estimate the importance of query-blocks relative to key-blocks.
    • Top-k Schedule: Determines how many blocks to keep based on layer_keep_ratios (per-layer retention) and stem_alpha (intra-layer budget decay).
    • Block-Sparse Attention: Executes sparse attention based on the top-k mask. If the block-sparse-attn library is missing, you can set allow_pseudo_sparse: true to fall back to a dense masked implementation (though scoring still requires Triton/CUDA).

    Note: Stem only affects the Prefill stage. The Decode stage continues to use the model's original attention implementation (e.g., FlashAttention-2, SDPA).

  8. What is Quantization-Aware Distillation (QAD)?

    main

    Quantization-Aware Distillation (QAD) is a training method that trains a quantized student model using an independent full-precision teacher model. It acts as a bridge between Quantization-Aware Training (QAT) and standard Distillation.

    Key distinctions:

    • Use Distill when the student is full-precision.
    • Use QAD when the student should be quantized during the training process.

    QAD reuses QAT components (quantization modules, learnable-scale plugins, conversion, and save logic) while utilizing the Distill trainer and distillation losses. For scale-only distributed training on Megatron-Core (supporting Qwen3-MoE and HunYuan-3 with TP/EP/CP/SP parallelism), use the MCoreQAD backend instead.

  9. What is DAQ (Delta-Aware Quantization)?

    main

    DAQ (Delta-Aware Quantization) is a data-free post-training quantization (PTQ) method specifically designed for Large Language Models (LLMs). Unlike standard quantization methods that minimize reconstruction error (MSE) and can inadvertently destroy the small parameter increments ($\Delta W$) introduced during post-training (e.g., SFT, RLHF, DPO), DAQ optimizes for the directional fidelity of $\Delta W$.

    It achieves this by using two delta-aware metrics:

    1. Sign Preservation Rate: Measures the proportion of elements where the sign of the quantized delta matches the sign of the post-training delta.
    2. Cosine Similarity: Measures the directional alignment between the original delta vector and the quantized delta vector.

    Key Advantages:

    • Data-free: Requires only the base model and the post-training model weights. No calibration data, activation statistics, or Hessian estimates are needed.
    • Efficient Search: Uses a coarse-to-fine two-stage search for scale optimization.
    • Flexible: Supports both blockwise and per_channel quantization.

  10. Overview of Sparse Attention in AngelSlim

    main

    Sparse Attention is a Prefill acceleration module in AngelSlim designed for long-context Large Language Model (LLM) inference. Its primary goal is to dynamically skip unimportant attention blocks during the prefill stage, significantly reducing computation and latency.

    To use a sparse algorithm, set compression.name: Sparsity in your configuration and invoke it via the tools/run.py or tools/infer.py entry points.

  11. What is SpinQuant Transform?

    main

    SpinQuant is a weight-space rotation transform module in AngelSlim. It applies orthogonal rotations (Hadamard or random orthogonal matrices) to model weights to suppress outlier channels in activations, thereby improving Post-Training Quantization (PTQ) accuracy. The rotation is mathematically equivalent to the original model (output remains unchanged) but makes the weight distribution more uniform and quantization-friendly.

    Key Features:

    • Supports R1, R2, and R4 rotation types (R3 is currently unimplemented).
    • Offline Rotations (R1/R2): Fused into weights, resulting in zero inference overhead.
    • Online Rotation (R4): Implemented via a forward pre-hook, which can be exported to vLLM.
    • Extensibility: Uses a factory registration mechanism (@TransformFactory.register("SpinQuant")).

  12. SpinQuant implementation details: CPU computation and Parallelism

    main

    The SpinQuant implementation in AngelSlim (angelslim/compressor/transform/rotation/spin.py) uses two key optimization strategies:

    1. CPU-based Weight Fusion: To avoid high GPU memory consumption when generating large rotation matrices (e.g., [hidden_size, hidden_size]), rotation matrix generation and weight fusion are performed on the CPU (DEVICE='cpu'). The weights are moved to CPU, processed, and then written back to their original device (e.g., GPU).
    2. Multi-threaded Parallelism: The _parallel_apply() method uses a ThreadPoolExecutor (up to 64 threads) to execute independent weight fusion tasks in parallel, significantly reducing the time required for large model rotations compared to serial processing.