DOCSEARCHbyAlgolia
Submit Docs

Neural Network Compression Framework (NNCF)

repository·develop·Indexed 22 days ago

https://github.com/openvinotoolkit/nncf

NNCF provides post-training and training-time algorithms to optimize neural network inference for OpenVINO with minimal accuracy loss. It supports PyTorch, ONNX, and OpenVINO backends, offering capabilities such as 4/8-bit mixed-precision quantization, static FP8 quantization, adaptive codebook compression, and Scale Estimation for Large Language Models (LLMs).

Tokens
45.7K
Snippets
151
Records
235
Agent score
78%

What's inside NNCF

  1. Overview of Neural Network Compression Framework (NNCF)

    develop

    NNCF is a suite of post-training and training-time algorithms designed to optimize neural network inference for [OpenVINO™] with minimal accuracy loss. It supports models from PyTorch, TorchFX, ONNX, and OpenVINO.

    Key capabilities include:

    • Automatic model graph transformation.
    • Common interface for various compression methods.
    • GPU-accelerated layers for fine-tuning.
    • Distributed training support.
    • Exporting compressed PyTorch models to ONNX, SavedModel, or Frozen Graph formats for use with OpenVINO.

  2. Overview of NNCF compression algorithms and support

    develop

    NNCF provides both post-training and training-time compression algorithms. Support varies depending on the model format (OpenVINO, PyTorch, TorchFX, or ONNX).

    Post-Training Compression

    • Post-Training Quantization: Supported for OpenVINO, PyTorch, and ONNX. Experimental for TorchFX.
    • Weights Compression: Supported for OpenVINO, PyTorch, and ONNX. Experimental for TorchFX.
    • Activation Sparsity: Experimental for PyTorch only.

    Training-Time Compression

    • Quantization Aware Training (QAT): Supported for PyTorch.
    • Weight-Only QAT with LoRA and NLS: Supported for PyTorch.
    • Pruning: Supported for PyTorch.

  3. Post-training Compression Methods

    develop

    NNCF provides several post-training compression algorithms that do not require retraining the model. These methods are compatible with OpenVINO, PyTorch, TorchFX, and ONNX backends.

    Post Training Quantization (PTQ)

    Supports symmetric and asymmetric quantization, signed and unsigned types, and per-tensor or per-channel granularity. Results can be exported to the OpenVINO format.

    Weights Compression

    Offers advanced weight compression techniques including:

    • 8-bit: Symmetric compression.
    • 4-bit: Symmetric and asymmetric modes.
    • NF4: NormalFloat4 compression using arbitrary look-up tables (CODEBOOK) or predefined NF4 look-up tables (CB4).
    • Microscaling (MX) types: MXFP4 and MXFP8_E4M3.
    • Floating Point types: FP8_E4M3, FP4, and NVFP4.
    • Advanced strategies: Mixed precision weights compression and grouped weights compression.

  4. Training Time Compression Methods

    develop

    NNCF supports compression algorithms that are integrated into the training process, primarily for PyTorch models. These methods allow the model to adapt to quantization or pruning during training to minimize accuracy loss.

    Quantization Aware Training (QAT)

    Trains a quantized model (often initialized from a PTQ model). Supports symmetric/asymmetric, signed/unsigned, and per-tensor/per-channel quantization. Exports to OpenVINO format.

    Weight-Only QAT with LoRA

    Uses absorbable Low-Rank Adapters (LoRA) for weight-only quantization. It uses Post Training Weight Compression as an initialization step. It supports two formats:

    • FQ_LORA: For general accuracy improvement via distillation.
    • FQ_LORA_NLS: For tuning for downstream tasks.

    Supports symmetric/asymmetric modes, signed/unsigned types, per-channel 8-bit quantization, and group-wise 4-bit quantization. Exports to OpenVINO format with packed weight constants and a decompressor.

    Pruning

    Supports unstructured pruning for PyTorch models.

  5. Enhance 4-bit LLM accuracy with QAT and absorbable LoRA adapters

    develop

    This example demonstrates a method to improve the accuracy of Large Language Models (LLMs) compressed to 4-bit weights. It combines Quantization-Aware Training (QAT) with absorbable LoRA adapters using the FQ_LORA compression format.

    Key components of this approach:

    • Absorbable LoRA adapters: Uses traditional low-rank adaptation with fixed ranks across layers.
    • Knowledge Distillation: An uncompressed model (teacher) guides the compressed model (student) to prevent task-specific over-fitting.
    • Performance: This method significantly reduces the perplexity increase compared to standard Post-Training Weight Compression (PTWC) techniques like AWQ or GPTQ.

  6. Integrate NNCF with third-party repositories

    develop

    NNCF serves as a compression backend for several major AI and optimization ecosystems. Key integrations include:

    • HuggingFace Optimum Intel: Use optimum-cli to export models (e.g., Llama-3.2) to OpenVINO format with INT4 quantization.
    • Ultralytics: Integrated into the Intel OpenVINO export pipeline for model quantization.
    • ExecuTorch: Acts as the primary quantization framework for the OpenVINO integration.
    • torch.compile: Used as the primary quantization framework for the OpenVINO integration.
    • OpenVINO Training Extensions: Functions as a model optimization backend.
    • Microsoft Olive: Used to quantize OpenVINO IR and ONNX models via the OpenVINO integration.
    # Example: Exporting Llama-3.2-3B-Instruct to OpenVINO with INT4 quantization via HuggingFace Optimum Intel
optimum-cli export openvino -m meta-llama/Llama-3.2-3B-Instruct --weight-format int4 ./Llama-3.2-3B-Instruct-int4

  7. Use the NNCF Profiler to analyze model activations

    develop

    The NNCF Profiler is a tool designed to collect and analyze activation statistics from OpenVINO models. It allows for layer-by-layer profiling, which is useful for debugging quantization issues, comparing model variants (such as FP32 vs INT8), and understanding activation distributions.

    Key capabilities include:

    • Layer Selection: Collect raw activations at the input and output of specific layers using regex pattern matching.
    • Statistics Calculation: Compute metrics like min, max, mean, std, and various percentiles on collected activations.
    • Model Comparison: Compare activations between two model variants using built-in or custom metrics.
    • Visualization: Use matplotlib to visualize activation distributions and comparison results.
    • Extensibility: Register custom statistics, comparators, and visualizers to extend the tool's functionality.

  8. What is Memory Monitor and which memory types are supported?

    develop

    Memory Monitor is a tool used to measure the RAM footprint of a Python program over time. It supports two primary memory types:

    • MemoryType.RSS: Resident Set Size. This represents the portion of memory occupied by a process that is held in RAM. Note that RSS reports memory read via mmap (e.g., during OpenVINO model loading) as allocated, even if it is not strictly occupying physical RAM.
    • MemoryType.SYSTEM: Defined as the difference between total system virtual memory and system available memory. This metric is affected by other processes.

    Best Practices:

    • Use MemoryType.SYSTEM when analyzing Python scripts that involve OpenVINO model loading to avoid the mmap reporting bias seen in RSS.
    • When using MemoryType.SYSTEM, it is recommended to call get_data(memory_from_zero=True) to subtract the starting memory from all values, providing a cleaner view of the specific process's footprint.
    • Use the monitor for large memory processes, as the monitor itself consumes memory (especially when saving figures).

  9. What is an NNCFGraph

    develop

    To decouple compression logic from specific deep learning backends (like PyTorch), NNCF uses an abstraction called NNCFGraph.

    An NNCFGraph is a wrapper around a directed acyclic graph (DAG) that represents the control flow or execution graph of a DNN.

    Key characteristics:

    • Nodes: Each node in the graph represents a call to a backend-specific function (an "operator").
    • Dual Representation: NNCF builds NNCFGraph objects for both the original, unmodified model and for the model after compression algorithms have been applied. The compressed version may contain additional operations compared to the original graph.

  10. How NNCF computes gradients for Quantization-Aware Training (QAT)

    develop

    During Quantization-Aware Training (QAT), NNCF uses a learned-range fake quantization approach. Because operations like clamping and rounding are non-differentiable, NNCF employs custom surrogate gradients:

    1. Straight-Through Estimator (STE): Used for the rounding operation. It treats rounding as an identity function plus a constant residual during differentiation.
    2. Piecewise-defined surrogate gradients: Used for the clamp boundaries.

    Parameterization

    Instead of using (step size, zero point) as seen in LSQ (Learned Step Size Quantization), NNCF uses two parameters:

    • input_low: The lower bound of the quantization range.
    • input_range: The width of the quantization range (where input_range > 0).

    The scale factor $s$ is derived as: $s = (levels - 1) / input_range$.

    Input Partitioning

    Inputs are partitioned into three regions for gradient calculation:

    • Below range: $x < input_low$
    • In range: $input_low \le x \le input_low + input_range$
    • Above range: $x > input_low + input_range$

    Gradient Behavior

    • Gradient w.r.t. $x$: Passed through unchanged if $x$ is in range; otherwise, it is zeroed out.
    • Gradient w.r.t. $input_range$: Nudges the range to reduce quantization error. If $FQ(x) > x$, the gradient encourages shrinking the range (finer step size).
    • Gradient w.r.t. $input_low$: In asymmetric mode, shifting input_low moves the grid, but the zero-point ($ZP$) shifts to compensate, resulting in zero gradient for in-range values. Outside the range, the gradient is passed through.

    Note on Symmetric Quantization: In symmetric mode, input_low is derived from input_range and is not an independent learnable parameter; therefore, its gradient is not used directly.

  11. Handle group size mismatches during weight compression

    develop

    When performing weight compression, if a layer's channel size is not divisible by the specified group_size, NNCF provides a mechanism to handle the error.

    As of Release 2.19.0, you can control this behavior using the group_size_fallback_mode parameter (introduced in 2.18.0). The available modes are:

    • IGNORE: The default behavior; nodes that do not support the group size are skipped.
    • ERROR: Raises an exception if the channel size is not divisible by the group size.
    • ADJUST: Automatically adjusts the group size to a valid value for the problematic layers.
    # Example of using the fallback mode
ncf.compress_weights(
    model,
    group_size=64,
    group_size_fallback_mode=nncf.GroupSizeFallbackMode.ADJUST
)

  12. Organize test folder structure

    develop

    Test files should be organized into directories based on the target functionality and backend:

    • Framework-specific tests: Respective directories for each framework.
    • common/: Tests for common code functionality without backend-specific dependencies.
    • cross_fw/: Tests that reuse the same code for multiple frameworks or templates.
    • shared/: Auxiliary (non-test-case) code that can be reused across tests.