DOCSEARCHbyAlgolia
Submit Docs

AI Model Efficiency Toolkit (AIMET)

repository·develop·Indexed 25 days ago

https://github.com/qualcomm/aimet

A software toolkit for quantizing and compressing trained PyTorch and ONNX machine learning models to improve runtime performance and reduce memory footprint on edge devices. It features techniques such as Post Training Quantization (PTQ) including Cross Layer Equalization and AdaRound, Quantization Aware Training (QAT), and model compression via Spatial SVD, Channel Pruning, and Weight SVD. The toolkit includes a Regression Framework for automated testing of quantization accuracy and performance.

Tokens
71.1K
Snippets
116
Records
402
Agent score
83%

What's inside AIMET

  1. Overview of AIMET Regression Framework

    develop

    The AIMET Regression Framework provides automated testing and evaluation for AIMET (AI Model Efficiency Toolkit) quantization techniques. It supports both ONNX and PyTorch (Torch) pipelines, allowing developers to test quantization accuracy and performance locally or on-device via Qualcomm AI Hub.

    Key Capabilities

    • Dual Framework Support: Dedicated pipelines for ONNX and Torch.
    • Quantization Techniques: Supports QuantSim, Lite Mixed-Precision (ONNX only), AdaRound, and Automatic Mixed-Precision (AMP).
    • Hierarchical Configuration: Uses a 4-level merging system (defaults → profile → model → test) to manage complex settings.
    • Suite-Based Testing: Predefined suites like nightly, weekly, and smoke for different regression frequencies.
    • Reporting: Generates HTML and CSV reports with baseline comparisons.

  2. Overview of AIMET capabilities

    develop

    AIMET (AI Model Efficiency Toolkit) is a quantization toolkit designed for deep learning models like PyTorch and ONNX. It allows developers to:

    • Simulate Quantization: Test the effects of quantization without full deployment.
    • Post-Training Quantization (PTQ): Quantize models using PTQ techniques via the AIMET-ONNX package.
    • Quantization Aware Training (QAT): Perform QAT on PyTorch models using the aimet-torch package.
    • Visualize and Experiment: Analyze model accuracy across various precision levels for activations and weights.
    • Mixed-Precision Modeling: Create models with varying precision levels.
    • Export Models: Export quantized models to the deployable ONNX format, which can then be run via ONNX-Runtime or compiled for QNN.

  3. Overview of AIMET

    develop
    The AI Model Efficiency Toolkit (AIMET) is a software toolkit designed to quantize trained machine learning models. It improves runtime performance by reducing compute load and memory footprint, making models suitable for deployment on edge devices such as mobile phones or laptops. AIMET supports models from the ONNX and PyTorch frameworks and utilizes post-training and fine-tuning techniques to minimize accuracy loss during quantization and compression.

  4. Overview of Mixed Precision in AIMET

    develop

    Mixed precision in AIMET bridges the accuracy gap caused by quantization by running sensitive model layers at higher precisions while keeping other layers at lower bit widths. This approach maintains higher accuracy with a smaller model footprint compared to uniform quantization.

    To implement mixed precision, follow these steps:

    1. Create a quantization simulation (QuantSim) object with a base precision.
    2. Run the model in mixed precision by adjusting the bit width of selected activation and parameter quantizers.
    3. Calibrate and simulate the accuracy of the mixed precision model.
    4. Export configuration artifacts to create the final mixed-precision model.

  5. Overview of AIMET quantization features

    develop

    The AI Model Efficiency Toolkit (AIMET) provides tools to modify trained floating-point models for efficient and accurate execution on edge (target) devices. The core workflow involves using Quantization simulation (QuantSim) to add quantization and dequantization (QDQ) nodes to a model, allowing you to estimate accuracy on floating-point hardware before deploying to actual quantized hardware.

    To restore accuracy lost during quantization, AIMET offers three primary techniques:

    1. Post-training quantization (PTQ): Efficient and easy to use; requires no retraining or fine-tuning. Recommended for non-LLM models or models with fewer than billions of parameters.
    2. Quantization-aware training (QAT): Fine-tunes the model with QDQ operations inserted into the graph to make parameters robust to quantization noise. Requires a training pipeline, dataset, and hyperparameter search, but can provide better accuracy at lower bit-widths.
    3. Mixed precision: Identifies sensitive layers and runs them at higher precisions to balance accuracy and model size.

  6. Overview of Quantized LoRa (QW-LoRa and QWA-LoRa) workflows

    develop

    Quantized LoRa (Low-Rank Adaptation) allows for training lightweight adapter weights while keeping the base model weights quantized. This reduces memory requirements during adaptation. There are two primary workflows:

    1. QW-LoRa: Performs LoRa training on a base model with quantized weights.
    2. QWA-LoRa: Performs LoRa training on a base model with both quantized weights and quantized activations.

    Both workflows follow these high-level steps:

    1. Calculate encodings for the base model.
    2. Perform LoRa training.
    3. Calculate all other encodings.

  7. Overview of AI Model Efficiency Toolkit (AIMET)

    develop
    AIMET is a library designed for advanced model quantization and model compression of trained neural network models. It aims to improve the run-time performance of deep learning models by reducing compute and memory requirements while maintaining task accuracy. It is specifically optimized to simulate Qualcomm Snapdragon DSP accelerators.

  8. Overview of AIMET Quantization and Compression techniques

    develop

    The AIMET examples demonstrate several key techniques for model optimization:

    Post Training Quantization (PTQ)

    • Cross Layer Equalization (CLE): Performs BatchNorm Folding, Cross Layer Scaling, and High Bias Fold. Includes Bias Correction to fix output shifts. Available for Torch and ONNX.
    • AdaRound (Adaptive Rounding): A weight-rounding mechanism that adapts to data and task loss. It is computationally fast and works on convolutional or fully connected layers without end-to-end finetuning. Available for Torch.

    Quantization Aware Training (QAT)

    • Quantization-aware Training: Simulates on-target quantized inference during training to improve accuracy. Available for Torch.

    Model Compression

    • Spatial SVD: A tensor decomposition technique that decomposes a large layer into two smaller layers using rank decomposition. Available for Torch.
    • Channel Pruning: Removes redundant input channels from a layer and reconstructs weights. Available for Torch.
    • Weight SVD: Decomposes a kernel of size (m, n, h, w) into two kernels of size (m, k, 1, 1) and (k, n, h, w), where k is the rank. Available for Torch.

  9. Overview of LM Driver Key Classes

    develop

    The models/ package provides the inference driver for LLMs and VLMs with static-shape constraints. Key components include:

    • Generator: Manages KV cache, pads inputs to static shapes, and slices long sequences into multiple forward passes. Supports HF-compatible forward() and generate().
    • VLM_Generator: An extension of Generator that includes vision encoding, embedding merging, and position ID processing.
    • ONNXExportableModuleWithCache: A wrapper for HuggingFace PreTrainedModel objects that converts Cache objects into flat tensors to satisfy the I/O contract.
    • LLM / VLM: Abstract base classes defining the interface for model instantiation, I/O naming, and quantsim setup.

  10. Model Compression techniques

    develop

    AIMET provides several methods to reduce model size and increase inference speed:

    • Spatial SVD: A tensor decomposition technique that splits a large layer into two smaller ones.
    • Channel Pruning: Removes redundant input channels from a layer and reconstructs layer weights.
    • Per-layer compression-ratio selection: Automatically selects the optimal compression ratio for each layer in the model.