Apply model quantization
mainOlive supports a wide range of model quantization techniques to reduce precision and optimize model deployment.
Microsoft Olive Documentation
repository·main·Indexed 25 days ago
https://github.com/microsoft/oliveOlive (Onnx LIVE) is an AI Model Optimization Toolkit for the ONNX Runtime. It provides tools to compose optimal optimization techniques for specific models and target hardware to produce efficient ONNX models for cloud or edge inference. The toolkit includes an automatic optimizer, a Model Context Protocol (MCP) server for AI assistants, and capabilities for quantization (RTN, GPTQ, AWQ, HQQ), LoRA/QLoRA fine-tuning, and benchmarking.
What's inside Olive
Integrate with IHV toolkits
mainOlive provides integration support for IHV (Intel Hardware Verification) toolkits.Use PEFT adapters like LoRA
mainOlive supports Parameter-Efficient Fine-Tuning (PEFT) adapters, specifically including LoRA (Low-Rank Adaptation).Convert models between formats
mainOlive provides tools to convert models from one format to another (e.g., converting PyTorch models to ONNX).Integrate with Hugging Face models, data, and metrics
mainOlive provides native integration with the Hugging Face ecosystem. You can use Hugging Face to source models, datasets, and evaluation metrics for your optimization pipelines.Apply ONNX graph transformations
mainOlive supports various ONNX graph transformations to optimize the computational graph for better performance on target hardware.Share model cache on Azure
mainOlive provides functionality to share a model cache on Azure, which can be useful for distributed environments or collaborative workflows.Perform model compression and pruning
mainOlive includes capabilities to compress or prune models to reduce their size and improve inference efficiency.Key features and capabilities of Olive
mainOlive provides several high-level capabilities for model deployment and optimization:
- Automated Optimization: Reduces manual trial-and-error for graph optimization, compression, and quantization by automatically finding the best parameters for a defined target and precision.
- Extensive Component Library: Includes over 40 built-in optimization components for quantization, compression, graph optimization, and finetuning.
- Orchestration: Users can use an easy-to-use CLI for common tasks or construct complex workflows using YAML/JSON files to orchestrate model transformations.
- Deployment Support: Built-in model packaging, deployment, and support for Multi LoRA serving.
- Integrations: Native support for Hugging Face and Azure AI.
- Productivity: Includes a built-in caching mechanism to speed up optimization workflows.
Split models into multiple components
mainOlive supports model splitting, allowing you to break a large model into multiple smaller, manageable components.Train LoRA adapters for Diffusion models
mainOlive can be used to train LoRA adapters for specific diffusion models, including Stable Diffusion, SDXL, and Flux.What is Olive and how does it work?
mainOlive (ONNX LIVE) is a model optimization toolkit and CLI designed to optimize models (typically PyTorch or Hugging Face) for the ONNX runtime. It targets specific deployment hardware (NPUs, GPUs, CPUs) from vendors like Qualcomm, AMD, Nvidia, or Intel.
Core Concepts
- Workflow: An ordered sequence of optimization tasks.
- Passes: Individual optimization tasks within a workflow, such as model compression, graph capture, quantization, and graph optimization.
- Evaluator: A component that measures metrics like accuracy and latency to determine the quality of a pass.
- Search Strategy & Sampler: Mechanisms used to auto-tune pass parameters to find the optimal balance of performance and quality.
- Deployment Target: The specific AI accelerator (NPU, GPU, CPU) for which the model is being optimized.