DOCSEARCHbyAlgolia
Submit Docs

Optimum Intel

repository·main·Indexed 20 days ago

https://github.com/huggingface/optimum-intel

An interface between Hugging Face libraries (Transformers, Diffusers, etc.) and Intel's OpenVINO toolkit. It enables users to optimize, convert, and run high-performance inference on Intel CPUs, GPUs, and accelerators. Key features include exporting models to OpenVINO IR format via optimum-cli, using OVModelForXxx classes for inference, and applying post-training static quantization with OVQuantizationConfig.

Tokens
29.1K
Snippets
80
Records
105
Agent score
70%

What's inside Optimum Intel

  2. Supported model architectures for OpenVINO export

    main
    🤗 Optimum Intel provides the exporters.openvino module to export various model architectures to the OpenVINO format. This module includes classes, functions, and a CLI for performing exports. Supported architectures span several major libraries including Transformers, Diffusers, Timm, Sentence Transformers, and specialized speech/vision models.

  3. Optimize models using OpenVINO and NNCF

    main

    🤗 Optimum Intel provides an openvino package that allows you to apply various model quantization methods to models hosted on the 🤗 Hub. This optimization leverages the NNCF (Neural Network Compression Framework) to reduce computational and memory costs during inference.

    Quantization works by representing weights and/or activations using lower precision data types, such as 8-bit or 4-bit, which speeds up inference and reduces the model's memory footprint.

  4. Configure static shapes for OpenVINO inference

    main

    Reshaping a model to use static input shapes can improve inference performance. To do this:

    1. Load the model with compile=False to allow reshaping before compilation.
    2. Use the .reshape() method to define the desired batch_size and sequence_length (or other dimensions).
    3. Call .compile() to prepare the model for inference.

    When using a pipeline with static shapes, you must also pass parameters like max_seq_len, padding="max_length", and truncation=True to ensure the input data matches the model's expected shape.

    from transformers import AutoTokenizer, pipeline
from optimum.intel import OVModelForQuestionAnswering

model = OVModelForQuestionAnswering.from_pretrained(
    "helenai/distilbert-base-uncased-distilled-squad-ov-fp32", compile=False
)
tokenizer = AutoTokenizer.from_pretrained("helenai/distilbert-base-uncased-distilled-squad-ov-fp32")

max_length = 128
model.reshape(batch_size=1, sequence_length=max_length)
model.compile()

ov_pipe = pipeline(
    "question-answering",
    model=model,
    tokenizer=tokenizer,
    max_seq_len=max_length,
    padding="max_length",
    truncation=True,
)

  5. Use Stable Diffusion pipelines with OpenVINO

    main

    Stable Diffusion models can be run using OpenVINO by using specific OVStableDiffusion pipeline classes. When exported, these models are decomposed into components: text encoder, U-NET, VAE encoder, and VAE decoder.

    Available Auto Classes:

    • text-to-image: OVStableDiffusionPipeline
    • image-to-image: OVStableDiffusionImg2ImgPipeline
    • inpaint: OVStableDiffusionInpaintPipeline

  6. Apply weight-only vs full model quantization

    main

    When exporting to OpenVINO, you can choose between two primary quantization strategies:

    1. Weight-only quantization: Quantizes only the Linear, Convolutional, and Embedding layers. This is controlled via the --weight-format flag.

      • Use --weight-format int4 for default 4-bit weight-only quantization.
      • Use --weight-format int8 for 8-bit weight-only quantization.

    2. Full model quantization: Quantizes both weights and activations. This is controlled via the --quant-mode flag.

      • Use --quant-mode int8 for 8-bit weight and activation quantization.
    # 4-bit weight-only quantization
optimum-cli export openvino --model <model_id> --weight-format int4 ov_model/

# 8-bit weight and activation quantization
optimum-cli export openvino --model <model_id> --quant-mode int8 ov_model/

  8. Rename or move documentation sections while preserving links

    main

    To prevent breaking existing links in Issues, Forums, or social media when renaming a section or moving it to a different file, add a mapping at the end of the original document. This preserves the original anchor.

    If renaming a section within the same file:

    Sections that were moved:

[ <a href="#section-b">Section A</a><a id="section-a"></a> ]

    If moving a section to a different file:

    Sections that were moved:

[ <a href="../new-file#section-b">Section A</a><a id="section-a"></a> ]

    Use relative paths to ensure versioned documentation remains functional.

  10. Perform full post-training quantization (PTQ)

    main

    Full quantization quantizes both weights and activations. This requires a calibration step using a calibration_dataset to estimate activation parameters.

    To perform full quantization:

    1. Load the model with export=True.
    2. Initialize an OVQuantizer from the model.
    3. Create a calibration_dataset (e.g., using quantizer.get_calibration_dataset()).
    4. Call quantizer.quantize() with an OVConfig containing an OVQuantizationConfig.
    from transformers import AutoTokenizer
from optimum.intel import OVQuantizer, OVModelForSequenceClassification, OVConfig, OVQuantizationConfig

model_id = "distilbert-base-uncased-finetuned-sst-2-english"
model = OVModelForSequenceClassification.from_pretrained(model_id, export=True)
quantizer = OVQuantizer.from_pretrained(model)

# Prepare calibration data (example using glue/sst2)
calibration_dataset = quantizer.get_calibration_dataset(
    "glue",
    dataset_config_name="sst2",
    preprocess_function=preprocess_function, # user defined
    num_samples=300,
    dataset_split="train",
)

# Apply full quantization
save_dir = "ptq_model"
ov_config = OVConfig(quantization_config=OVQuantizationConfig())
quantizer.quantize(ov_config=ov_config, calibration_dataset=calibration_dataset, save_directory=save_dir)

  11. Quantize Stable Diffusion models with OpenVINO NNCF

    main

    For generative tasks like Stable Diffusion, you can apply post-training hybrid quantization using NNCF. This notebook provides specific instructions for optimizing Stable Diffusion models to run efficiently on Intel hardware.

    https://github.com/huggingface/optimum-intel/blob/main/notebooks/openvino/stable_diffusion_hybrid_quantization.ipynb

  12. Refine SDXL images using a Refiner model

    main

    You can refine the output of an SDXL base model by using an SDXL Refiner model. To do this, set output_type="latent" in the base model to obtain the latents, then pass those latents as the image input to the refiner pipeline.

    from optimum.intel import OVStableDiffusionXLImg2ImgPipeline

model_id = "stabilityai/stable-diffusion-xl-refiner-1.0"
refiner = OVStableDiffusionXLImg2ImgPipeline.from_pretrained(model_id, export=True)

# Get latents from base model
image = base(prompt=prompt, output_type="latent").images[0]
# Pass latents to refiner
image = refiner(prompt=prompt, image=image[None, :]).images[0]