DOCSEARCHbyAlgolia
Submit Docs

Intel Extension for PyTorch (IPEX)

repository·main·Indexed 24 days ago

https://github.com/intel/intel-extension-for-pytorch

Intel Extension for PyTorch (IPEX) provides performance optimizations for PyTorch on Intel CPUs and GPUs using hardware-specific instructions like AVX-512, AMX, and XMX. It features the ipex.llm module for Large Language Model acceleration, INT8 quantization via the oneDNN graph API, and ipex.fast_bert for BERT workload acceleration. The library supports C++ inference through libintel-ext-pt and provides specialized kernels via Tensor Processing Primitives (TPP).

Tokens
75.6K
Snippets
173
Records
333
Agent score
84%

What's inside Intel Extension for PyTorch

  1. Overview of TPP related tools and fused kernels

    main
    The csrc/cpu/tpp directory contains tools and fused kernel implementations optimized via TPP (Tensor Processing Primitives). It provides specialized kernels for common deep learning operations, particularly for BERT models, and includes infrastructure for loop generation, tuning, and JIT (Just-In-Time) compilation to optimize performance on Intel CPUs.

  2. Overview of IPEX CPU Jupyter Notebook samples

    main

    The following Jupyter Notebooks are available for exploring Intel® Extension for PyTorch (IPEX) optimizations on CPU:

    • Getting Started with IPEX: Demonstrates running a PyTorch inference workload on CPU using oneAPI AI Analytics Toolkit and analyzing CPU usage via oneDNN verbose logs. (15 mins)
    • PyTorch Inference Optimizations with Intel® AMX (BF16/INT8): Demonstrates ResNet50 and BERT inference using Intel® Advanced Matrix Extensions (AMX) with bfloat16 and integer8 precision. (5 mins)
    • Interactive Chat (DialoGPT) with IPEX Quantization: Shows how to add INT8 dynamic quantization to a HuggingFace DialoGPT model for interactive chat. (10 mins)
    • Optimize PyTorch Models using IPEX Quantization: Demonstrates calibrating a ResNet50 model with the CIFAR10 dataset using INT8 quantization. (5 mins)
    • Optimize PyTorch Models using IPEX: A comprehensive guide applying sequential optimizations to Faster-RCNN and DistilBERT models to examine incremental performance gains. (30 mins)

  3. Overview of Intel Extension for PyTorch

    main

    Intel® Extension for PyTorch* extends PyTorch* with optimizations for Intel hardware to provide an extra performance boost.

    Supported Hardware Optimizations:

    • Intel CPUs: Leverages Intel® Advanced Vector Extensions 512 (Intel® AVX-512), Vector Neural Network Instructions (VNNI), and Intel® Advanced Matrix Extensions (Intel® AMX).
    • Intel Discrete GPUs: Leverages Intel X<sup>e</sup> Matrix Extensions (XMX) AI engines and provides acceleration via the PyTorch xpu device.

    Quick Links:

  4. Overview of PyTorch JIT passes for DNNL

    main

    The csrc/cpu/jit directory contains experimental PyTorch JIT (Just-In-Time) passes designed to optimize PyTorch Graphs. These passes aim to leverage the full capabilities of DNNL (Deep Neural Network Library) and other potential optimization backends.

    Note that these passes are currently experimental and are implemented within the pytorch namespace to facilitate future migration into the main PyTorch repository. The graph manipulation logic is designed to be independent of other modules within the Intel Extension for PyTorch to ensure modularity.

  5. Overview of TorchRec Inference Library

    main
    TorchRec Inference is an experimental C++ library designed for multi-GPU inference. It uses torch.package to shard models defined in Python and leverages torch.deploy to serve inference from C++. By launching multiple Python interpreters to carry the packaged model, it effectively subverts the Global Interpreter Lock (GIL) for high-performance serving.

  6. Important: Intel® Extension for PyTorch Retirement Plan

    main

    Intel® Extension for PyTorch is being retired. Most features and optimizations have been upstreamed into official PyTorch.

    Key status updates:

    • Recommendation: Users are strongly encouraged to use PyTorch directly for Intel® CPU and GPU platforms.
    • Development: Active development has discontinued and official quarterly releases have ceased following the 2.8 release.
    • Maintenance: Critical bug fixes and security patches will be provided in the main branches (CPU and GPU/XPU) for two additional quarters.
    • End-of-Life: The project is planned to reach End-of-Life after March 2026 to allow dependencies to be removed.

  8. ipex.llm Optimized Model List for Inference

    main

    ipex.llm supports a wide range of open-source LLMs, including the Llama, Qwen, Phi, and DeepSeek series. Models are verified for two primary modes: Single Instance Mode and Distributed Inference Mode via DeepSpeed.

    Optimized models benefit from features like indirect access KV cache, fused ROPE, and customized linear kernels. Verification status varies by data type (FP32, BF16, INT8, or INT4 quantization).

  10. Important: Intel Extension for PyTorch is Archived

    main

    Project Status: ARCHIVED

    Intel will not provide or guarantee development of or support for this project. This includes maintenance, bug fixes, new releases, or updates. Patches are no longer being accepted.

    Retirement Details:

    • Official Releases: Official quarterly releases and binary wheels have ceased following the 2.8 release.
    • Maintenance: Critical bug fixes and security patches are only provided in the main branches (main for CPU and xpu-main for GPU) for a limited time.
    • End-of-Life: The project is planned to reach End-of-Life after March 2026.
    • Recommendation: Intel strongly recommends using PyTorch directly for Intel CPU and GPU platforms, as most optimizations have been upstreamed into official PyTorch.

  11. Overview of ipex.llm module and functional optimizations

    main

    Intel® Extension for PyTorch (IPEX) provides ipex.llm to accelerate Large Language Models (LLMs) through specialized module and functional level APIs. These optimizations include linear post-op fusions, attention-related fusions (like PagedAttention and RoPE), and generation-related fusions for HuggingFace APIs.

    To use these, import the extension:

    import intel_extension_for_pytorch as ipex

  12. Overview of TorchRec Inference

    main
    TorchRec Inference is a C++ library designed for multi-GPU inference. It works by sharding models that were originally written and packaged in Python using torch.package. To serve these models from C++ while bypassing the Global Interpreter Lock (GIL), the library utilizes torch.deploy, which launches multiple Python interpreters to carry the packaged models.