DOCSEARCHbyAlgolia
Submit Docs

IPEX-LLM Documentation

repository·main·Indexed 27 days ago

https://github.com/intel/ipex-llm

An acceleration library for Large Language Models (LLMs) optimized for Intel hardware, including CPUs, GPUs (Arc, Flex, Max), and NPUs. It enables high-performance inference and fine-tuning through integration with HuggingFace, llama.cpp, and vLLM. The library supports LoRA and QLoRA fine-tuning using mixed-precision (BF16) and distributed training via Intel OneCCL and Intel MPI, with deployment options for standalone Docker containers and Kubernetes using the Kubeflow MPI Operator.

Tokens
220.4K
Snippets
826
Records
1.2K
Agent score
93%

What's inside IPEX-LLM

  1. Overview of IPEX-LLM CPU Examples

    main

    The python/llm/example/CPU directory provides several implementation examples for running LLMs on Intel CPUs:

    • HF-Transformers-AutoModels: Running Hugging Face Transformers models using standard AutoModel APIs.
    • QLoRA-FineTuning: Performing QLoRA fine-tuning on Intel CPUs.
    • vLLM-Serving: Using the vLLM serving framework with IPEX-LLM low-bit optimized models.
    • Deepspeed-AutoTP: Distributed inference using DeepSpeed AutoTP with IPEX-LLM low-bit optimized models.
    • LangChain: Integrating IPEX-LLM into LangChain applications.
    • Applications: Running LLM applications (e.g., agents, streaming-llm) on BigDl-LLM.
    • PyTorch-Models: Running any PyTorch model with a minimal code change.
    • Native-Models: Converting and running llama/chatglm/bloom/gptneox/starcoder model families using native C++ implementations.
    • Speculative-Decoding: Running Hugging Face Transformers models with self-speculative decoding.
    • ModelScope-Models: Running ModelScope models on Intel CPUs.
    • StableDiffusion-Models: Running Stable Diffusion models on Intel CPUs.

  2. Overview of Self-Speculative Decoding for BF16 Inference

    main

    IPEX-LLM supports self-speculative decoding for BF16 inference of any Huggingface Transformer model on Intel CPUs. This technique allows for faster inference by using a smaller version of the model (or the model itself) to speculatively predict tokens.

    This directory provides dedicated folders for popular open-source models, each containing specific instructions for installation and execution.

  3. Overview of Intel® LLM Library for PyTorch (ipex-llm)

    main

    ipex-llm is an XPU acceleration library designed to run Large Language Models (LLMs) efficiently on Intel hardware, including:

    • Intel GPUs: Integrated GPUs (iGPU), Arc discrete GPUs, Flex Series, and Max Series data center GPUs.
    • Intel NPUs: Support for Intel Core Ultra NPUs (including 100H, 200V, 200K, and 200H series).
    • Intel CPUs.

    Key features include:

    • Algorithm Optimization: Advanced optimizations for LLM algorithms.
    • XPU Acceleration: High-performance execution on Intel hardware.
    • Low-bit Quantization Support: Support for FP8, FP6, FP4, and INT4 data types.
    • Seamless Integration: Works with popular frameworks and tools such as llama.cpp, Ollama, HuggingFace transformers, LangChain, LlamaIndex, vLLM, Text-Generation-WebUI, DeepSpeed-AutoTP, FastChat, Axolotl, HuggingFace PEFT, HuggingFace TRL, AutoGen, and ModelScope.

  4. Overview of IPEX-LLM

    main

    IPEX-LLM is an LLM acceleration library designed for Intel hardware, including:

    • Intel GPU (e.g., iGPU, Arc, Flex, and Max discrete GPUs)
    • Intel NPU (Neural Processing Unit)
    • Intel CPU

    It provides seamless integration with various LLM frameworks and tools such as llama.cpp, Ollama, vLLM, HuggingFace transformers, LangChain, LlamaIndex, Text-Generation-WebUI, DeepSpeed-AutoTP, FastChat, Axolotl, HuggingFace PEFT, HuggingFace TRL, and AutoGen. It supports over 70 optimized models (e.g., Llama, Phi, Mistral, Mixtral, DeepSeek, Qwen) with state-of-the-art optimizations, XPU acceleration, and low-bit quantization support (FP8, FP6, FP4, INT4).

  5. LLM Finetuning modes with IPEX-LLM on Intel GPU

    main

    IPEX-LLM supports various LLM finetuning modes on Intel GPUs. Available modes include:

    • LoRA: Low-Rank Adaptation finetuning.
    • QLoRA: Quantized LoRA finetuning.
    • QA-LoRA: Quantized and Adaptive LoRA finetuning.
    • ReLora: Re-parameterized LoRA finetuning.
    • DPO: Direct Preference Optimization finetuning.

    You can execute these using specialized code examples or via existing frameworks like Hugging Face PEFT (without code modification) or axolotl (without writing code).

  6. Run Distributed QLoRA Fine-Tuning on Kubernetes with OneCCL

    main

    IPEX-LLM provides CPU optimizations to accelerate QLoRA fine-tuning for models like Llama2-7b using mixed-precision and distributed training. It leverages Intel OneCCL (Hugging Face backend) for BF16 computation speedups on CPUs and Intel MPI for parallel processing on Kubernetes. Advanced quantization allows for large-scale fine-tuning by using NF4 model storage with BF16 computing types to improve memory utilization.

    The deployment uses the Kubeflow MPI Operator to manage MPIJob CRDs, handling worker cluster construction, public key distribution, SSH connections, and log collection.

  8. Run BF16-Optimized Lora Finetuning on Kubernetes with OneCCL

    main
    IPEX-LLM provides CPU optimization to accelerate LoRA (Low-Rank Adaptation) finetuning of Llama2-7b using mixed-precision and distributed training. It leverages Intel OneCCL for BF16 datatype acceleration on CPUs and Intel MPI for parallel processing on Kubernetes. The deployment is managed via the Kubeflow MPI Operator using MPIJob CRDs to handle worker cluster construction, SSH connections, and log collection.

  11. Run 4-bit AWQ models on Intel CPU

    main

    IPEX-LLM allows you to directly run 4-bit AWQ (Activation-aware Weight Quantization) models on Intel CPUs using INT4 optimizations. This process converts linear layers into INT4 format during loading.

    Memory Estimation: A model with XB parameters saved in 16-bit typically requires approximately 2X GB of memory for loading and ~0.5X GB for inference.

  12. Run Ollama with IPEX-LLM on Intel GPU

    main

    You can use the C++ interface of ipex-llm as an accelerated backend for ollama to run language models on Intel GPUs (including iGPU, Arc, Flex, and Max series).

    Important Notes:

    • Portable Option: Use Ollama Portable Zip to run Ollama on Intel GPU with ipex-llm without manual installation.
    • Intel Arc B-Series: For Intel Arc B-Series GPUs (e.g., B580), follow the specific BMG guide.
    • Windows Dependency Update: Starting from ipex-llm[cpp]==2.2.0b20250207, the oneAPI dependency on Windows switches from 2024.2.1 to 2025.0.1. To avoid the Can't find sycl8.dll error, you must create a new conda environment instead of upgrading an existing one.