DOCSEARCHbyAlgolia
Submit Docs

Intel LLM Scaler

repository·main·Indexed 16 days ago

https://github.com/intel/llm-scaler

A GenAI optimization solution for Intel Arc Pro B60 and B70 GPUs. It provides optimized container images and environments for frameworks like vLLM and ComfyUI. The suite includes ComfyUI-OmniXPU for transparent acceleration and omni_xpu_kernel, which provides high-performance Intel XPU kernels for PyTorch, featuring ESIMD Flash Attention, CUTLASS-SYCL fused Flash Attention, and FP8 GEMM via oneDNN.

Tokens
55.5K
Snippets
156
Records
210
Agent score
65%

What's inside intel-llm-scaler

  1. Introduction to ComfyUI in Omni

    main

    ComfyUI is a node-based graphical user interface (GUI) used to build and execute generative AI workflows. In the context of the Omni project, it provides:

    • Modular Design: Create complex pipelines by connecting different functional nodes.
    • Multi-Model Support: Capability to run image, video, 3D, and audio generation models.
    • Flexible Extension: Support for custom nodes to add new features.
    • Intel XPU Support: This implementation is specifically optimized for Intel GPUs.

  2. Overview of SGLang Diffusion

    main

    SGLang Diffusion is a high-performance inference engine module optimized for Intel XPU, designed for image and video generation tasks. It provides an OpenAI-compatible API and supports several advanced features for efficient inference:

    • OpenAI-compatible API: Uses standard /v1/images/generations and /v1/videos/generations endpoints.
    • Multi-GPU support: Implements tensor parallelism and sequence parallelism (Ulysses).
    • Intel XPU optimization: Patched with XCCL communication and XPU-specific kernels.
    • Memory Optimization: Supports VAE CPU offloading to reduce GPU memory usage and TeaCache acceleration to skip redundant computations.
    • LoRA support: Allows dynamic loading and unloading of LoRA adapters via API.

  3. Overview of LLM Scaler

    main

    LLM Scaler is a Generative AI (GenAI) solution designed for text, image, and video generation. It is optimized to run on Intel® Arc™ Pro B60 and B70 GPUs.

    To ensure high performance for state-of-the-art GenAI models on these specific hardware targets, LLM Scaler leverages standard industry frameworks, including:

    • vLLM
    • ComfyUI
    • SGLang Diffusion
    • Xinference

  4. What is ComfyUI?

    main

    ComfyUI is a node-based graphical user interface (GUI) used to build and execute workflows for generative AI models like Stable Diffusion.

    Key features include:

    • Modular Design: Build complex generation pipelines by connecting nodes.
    • Multi-Model Support: Supports image, video, 3D, and audio generation models.
    • Flexible Extension: Expand functionality via custom nodes.
    • Intel XPU Support: This project is specifically optimized for Intel GPUs.

  5. Overview of llm-scaler-vllm offerings

    main

    llm-scaler-vllm is an optimized version of vLLM tailored for Intel's Multi-GPU platforms. It provides two primary ways to set up your environment depending on your goal:

    1. Platform Evaluation: Used to evaluate hardware capabilities (GPU memory bandwidth, P2P/collective communication, GeMM compute) without running vLLM inference.

      • Steps: Install Ubuntu 24.04 $\rightarrow$ Run Offline Installer $\rightarrow$ Run platform evaluation scripts.

    2. vLLM Inference Benchmark: Used to run inference performance tests using vLLM/IPEX.

      • Steps: Install Ubuntu 24.04 $\rightarrow$ Run Offline Installer $\rightarrow$ Pull vLLM Docker image $\rightarrow$ Download target model $\rightarrow$ Run inference performance tests.

    Note: Both the offline installer and Docker images are intended for demo purposes only and are not recommended for production. For production, use the provided Dockerfiles to build your own images.

  6. Overview of LLM Scaler vLLM features

    main

    The llm-scaler-vllm component enables running text generation models using the vLLM framework with several advanced capabilities:

    • Parallelism & Communication: Supports CCL (P2P or USM), Tensor Parallel, Pipeline Parallel, and Data Parallel.
    • Quantization: Supports INT4 and FP8 quantized online serving, as well as pre-quantized FP8 model support.
    • Model Types: Supports Embedding, Reranker, Multi-Modal, and Omni models.
    • Utilities: Includes finding maximum Context Length, a Multi-Modal WebUI, and the BPE-Qwen tokenizer.

    For detailed setup and usage, refer to the Getting Started guide.

  7. HY-WorldPlay Features and Optimizations

    main

    HY-WorldPlay (Hunyuan Video World Simulator) on Intel Arc GPU includes several optimizations for high-performance video generation:

    • Full Intel Arc GPU (XPU) support: Utilizes the PyTorch XPU backend.
    • Multi-GPU sequence parallelism: Supported via XCCL.
    • Memory-efficient chunked attention: Designed for large video generation tasks.
    • Optimized VAE decoding: Implements tile parallelism for efficient decoding.

  8. Use LLM Scaler Omni for multimodal generation

    main

    LLM Scaler Omni (llm-scaler-omni) is an experimental feature that supports image, voice, and video generation. It operates in two primary modes:

    1. Omni Studio: A WebUI interaction mode using ComfyUI for tasks like Image/Video/Audio/3D generation and editing.
    2. Omni Serving: An OpenAI-API compatible serving mode for programmatic access to generation endpoints.

    To begin using Omni, refer to the Getting Started guide for instructions on using the Omni Docker image.

  9. Enable Multi-Token Prediction (MTP)

    main

    MTP is a speculative decoding method where the target model includes native multi-token prediction capability. This allows for model-based speculative decoding without a separate draft model.

    Supported Models: Qwen3.6-27B, Qwen3.6-35B-A3B, gemma-4-26B-A4B-it, and gemma-4-31B-it.

    Implementation:

    • For Qwen: Use --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'.
    • For Gemma 4: Use --speculative-config '{"method":"gemma4_mtp","model":"/path/to/gemma-4-31B-it-assistant","num_speculative_tokens":2}'. Note that you must download the assistant checkpoint first.
    # Qwen MTP example
vllm serve ... --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'

# Gemma 4 MTP example
vllm serve ... --speculative-config '{"method":"gemma4_mtp","model":"/path/to/gemma-4-31B-it-assistant","num_speculative_tokens":2}'

  10. Important constraints for Ubuntu environment and updates

    main

    When setting up or using the LLM Scaler platform, observe the following constraints:

    • OS Support: Ubuntu 24.04 LTS is not yet supported (targeting late 2025).
    • Kernel/Driver Updates: During the evaluation phase, do not update the kernel or system packages. Maintaining the validated environment is critical for stability and compatibility with pre-installed components.
    • PCIe Link Status: Seeing 2.5GT/s in lspci is a known issue that does not impact GPU operation.

  11. Understand vLLM service performance metrics

    main

    The monitoring setup provides several key metrics essential for evaluating LLM service performance and resource utilization.

    Latency Metrics

    • End-to-End (E2E) Request Latency: Total time from request submission to complete response receipt. High latency impacts user experience.
    • Time to First Token (TTFT): Duration from request initiation to the first output token. Critical for streaming responsiveness. High P99 TTFT suggests cold starts or scheduling bottlenecks.
    • Inter-Token Latency (Time Per Output Token Latency): Time between consecutive tokens. High P99 intervals may indicate memory bandwidth constraints or resource contention.
    • Prefill & Decode Time:
      • Prefill: Time spent processing input prompts. Long prefill suggests a need for prompt encoding optimization.
      • Decode: Time spent generating per-token outputs. Long decode suggests memory bandwidth limitations.
    • Queue Time: Time a request spends waiting in scheduling queues. High queue time indicates a need for horizontal scaling.

    Throughput and Efficiency

    • Token Throughput: Total tokens processed per second (input + output). Used for hardware planning.
    • Cache Utilization: Percentage of KV Cache memory used. Utilization >90% may trigger cache eviction and increase latency.

    Request and Scheduler Analysis

    • Scheduler State: Distribution of request statuses:
      • RUNNING: Actively computing.
      • WAITING: Queued for resources.
      • SWAPPED: Swapped out of memory (e.g., due to VRAM exhaustion).
    • Finish Reason: Distribution of why generation ended:
      • EOS Token: Natural termination.
      • Max Sequence Length: Hit the generation limit.
    • Heatmaps:
      • Request Prompt Length Heatmap: Correlates input prompt lengths with performance to optimize context window management.
      • Request Generation Length Heatmap: Analyzes how output token length impacts latency.

  12. Accelerate diffusion models with Cache-DiT and torch.compile

    main

    You can significantly speed up inference by inserting acceleration nodes between the model loader and the sampler.

    • Cache-DiT only: Add the ⚡ CacheDit Accelerator node after the model loader. Best for high step-count workflows (≥ 8 steps).
    • torch.compile only: Add the TorchCompileModel node after the model loader. Note: This is only supported in the intel/llm-scaler-omni Linux Docker image and has a one-time warm-up cost.
    • Combined: Chain TorchCompileModel after ⚡ CacheDit Accelerator for maximum speedup (up to ~2.2x on Z-Image-Turbo).

    Supported Models for Cache-DiT:

    • Image: Z-Image, Z-Image-Turbo, Qwen-Image-2512, Flux.2 Klein 4B / 9B
    • Video: LTX-2 T2V / I2V, Wan2.2 14B T2V / I2V