DOCSEARCHbyAlgolia
Submit Docs

LightLLM Documentation

repository·main·Indexed 26 days ago

https://github.com/modeltc/lightllm

A high-performance, lightweight Python-based inference and serving framework for Large Language Models (LLMs). LightLLM is optimized for speed and scalability using FlashAttention and KV cache management. It supports the deployment of models such as GLM-4.7-Flash and Qwen3.5-397B-A17B, featuring capabilities like Multi-Token Prediction (MTP) speculative decoding, XML-style function calling, and multimodal support for text, image, and video.

Tokens
59.5K
Snippets
142
Records
254
Agent score
87%

What's inside LightLLM

  1. Overview of Lightllm

    main

    Lightllm is a lightweight, high-performance large language model (LLM) inference and serving framework developed in pure Python. It integrates advantages from various open-source solutions including FasterTransformer, TGI, vLLM, SGLang, and FlashAttention.

    Key Features:

    • Multi-process Collaboration: Asynchronous execution of input text encoding, LLM inference, vision model inference, and output decoding to maximize GPU utilization.
    • Cross-process Request Object Sharing: Uses shared memory to share request objects across processes, reducing inter-process communication (IPC) latency.
    • Efficient Scheduling Strategy: A peak memory scheduling strategy with prediction that maximizes GPU memory utilization while minimizing request eviction.
    • High-Performance Inference Backend: Features efficient operator implementations, support for multiple parallelism modes (Tensor Parallelism, Data Parallelism, and Expert Parallelism), dynamic KV cache, extensive quantization support (int8, fp8, int4), structured output, and multi-result prediction.

  2. Understand the Efficient Router mechanism

    main

    The Efficient Router manages incoming requests by dynamically determining if they can be merged into existing inference batches. A request is eligible for merging if the estimated maximum token usage during the merged inference does not exceed the hardware's maximum capacity, defined by the parameter max_total_token_num.

    To prevent Out-of-Memory (OOM) situations, the router uses Token Attention to accurately track token usage. The router calculates the potential token usage at specific time points (based on the remaining output lengths and historical KV cache tokens) to ensure the total usage stays below max_total_token_num.

  3. Understand TokenAttention memory management mechanism

    main

    TokenAttention uses a token-level management strategy to optimize memory usage:

    1. Initialization: Pre-allocates KV cache based on max_total_token_num and initializes a Token Table.
    2. Allocation: When a new request arrives, the system attempts to allocate contiguous memory space to minimize memory access latency. If contiguous space is insufficient, it allocates non-contiguous memory. All allocations are recorded in the Token Table.
    3. Generation: For newly generated tokens, the system finds unused space in the pre-allocated cache and adds the entry to the Token Table. This process is performed on the GPU using Torch Tensor parallel computing for high efficiency.
    4. Release: Once a request is completed, memory is released by deleting the corresponding records in the Token Table.
    5. Optimization: By combining token-level management with an Efficient Router, the system can continuously add new requests to maximize GPU utilization.

  4. Understand Lightllm Architecture

    main

    Lightllm uses a multi-process collaboration design where modules communicate via zmq and rpc. The system is composed of several specialized servers and backends:

    • Http Server: Receives API requests. It handles system queries (via Metric Server and Health Server), plain text requests (tokenization and routing), and multimodal requests (hashing, caching, and routing to Visual Server).
    • Metric Server: Records system performance indicators.
    • Health Server: Monitors system health.
    • Router: Schedules requests. It manages the request queue and decides between prefill and decode rounds.
    • Visual Server: Handles multimodal request encoding.
    • Cache Manager Server: Manages the cache for multimodal inference results (original image data and encoded features) using host shared memory to reduce redundant memory reads.
    • Model Backend: Manages model inference for a single device. Multiple backends can exist simultaneously.

  5. Understand TokenAttention memory management

    main

    TokenAttention manages key and value (KV) cache at the token level rather than the block level (as seen in PagedAttention).

    Key behaviors:

    • Contiguous Allocation: The system attempts to allocate contiguous GPU memory for requests to minimize memory access latency. Non-contiguous memory is only used when contiguous space is insufficient.
    • Zero Waste: Because management is performed token-by-token, it achieves near-zero memory waste and higher throughput compared to vLLM.
    • Efficient Deallocation: When a request completes, memory is released by deleting records in the Token Table and updating the memory state on the GPU, allowing for immediate reuse by new requests.

  6. Understand Http Server Request Handling

    main

    The Http Server acts as the entry point for the system and performs the following tasks based on request type:

    • System Query Requests: Collaborates with Metric Server and Health Server to retrieve system information.
    • Pure Text Requests: Tokenizes the text and wraps it into a pure text request to be sent to the Router.
    • Multi-modal Requests:
      1. Calculates the MD5 hash of the image data.
      2. Requests a cache from the Cache Manager Server using the MD5 hash.
      3. Stores the image data in the cache.
      4. Tokenizes the text.
      5. Wraps the tokenized text and multi-modal information into a multi-modal request to be sent to the Visual Server.

  7. Understand Router Scheduling

    main

    The Router receives requests from the Http Server or Visual Server and manages request scheduling. Its primary responsibilities are:

    • Maintaining a request queue.
    • Deciding whether the current iteration should be a prefill phase or a decode phase.
    • Selecting which specific requests to process during a prefill phase.
    • Selecting which specific requests to process during a decode phase.

  8. Understand the LightLLM Inference Architecture

    main

    To add a new model to LightLLM, you must implement components across four main areas located under lightllm/common/basemodel:

    1. Weights: Implement weight loading and Tensor Parallel (TP) splitting.
    2. Inference: Implement the actual forward pass logic for different layer types.
    3. State: Manage information passed between layers during inference.
    4. Model Framework: The entry point that orchestrates weights, inference, and state classes.

  9. Understand Visual Server and Cache Manager Server

    main

    These modules are specifically designed to support multi-modal model inference:

    • Visual Server: Responsible for encoding image information for multi-modal models.
    • Cache Manager Server: Responsible for caching both the raw image data and the encoded image features. This cache is stored in the host's shared memory to reduce redundant memory reads across multiple processes and avoid re-encoding the same image data.

  10. Understand the LightLLM Architecture

    main

    LightLLM is designed around multi-process collaboration, where each process manages a specific module and communicates via zmq and rpc.

    Key modules include:

    • Http Server: Receives API requests.
    • Metric Server: Records system performance metrics.
    • Health Server: Monitors system health.
    • Router: Schedules requests.
    • Visual Server: Handles multi-modal requests.
    • Cache Manager Server: Manages inference result caches for multi-modal information.
    • Model Backend: Manages model inference on individual devices (multiple backends can exist).

  11. Configure Environment Variables for Qwen3-8B PD Disaggregation

    main

    Before starting the api_server processes for PD (Prefill/Decode) disaggregation, export the following environment variables to define paths, networking, and device allocation.

    Key Variables:

    • LOG_DIR: Root directory for all logs and summary.txt.
    • MODEL_DIR: Path to the Qwen3-8B model (must match the tokenizer path in lm_eval).
    • PD_MASTER_IP: The IP address the pd_master binds to; used as the base_url for lm_eval.
    • HOST: The binding IP for prefill and decode nodes. If co-located, set export HOST="${PD_MASTER_IP}".
    • PREFILL_CUDA_DEVICES: Two physical GPU indices for the prefill node.
    • DECODE_CUDA_DEVICES: Two physical GPU indices for the decode node (must not overlap with prefill).
    • UCX_NET_DEVICES: HCA list for RDMA (e.g., mlx5_0:1,mlx5_1:1), determined by ibv_devinfo.
    • UCX_TLS: Transport layer settings, typically rc,cuda,gdr_copy.

  12. Expose the Anthropic Messages API endpoint

    main

    LightLLM can host an experimental /v1/messages endpoint that implements the Anthropic Messages API wire protocol. This allows you to use existing Anthropic Python or TypeScript SDKs with locally hosted open-source models. The endpoint is enabled by default when running the API server.

    python -m lightllm.server.api_server \
    --model_dir /path/to/model \
    --port 8088