DOCSEARCHbyAlgolia
Submit Docs

LMDeploy Documentation

repository·main·Indexed 27 days ago

https://github.com/internlm/lmdeploy

A high-performance toolkit for compressing, deploying, and serving Large Language Models (LLMs) and Vision Language Models (VLMs). It features optimized inference engines including TurboMind and a PyTorch engine, supports PD disaggregated deployment via LMDeploy-DistServe, and provides tools for throughput profiling, guided decoding benchmarking, and building wheel packages across Linux and Windows.

Tokens
134.9K
Snippets
348
Records
535
Agent score
91%

What's inside LMDeploy

  1. Overview of the TurboMind Inference Engine

    main

    TurboMind is a high-efficiency LLM inference engine developed based on NVIDIA's FasterTransformer. It is specifically optimized for LLaMa-style models and features:

    • Persistent Batching: Also known as continuous batching, it models conversational LLM inference as a continuous batch that lives throughout the service lifecycle. It uses pre-allocated batch slots that are immediately released upon sequence completion to accept new requests.
    • KV Cache Manager: A memory-pool-based manager with an LRU (Least Recently Used) eviction policy. It acts as a "cache for KV caches," allowing sequences to hit existing cache slots to skip context decoding. When slots are full, the manager automatically evicts the least recently used sequences, converting them into compact token IDs to be re-decoded later if needed. This process is transparent to the user.
    • Optimized LLaMa Implementation: Includes support for fast text decoding in multi-turn dialogues using a CUTLASS-based FMHA implementation, INT8 KV cache to reduce memory overhead, and support for non-contiguous KV caches in batches.

  2. Overview of LMDeploy core features

    main

    LMDeploy is an efficient and user-friendly toolkit for deploying Large Language Models (LLMs). Its core capabilities include:

    • Efficient Inference: Utilizes Persistent Batch (Continuous Batch), Blocked K/V Cache, dynamic splitting and fusion, tensor parallelism, and high-efficiency compute kernels. It claims inference performance up to 1.8x that of vLLM.
    • Reliable Quantization: Supports weight quantization and K/V quantization. 4-bit model inference can be up to 2.4x faster than FP16. Reliability is verified via OpenCompass.
    • Convenient Serving: Supports multi-model inference services across multiple machines and multiple GPUs via request dispatching services.
    • Excellent Compatibility: Supports simultaneous use of KV Cache quantization, AWQ, and Automatic Prefix Caching.

  3. Overview of LMDeploy features

    main

    LMDeploy is a toolkit for LLM compression, deployment, and serving developed by the MMRazor and MMDeploy teams. Key features include:

    • Efficient Inference: Uses persistent batching, blocked KV cache, dynamic splitting/fusion, tensor parallelism, and high-performance CUDA kernels. It can provide up to 1.8x higher request throughput than vLLM.
    • Effective Quantization: Supports weight-only and KV cache quantization. 4-bit inference performance can be up to 2.4x faster than FP16.
    • Easy Distributed Serving: Facilitates multi-model deployment across multiple machines and cards using request dispatching services.
    • High Compatibility: Supports combining KV Cache Quant, AWQ, and Automatic Prefix Caching.

  4. Understand the TurboMind Engine Async Execution Model

    main

    The TurboMind C++ engine uses an asynchronous execution model to decouple host-side scheduling from device-side execution. This is achieved through a pipeline of 'phases' where the engine thread and the model executor thread communicate via BatchData slots.

    Key Components

    • Engine Thread: Manages request admission, validation, cancellation, scheduling (via Scheduler::Schedule()), host-side module setup, and lifecycle retirement.
    • Model Executor Thread: Manages the CUDA execution context. It performs BatchOp::kPrepare, BatchOp::kForward, and BatchOp::kUnprep operations.
    • Gateway: The entry point for external requests via RequestQueue objects and the owner of the signal thread for user callbacks.
    • BatchData: A reusable carrier used to pass phase information, batch sizes, permutations, and CUDA events between the engine and executor threads.

    Request Data Path

    Request $\rightarrow$ Sequence $\rightarrow$ BatchData (and module buffers) $\rightarrow$ device/module state $\rightarrow$ BatchData $\rightarrow$ Sequence $\rightarrow$ Request outputs/signals.

  5. Run models quantized by llm-compressor

    main

    LMDeploy's TurboMind engine supports running models quantized using the llm-compressor tool. Currently, supported quantization types include int4 quantization (e.g., AWQ, GPTQ).

    Supported NVIDIA GPU architectures include:

    • Volta: V100, Jetson Xavier
    • Turing: GeForce RTX 20 series, T4
    • Ampere: A100, A800, A30, GeForce RTX 30 series, A40, A10, Jetson Orin
    • Ada Lovelace: GeForce RTX 40 series, L40, L20
    • Hopper: H20, H200, H100, GH200
    • Blackwell: GeForce RTX 50 series

  6. Understand TurboMind Engine Ownership and Lifecycle

    main

    The TurboMind engine uses a distributed ownership model to manage requests, sequences, and memory. Understanding these roles is critical for debugging execution state and resource management:

    • Gateway: Owns request queues and signal delivery. It routes incoming requests via round-robin but does not own engine-local execution state.
    • Request: Represents shared API state referenced by the gateway, engine, and callbacks. The engine updates Request::cancel_flag and Request::ec.
    • Sequence: Owned by the engine (Engine::Impl::State::rc). It holds the actual execution details.
    • Scheduler: Owned by Engine::Impl. It manages the CacheRegistry, LogicalBlockPool, PrefixTrie, and CacheBlockPool.
    • ObjectAllocator: Owns the backing cache memory region and allocation validity.
    • Callbacks: Owned outside the engine scheduling path; the gateway signal thread invokes these closures.

    Request Cleanup Invariant: A request is only ready for physical cleanup when it is both retiring and has no more inflight batches referencing it.

  7. Key features of lmdeploy.pytorch

    main

    The lmdeploy.pytorch engine provides several advanced inference features:

    • Continuous Batching: Avoids padding and unnecessary computation by concatenating all sequences in a batch into a single long sequence.
    • Tensor Parallelism: Enables running large models that exceed a single GPU's capacity by partitioning weights across multiple devices.
    • S-LoRA: Supports efficient inference with multiple LoRA adapters. Instead of merging adapters into weights (which consumes high memory), adapters are paged and swapped in using specialized kernels.
    • Quantization: Supports w8a8 quantization for low-precision computation.

  8. Understand TurboMind Architecture

    main

    TurboMind is a high-throughput inference engine for conversational LLMs based on NVIDIA's FasterTransformer. It utilizes several key architectural components to optimize performance:

    • Persistent Batch: Also known as continuous batching. It uses pre-configured batch slots that requests join when available. It automatically grows or shrinks to minimize computation and allows for instant response generation on cache-hits.
    • KV Cache Manager: A memory-pool-based manager that implements an LRU (Least Recently Used) eviction policy. It manages device memory for KV caches and automatically handles the conversion of evicted sequences into compact token IDs to allow for efficient re-decoding (cache-misses).
    • LLaMA Implementation: Optimized for conversational models with features like fast context decoding (using cutlass-based FMHA), INT8 KV cache support for increased batch size, and improved synchronization for tensor parallel mode.

  9. Use lmdeploy.pytorch as an inference backend

    main
    The lmdeploy.pytorch backend provides a highly extensible and developer-friendly implementation for Large Language Model (LLM) inference. While it may have slightly higher performance overhead compared to the performance-optimized turbomind backend, it is designed for easier development and extension. It supports advanced features like Continuous Batching, Tensor Parallelism, S-LoRA, and W8A8 quantization.

  11. Install LMDeploy from source

    main

    You can install LMDeploy directly from the GitHub repository. By default, this builds with NVIDIA CUDA support for both TurboMind and PyTorch backends. Ensure the CUDA Toolkit is installed before proceeding.

    To install the latest version:

    pip install git+https://github.com/InternLM/lmdeploy.git

    To install a specific version (e.g., v0.11.0):

    pip install https://github.com/InternLM/lmdeploy/archive/refs/tags/v0.11.0.zip

    To avoid CUDA compilation by disabling the TurboMind backend, set the DISABLE_TURBOMIND environment variable to 1 during installation.