DOCSEARCHbyAlgolia
Submit Docs

SGLang Serving Framework

repository·main·Indexed 12 days ago

https://github.com/sgl-project/sglang

A high-performance serving framework for large language models (LLMs) and vision language models (VLMs), optimized for high throughput and low latency across hardware ranging from single GPUs to large clusters. It includes support for distributed inference via sgl-model-gateway and specialized builds for AMD ROCm (amd-sglang).

Tokens
764K
Snippets
1.9K
Records
2.7K
Agent score
99%

What's inside SGLang

  1. Overview of SGLang code structure

    main

    SGLang is organized into several functional modules that separate the frontend language, the backend runtime, and evaluation utilities:

    • lang: The frontend language used for defining programs.
    • srt (SGLang Runtime): The backend engine responsible for running local models.
    • multimodal_gen: An inference framework optimized for accelerated image and video generation.
    • eval: Utilities for model evaluation.
    • api.py: The public API surface.
    • launch_server.py: The primary entry point for starting a local SGLang server.
    • bench_* scripts: Various scripts for benchmarking performance, including offline throughput, single batch latency (with and without a server), and online serving latency.
    • check_env.py: A utility to verify environment variables and dependencies.

  2. Overview of sgl-router

    main

    sgl-router is a slim, KV-aware, OpenAI-compatible router designed for SGLang workers. It serves a single model and distributes requests across a pool of workers.

    Key Features:

    • OpenAI Compatibility: Exposes /v1/chat/completions (both buffered and SSE), /v1/tokenize, /v1/detokenize, and /v1/models.
    • Observability: Provides /healthz, /readyz, and /metrics endpoints.
    • Flexible Worker Discovery: Supports worker pools via static URL lists or Kubernetes EndpointSlice discovery.
    • Disaggregated Serving: Supports prefill/decode disaggregation via specific Kubernetes selectors.

  3. Overview of Qwen3.5 Model Architecture and Features

    main

    Qwen3.5 is a flagship multimodal model series from Alibaba featuring a hybrid architecture designed for high-throughput inference and long-context processing.

    Key Architectural Features

    • Hybrid Attention: Combines Gated Delta Networks (linear, $O(n)$ complexity) with full attention layers every 4th layer to balance long-context efficiency with high associative recall.
    • Sparse MoE: Uses a Mixture-of-Experts architecture where only 17B parameters are active out of 397B total (Top-10 active out of 512 experts + 1 shared expert).
    • Native Multimodal: Uses a DeepStack Vision Transformer with Conv3d temporal encoding for integrated text, image, and video understanding.

    Capabilities

    • Context Length: Natively supports up to 262,144 tokens, extensible to over 1M tokens.
    • Reasoning: Features a 'Thinking mode' enabled by default for step-by-step reasoning.
    • Tool Calling: Built-in support using the qwen3_coder parser.
    • Multi-Token Prediction (MTP): Supports speculative decoding for reduced latency.
    • Multilingual: Supports 201 languages and dialects.

  4. Overview of SGLang Model Gateway

    main

    SGLang Model Gateway is a high-performance model-routing gateway designed for large-scale LLM deployments. It centralizes worker lifecycle management and balances traffic across multiple protocols including HTTP, gRPC, and OpenAI-compatible backends.

    Key capabilities include:

    • Unified Control Plane: Manages worker registration, monitoring, and orchestration (regular, prefill, and decode workers).
    • Multi-protocol Data Plane: Routes traffic via HTTP, gRPC, or as an OpenAI backend proxy.
    • Inference Gateway Mode (--enable-igw): Allows a single router to handle multiple models and tenants using per-model policies.
    • PD Disaggregation: Supports Prefill-Decode (PD) disaggregation for optimized caching and balancing.
    • Enterprise Privacy: Centralizes conversation history (via memory, PostgreSQL, etc.) at the router tier, preventing data leakage to upstream vendors during multi-turn or MCP (Model Context Protocol) loops.

  5. Overview of SGLang Diffusion features and interfaces

    main

    SGLang Diffusion is a high-performance framework for image and video generation, supporting models such as Wan, Hunyuan, Qwen-Image, FLUX, Z-Image, and GLM-Image. It utilizes an optimized kernel stack including sgl-kernel operators and JIT kernels.

    Available Interfaces

    • CLI Generation: Using sglang generate for direct tasks.
    • CLI Serving: Using sglang serve to host a model.
    • OpenAI-Compatible API: An HTTP server interface for sending image and video requests.

    Supported Platforms

    • NVIDIA
    • AMD
    • Intel XPU
    • Ascend
    • Apple Silicon
    • Moore Threads

  6. Overview of Go SGLang Router

    main

    Go SGLang Router is a high-performance, OpenAI-compatible API server designed to interface with an SGLang backend. It uses gRPC for communication and leverages Rust FFI (Foreign Function Interface) to perform efficient preprocessing and postprocessing tasks.

    Key capabilities include:

    • OpenAI API Compatibility: Supports the standard OpenAI Chat Completions API.
    • High Performance: Optimized for low latency and high throughput via gRPC and Rust.
    • Streaming: Supports Server-Sent Events (SSE) for streaming responses.
    • Concurrency & Safety: Features a thread-safe tokenizer and a context cancellation mechanism for graceful shutdowns to prevent resource leaks.

  7. Overview of NVIDIA Nemotron 3 Nano Omni

    main

    NVIDIA Nemotron 3 Nano Omni is a 30B-parameter hybrid Mixture-of-Experts (MoE) multimodal model that activates 3B parameters per forward pass. It is designed for unified vision, audio, and language reasoning.

    Key Specifications

    • Modalities: Input (text, image, video, audio), Output (text).
    • Context Window: 1M tokens.
    • Quantization Support: FP8 and NVFP4 (NVFP4 requires Blackwell hardware).
    • Supported GPUs: NVIDIA B200, H100, H200, A100, L40S, DGX Spark, RTX 6000.

    Model Variants (HuggingFace)

    • nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
    • nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8
    • nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4

  8. Overview of SGLang

    main

    SGLang is a high-performance serving framework designed for large language models (LLMs) and multimodal models. It is optimized for low-latency and high-throughput inference, scaling from single GPUs to large distributed clusters.

    Key capabilities include:

    • Fast Runtime: Features RadixAttention for prefix caching, zero-overhead CPU scheduling, prefill-decode disaggregation, speculative decoding, continuous batching, and various parallelism strategies (tensor, pipeline, expert, and data parallelism).
    • Broad Model Support: Supports diverse models including Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, and Mistral, as well as embedding, reward, and diffusion models.
    • Extensive Hardware Support: Compatible with NVIDIA GPUs (H100, A100, etc.), AMD GPUs (MI300, MI355), Intel Xeon CPUs, Google TPUs, and Ascend NPUs.
    • RL & Post-Training: Serves as a rollout backend for reinforcement learning (RL) and post-training frameworks like AReaL, Miles, slime, Tunix, and verl.

  9. Overview of Unified Radix Cache

    main

    The Unified Radix Cache is a component-based, pluggable prefix cache framework for SGLang. It unifies different KV cache types—specifically Full-attention, Sliding-Window-Attention (SWA), and Mamba/SSM—into a single, cohesive radix tree structure.

    Instead of maintaining separate specialized radix caches for different attention mechanisms, the Unified Radix Cache uses a single tree where each node can independently store data for multiple component types. This allows for efficient prefix sharing across different model architectures or attention modes.

  10. Overview of LFM2.5 Models

    main

    LFM2.5 is a family of hybrid models from Liquid AI designed for on-device deployment. The architecture interleaves gated short convolution blocks (for fast local mixing) with grouped query attention (GQA) blocks (for global context). This hybrid layout allows for low compute/memory costs and small KV caches even at long context lengths.

    Key Features

    • Hybrid Layout: Uses depthwise convolutions (kernel size 3) and GQA (8 KV groups, head size 64) with RoPE, QK-Norm, RMSNorm, and SwiGLU MLPs.
    • Sparse MoE (8B-A1B): Features 8.3B total parameters with 1.5B active parameters. Every layer except the first two uses a 32-expert MoE block.
    • Pythonic Tool Calling: Function calls are emitted as a Python list between <|tool_call_start|> and <|tool_call_end|> tokens. The lfm2 tool-call parser surfaces these as standard message.tool_calls.
    • Reasoning Variants: The 8B-A1B and 1.2B-Thinking checkpoints are reasoning-only models that emit an explicit <think>...</think> chain-of-thought.
    • Vision Support: LFM2.5-VL-1.6B and LFM2.5-VL-450M provide vision-language capabilities (OCR, document understanding, captioning, and object detection).

  11. Overview of WASM guest middleware examples

    main

    The sgl-model-gateway supports custom middleware implemented via the WebAssembly Component Model. The following examples demonstrate common use cases:

    wasm-guest-auth (Authentication)

    • Purpose: Validates API keys for /api and /v1 paths.
    • Logic: Checks Authorization or x-api-key headers; returns 401 Unauthorized if invalid.
    • Attach Point: OnRequest only.

    wasm-guest-logging (Tracing & Error Handling)

    • Purpose: Request tracking and status code conversion.
    • Logic: Adds tracking headers (x-request-id, x-wasm-processed, x-processed-at, x-api-route) and converts 500 errors to 503.
    • Attach Points: OnRequest and OnResponse.

    wasm-guest-ratelimit (Rate Limiting)

    • Purpose: Protects against flooding.
    • Logic: Limits requests per identifier (API Key, IP, or Request ID). Defaults to 60 requests per minute. Returns 429 Too Many Requests when exceeded.
    • Attach Point: OnRequest only.
    • Note: This example uses per-instance state. For production, use router-level rate limiting with shared state.

  12. Overview of the SGLang Plugin System

    main

    SGLang uses a plugin system to allow hardware vendors and developers to extend the framework without modifying the main repository code. The system is designed to be non-intrusive (adding OOT code paths alongside existing ones) and requires zero configuration as plugins are automatically discovered via Python setuptools entry points after installation.

    There are two primary plugin types:

    1. Hardware Platform Plugins (sglang.srt.platforms): Used to register custom hardware platforms, including device operations, KV cache pools, attention backends, graph capture, and compilation backends.
    2. General Plugins (sglang.srt.plugins): Used to inject hooks (before/after/around/replace) into functions/methods or to replace entire classes.

    Control these via environment variables:

    • SGLANG_PLATFORM: Selects or validates the active platform plugin.
    • SGLANG_PLUGINS: A comma-separated list of general plugins to load.