DOCSEARCHbyAlgolia
Submit Docs

SGLang Cookbook

website·Indexed 30 days ago

https://cookbook-sg-lang.vercel.app

Documentation and guides for deploying and benchmarking large language models and diffusion models using SGLang. Includes installation instructions, server argument references, and specific implementation guides for models such as Qwen3, GLM-4.6V, Mistral, and Wan2.2, as well as tools like SpecBundle and sglang.bench.

Tokens
101.2K
Snippets
516
Records
607
Agent score
74%

What's inside SGLang

  1. Overview of Z-Image and Z-Image-Turbo

    Z-Image is a 6B parameter image generation model family using a Scalable Single-Stream DiT (S3-DiT) architecture. Z-Image-Turbo is a distilled version optimized for efficiency, requiring only 8 NFEs (Number of Function Evaluations) to achieve high-quality, photorealistic images with bilingual text rendering (English and Chinese) and strong instruction adherence.

  2. Overview of DeepSeek-OCR

    DeepSeek-OCR is an advanced Optical Character Recognition (OCR) model developed by DeepSeek, designed for high-accuracy text extraction from images and documents. It supports various image formats and document types. The recommended base model is available on Hugging Face as deepseek-ai/DeepSeek-OCR. Usage requires agreement to the DeepSeek Community License.

  3. Overview of SGLang Diffusion Cookbook

    The SGLang Diffusion Cookbook is a collection of battle-tested recipes for image and video generation models. It covers inference serving, deployment strategies, GPU/CPU hardware configurations, and performance tuning. The documentation is organized by model series (e.g., Qwen-Image, Wan, Z-Image) within the sgl-cookbook/docs/diffusion/ directory.

  4. Overview of GLM-OCR Model

    GLM-OCR is a multimodal OCR model designed for complex document understanding, utilizing the GLM-V encoder–decoder architecture. It features a CogViT visual encoder, a lightweight cross-modal connector, and a GLM-0.5B language decoder. It is optimized for high-accuracy recognition of formulas, tables, code documents, and seals. The model is designed for efficient inference and is compatible with vLLM and SGLang.

  5. Overview of Ling-2.5-1T Model

    Ling-2.5-1T is a trillion-scale flagship instant model (1T total parameters, 63B active) designed for high throughput and efficiency. It features a hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention) and supports context lengths up to 1M tokens via YaRN. It is licensed under MIT and available in BF16 format.

  6. Overview of NVIDIA Nemotron3-Super

    NVIDIA Nemotron3-Super is a 120B parameter hybrid MoE (Mixture of Experts) model that activates only 12B parameters per forward pass. It is designed for agentic systems involving planning, reasoning, and tool use, supporting a 1M token context window. Key architectural features include a Hybrid Transformer-Mamba architecture, Multi-Token Prediction (MTP) for accelerated long-form text generation, and Thinking Budget support for optimized reasoning accuracy.

  7. Overview of Baidu ERNIE-4.5 Model Series

    ERNIE-4.5 is a family of large language models developed by Baidu, optimized for general-purpose tasks and conversational scenarios. Key technical features include:

    • MoE Architecture: Supports parameter sharing across modalities with dedicated parameters for individual modalities.
    • Vision Encoder: Uses an adaptive-resolution ViT with 2D RoPE and image packing; supports adaptive frame sampling and timestamp rendering for video.
    • Adapter: A shared modality-bridging module with spatial and temporal compression to align vision to text embedding space.
    • Multimodal Position Embedding: Employs unified 3D RoPE (temporal, height, width) for vision and 1D RoPE for text.
    • Hardware Optimization: Specifically tuned for AMD MI300X, MI325X, and MI355X GPUs.

  8. Overview of SpecBundle and EAGLE3 Speculative Decoding

    SpecBundle is an open-source initiative (supported by Ant Group, Meituan, Nex-AGI, and EigenAI) that provides high-performance, production-grade EAGLE3 draft model weights for mainstream open-source LLMs. It aims to democratize speculative decoding by addressing the lack of production-ready training infrastructure and the scarcity of high-quality, large-scale draft models.

  9. Overview of Glyph Language Model

    Glyph is a language model developed by Zhipu AI with capabilities in reasoning, function calling, and multi-modal understanding. It is optimized for high throughput and low latency and supports NVIDIA (B200, H100, H200) and AMD (MI300X, MI325X, MI355X) hardware. Available versions include BF16 (full precision) and FP8 (8-bit quantized).

  10. Overview of Qwen3.5 Model Series

    Qwen3.5 is a flagship model series by Alibaba featuring a hybrid architecture of Gated Delta Networks and sparse Mixture-of-Experts (MoE). Key capabilities include:

    • Multimodal Support: Handles text, image, and video inputs.
    • Context Window: Native support for up to 262,144 tokens, extensible beyond 1M tokens.
    • Reasoning & Tools: Includes a default 'Thinking mode' for step-by-step reasoning and built-in tool calling via the qwen3_coder parser.
    • Performance: Supports Multi-Token Prediction (MTP) for speculative decoding to reduce latency.
    • Multilingual: Supports 201 languages and dialects.
    • License: Apache 2.0.

  11. Overview of Qwen3.6-35B-A3B

    Qwen3.6-35B-A3B is an open-weight multimodal model from Alibaba featuring a Gated Delta Networks and sparse Mixture-of-Experts (MoE) architecture (35B total parameters, 3B active). It supports text, image, and video inputs with a native context length of 262,144 tokens (extensible to 1M+). Key capabilities include agentic coding, thinking preservation (retaining reasoning context), and built-in tool calling via the qwen3_coder parser.

  12. Overview of Qwen3-Coder-Next

    Qwen3-Coder-Next is a cost-efficient, code-focused language model from Alibaba. It features a hybrid architecture combining Gated DeltaNet and Gated Attention with Mixture-of-Experts (512 total experts, 10 activated, 1 shared). Key capabilities include native tool calling via the qwen3_coder parser, a context length of up to 256K tokens, and high efficiency with only 3B activated parameters per token.