DOCSEARCHbyAlgolia
Submit Docs

Dynamo Documentation

repository·main·Indexed 27 days ago

https://github.com/ai-dynamo/dynamo

An open-source orchestration layer for datacenter-scale LLM inference that coordinates engines like SGLang, vLLM, and TensorRT-LLM. It provides disaggregated serving, KV-aware routing, and SLA-driven scaling. The repository includes aisimulate (v0.1.0) for experimental replay-backed deployment configuration search via Spica, a BurstGPT load generator converter for transforming CSV logs to mooncake-style JSONL, and a comprehensive benchmarking suite for frontend performance analysis using aiperf and eBPF tracing.

Tokens
316.3K
Snippets
698
Records
1.5K
Agent score
92%

What's inside Dynamo

  1. Overview of Global Planner

    main

    The GlobalPlanner is a centralized scaling execution service designed for multi-DGD (DynamoGraphDeployment) planner deployments. It acts as a common scale-execution endpoint that receives scaling decisions from multiple local planners and executes replica updates against Kubernetes DynamoGraphDeployment resources.

    Key Use Cases

    • Centralized Scaling Policies: Apply consistent scaling logic across multiple independent DGDs.
    • Shared Constraints: Enforce global limits such as total GPU budgets or authorization requirements.
    • Hierarchical Deployments: Coordinate scaling for single-endpoint, multi-pool deployments where multiple pools sit behind a GlobalRouter.

  2. Overview of DynoSim components and entry points

    main

    DynoSim is a simulation stack used to explore serving configurations before deploying to real GPU clusters.

    Component Summary

    ComponentEntry PointRole
    DynoSim runpython -m dynamo.replayRuns one workload against one simulated configuration and emits metrics/reports
    DynoSim sweepdynamo.profiler.utils.replay_optimizeSweeps trials across TP shape, worker split, router knobs, SLA, and GPU budget
    Live simulation (Mocker)python -m dynamo.mockerRuns simulated workers in a live Dynamo deployment path
    Mocker corelib/mockerModels engine scheduling, KV allocation, prefix caching, preemption, and timing
    AICAI Configurator SDKSupplies calibrated timing and candidate-shape data
    Planner simulation--planner-configRuns Planner decisions in the simulation loop

    Choosing an Entry Point

    • Single workload/config: Use python -m dynamo.replay.
    • Sweep topology/router choices: Use dynamo.profiler.utils.replay_optimize.
    • Exercise Kubernetes frontend/router (no GPUs): Use python -m dynamo.mocker.
    • Exercise Local frontend/router (no GPUs): Use python -m dynamo.mocker (refer to mocker-local.mdx for local specifics).
    • Benchmark Planner decisions: Use --planner-config with DynoSim runs.

  3. Overview of the Offline Replay Harness

    main

    The Offline Replay Harness (dynamo_mocker::replay) simulates trace execution without requiring async runtimes, network planes, or real worker tasks. It uses a logical clock to step mock engine cores directly and records request/token timing into a TraceCollector.

    There are three primary implementation paths chosen in lib/mocker/src/replay/offline/mod.rs:

    • Single-Worker Fast Path: Used when num_workers == 1 and dp_size == 1. It avoids cluster event queues and router machinery.
    • Aggregated Multi-Worker Harness: A general cluster simulator for multi-worker and kv_router replay.
    • Disaggregated Replay: A two-stage harness for separate prefill and decode pools.

  4. Overview of Dynamo Feature Benchmarks

    main
    Dynamo Feature Benchmarks evaluate specific features, topologies, and feature stacks under controlled traffic conditions. These benchmarks compare different deployable configurations to answer performance questions. Each benchmark provides details on the question being asked, the configurations compared, how to reproduce the run, and links to specific Recipes for direct deployment.

  5. Overview of Dynamo Rust Sidecars

    main
    Rust sidecars in Dynamo act as a bridge between Dynamo workers and inference engines. They connect workers to inference engines using the engine's native gRPC APIs. In this architecture, Dynamo manages worker registration and request handling, while the inference engine runs as a separate process.

  6. Overview of the Dynamo Profiler

    main
    The Dynamo Profiler analyzes model inference performance to generate optimized DynamoGraphDeployments (DGD). It takes a DynamoGraphDeploymentRequestSpec (DGDR) as input—defining the model, hardware, SLA targets, and backend—and uses the AI Configurator (AIC) to determine the best parallelization strategy and engine configurations. The output is a ready-to-deploy DGD YAML file.

  7. Overview of the Vendored TensorRT-LLM gRPC protocol

    main

    The trtllm_service.proto file provides a standalone version of the TensorRT-LLM gRPC protocol used by the sidecar. It is a modified version of the smg-grpc-proto package (version 0.4.14).

    To allow the contract to compile without requiring the external common.proto dependency, the following RPCs have been removed:

    • GetTokenizer
    • SubscribeKvEvents

    Despite these removals, all other messages exchanged by the sidecar are preserved byte-for-byte to ensure compatibility. The sidecar is designed to drive only the Generate RPC.

  8. Overview of kvbm-engine

    main

    The kvbm-engine crate provides distributed coordination primitives for KV cache block management (KVBM). It utilizes a leader/worker architecture to manage KV cache blocks across a tiered storage hierarchy:

    • G1: GPU HBM
    • G2: Pinned DRAM
    • G3: NVMe/SSD
    • G4: S3/MinIO

    Core Roles:

    • Leaders: Own block metadata and make placement decisions.
    • Workers: Execute data transfers using RDMA, NVMe, or object storage.
    • Sessions: Coordinate multi-instance block transfers.

  9. Overview of Dynamo Platform Helm Chart

    main
    The dynamo-platform Helm chart deploys the complete Dynamo Kubernetes Platform infrastructure. It includes the Dynamo Operator for managing deployments, and optional components such as NATS (messaging), etcd (distributed key-value store), Grove (multi-node inference orchestration), and Kai Scheduler (advanced workload scheduling).

  10. Overview of Dynamo KV Block Manager (KVBM)

    main

    The Dynamo KV Block Manager (KVBM) is a scalable runtime component that provides a unified memory layer and write-through KV cache. It manages memory allocation, management, and remote sharing of Key-Value (KV) blocks across heterogeneous and distributed environments, including GPU memory, pinned host memory, remote RDMA-accessible memory, local/distributed SSDs, and remote cloud storage.

    Key capabilities include:

    • Unified Memory API: Spans GPU, Host, SSD, and remote storage.
    • Block Lifecycle Management: Supports allocate → register → match transitions via an event-based system.
    • NIXL Integration: Uses the NIXL dynamic memory exchange layer for remote registration, sharing, and access of memory blocks.

  11. Overview of Grove Kubernetes API

    main
    Grove is a Kubernetes API designed for orchestrating disaggregated inference systems. It provides a unified API to define, configure, and scale specialized components like prefill, decode, and routing within a single custom resource. It is designed to work seamlessly with NVIDIA Dynamo to manage complex AI infrastructure.

  12. Overview of AI Simulate (Experimental)

    main

    AI Simulate is an experimental standalone Python distribution within the Dynamo repository. It provides simulation and configuration-search tools that utilize Dynamo models and replay capabilities.

    Note: This package is experimental and intended for evaluation and feedback. Its Python APIs, configuration schemas, search results, and deployment output are subject to change without notice. It does not provide SLAs, accuracy, or configuration-optimality guarantees.