DOCSEARCHbyAlgolia
Submit Docs

LLMServingSim

repository·main·Indexed 43 days ago

https://github.com/casys-kaist/llmservingsim

A cycle-level simulator for LLM serving infrastructure designed for end-to-end study of heterogeneous accelerators, disaggregated memory, and parallelism strategies including Tensor Parallelism (TP), Pipeline Parallelism (PP), Expert Parallelism (EP), and Data Parallelism (DP). It supports PIM (Processing-In-Memory) configurations via DRAMSim3 and integrates with vLLM for profiling and model configuration.

Tokens
86.6K
Snippets
168
Records
341
Agent score
57%

What's inside LLMServingSim

  1. Overview of llm_profile v0

    main

    The llm_profile tool is a PyTorch-based profiler designed to measure LLM performance metrics for use in LLMServingSim. It measures:

    • Layer latencies: Execution time for non-attention layers.
    • Attention latencies: Execution time for attention mechanisms across various batch sizes and sequence lengths.
    • Power consumption: GPU-level (via nvidia-smi) and system-level (via ipmitool) power usage.

    It supports dense and MoE (Mixture-of-Experts) architectures and generates CSV outputs and a scikit-learn-based attention latency predictor used for real-time latency prediction during simulation.

  2. What is LLMServingSim?

    main

    LLMServingSim is a cycle-level simulator designed for studying LLM serving infrastructure. It provides a unified environment to evaluate:

    • Heterogeneous accelerators
    • Disaggregated memory tiers (e.g., CPU, CXL, PIM)
    • MoE (Mixture-of-Experts) routing
    • Multi-instance parallelism (including TP, PP, EP, and DP)

    Architecture Overview

    The simulator works by pairing a Python frontend (which mirrors the vLLM continuous-batching scheduler) with the ASTRA-Sim C++ analytical network backend. It uses per-hardware latency data captured from a vLLM-based layerwise profiler to drive the simulation.

  3. Overview of LLMServingSim capabilities

    main

    LLMServingSim is a sandbox for modeling heterogeneous and disaggregated LLM serving infrastructure. It allows developers to simulate various hardware configurations, parallelism strategies, and memory tiers without requiring physical access to the hardware.

    Key Capabilities:

    • Topology Modeling: Configure TP (Tensor Parallelism), PP (Pipeline Parallelism), EP (Expert Parallelism), and DP+EP (Data Parallelism + Expert Parallelism) across multiple instances. Supports chunked prefill, prefix caching, MoE expert routing, and KV-cache offloading via configuration files.
    • Hardware Profiling: Plug in new hardware like GPUs, CXL tiers, or PIM (Processing-in-Memory) devices. It uses a vLLM-based layerwise profiler to capture real CUDA timings.
    • Validation: The simulator is validated against vLLM with sub-3% error on TTFT (Time to First Token), TPOT (Time Per Output Token), and throughput.
    • Experimental Scenarios: Supports ShareGPT traces, agentic sessions, large-scale clusters (e.g., 10x), and unreleased hardware/memory tiers.

  4. Choose a workload generation path

    main

    Depending on your simulation goals, you can use one of three approaches to define workloads:

    1. Real-world traffic: Generate JSONL from ShareGPT or any Hugging Face text dataset. This provides realistic prompt/output length distributions and configurable arrival rates.
    2. Agentic / closed-loop: Model multi-step LLM calls with tool waits (e.g., SWE-bench, ReAct, browser agents). The simulator models the dependency chains between sub-requests.
    3. Synthetic / custom: Hand-craft your own JSONL, create fixed-length stress tests, or replay your own production logs.

  6. Understand the three types of simulator output

    main

    The simulator produces three distinct types of output that you can use for analysis:

    1. Per-request CSV: A detailed file containing metrics for every completed request. The path is specified via the --output flag.
    2. Throughput log line: Periodic status updates printed to stdout every --log-interval seconds.
    3. Final power summary: A cluster-wide average power consumption figure (only generated if the cluster configuration includes a power: block).

    Use the CSV for granular, per-request analysis and the log lines for real-time monitoring of throughput and resource utilization during a run.

  7. Understand the serving core modules

    main

    The core logic of the simulator resides in serving/core/. When extending the simulator or understanding its behavior, refer to these key modules:

    • request.py: Defines Request and Batch data classes. It tracks per-request state and latency metrics like TTFT (Time to First Token), TPOT (Time Per Output Token), and ITL (Inter-Token Latency).
    • scheduler.py: Implements vLLM-style continuous batching. It manages request queuing, memory-constrained batch formation, KV cache block eviction/swapping, and prefix cache lookup. You can add custom scheduling policies here.
    • router.py: Routes requests across instances in real-time. The default LOAD policy uses vLLM-style weighted least-loaded scoring (waiting * 4 + running). It handles request transfer in Prefill/Decode disaggregation mode.
    • gate_function.py: Routes tokens to MoE experts. Supported policies include Copy (default, enables block copy optimization), Round Robin, Random, and Custom. It provides EP-aware routing via route_ep().
    • memory_model.py: Handles static sizing and byte-level views of block pools. Use calculate_sizes(parallel=) and get_weight for per-layer tensor size computation. The parallel parameter represents TP degree for dense layers and EP degree for MoE experts.
    • block_pool.py: Manages memory tiers (NPU/CPU/CXL) using a doubly linked free list in eviction order. It is a port of vLLM v0.19.0.
    • kv_cache_manager.py: Implements TieredKVCacheManager for per-request NPU block tables and tier lookup. It uses a unified block hash space across tiers.
    • trace_generator.py: The core performance estimator that converts hardware profiles and model architectures into execution traces.
    • config_builder.py: Parses cluster JSON configs and generates ASTRA-Sim input files (network/network.yml, memory/memory_expansion.json, system/system.json).
    • power_model.py: Estimates power/energy for NPU, CPU, DRAM, interconnect, NIC, and storage.
    • controller.py: Manages the IPC protocol with the ASTRA-Sim subprocess.
    • graph_generator.py: Invokes the Chakra converter to transform text traces into protobuf workload graphs.
    • pim_model.py: Parses PIM device INI configurations to derive bandwidth, latency, and power parameters.

  8. System requirements for LLMServingSim

    main

    LLMServingSim requirements vary depending on whether you are running the simulator alone or using the profiler/benchmarking tools. If you only plan to run pre-profiled simulations (e.g., bundled RTXPRO6000 profiles), a GPU is not required.

    FeatureRequired for SimulatorRequired for Profiler / Bench
    OSLinux (Ubuntu 22.04+ tested)Linux (Ubuntu 22.04+ tested)
    Docker✓✓ (or bare-metal install)
    NVIDIA GPU✓
    NVIDIA Container Toolkit✓ (for GPU passthrough into Docker)
    CUDA driver12.x for vllm/vllm-openai:v0.19.0; 13.x for v0.19.0-cu130
    Disk~3 GB~10 GB additional (vLLM image + HF model cache)
    RAM16 GB32 GB+ recommended

  9. What is a workload in llmservingsim

    main

    A workload is a JSONL file that drives the simulator's request queue. It defines when requests arrive and what they contain. The simulator reads the file at startup and the router schedules each request when its arrival_time_ns is reached on the simulated clock.

    There are two primary formats supported:

    1. Flat format: Each line represents one independent request.
    2. Agentic format: Each line represents a single session containing chained sub-requests (e.g., multi-step LLM calls with tool waits).

    Both python -m serving --dataset and python -m bench run --dataset consume these files.

  10. Overview of the vLLM-based Profiler

    main

    The profiler is a vLLM-based layerwise profiler used to generate the input data required by the simulator's trace_generator. It drives a real vLLM engine using synthetic batches and records per-layer CUDA kernel latencies into category-specific CSV files.

    Workflow Integration:

    1. Profiler (runs in vLLM Docker/bare metal) $\rightarrow$ produces CSV bundle in profiler/perf/<hardware>/<model>/<variant>/.
    2. Simulator (runs in simulator container) $\rightarrow$ reads the CSV bundle via _load_perf_db to generate traces and run simulations.

    The profiler and simulator communicate exclusively through the shared profiler/perf/ directory.

  12. Interpret the Heartbeat Block (Throughput Logs)

    main

    Every --log-interval simulated seconds, the simulator prints a heartbeat block. The leading timestamp (e.g., [1.0s]) is the simulated clock, not wall-clock time.

    Throughput Metrics

    • Avg prompt throughput: Input tokens/s over the interval (includes prefix-cache hits).
    • Avg generation throughput: Generated tokens/s over the interval.

    Instance and Node Metrics

    • Running Instance[i]: The number of persistent running requests (len(scheduler.running)). This is the analogue of vLLM's num_running_reqs.
    • Waiting: Requests that have arrived but are not yet scheduled.
    • Each NPU Memory Usage: Weights + KV per rank as a percentage of npu_mem.mem_size. The ceiling is mem_util * 100.
    • Prefix Cache Hit ratio: Cumulative ratio since the start of the run: (hit tokens / requested tokens).
    • Node[i]: Host memory used by the node's lower KV tier. If multiple instances exist on a node, the line shows the per-instance share of the node's CPU usage (summing to 100% if resident).

    Specialized Hardware/Models

    • CXL: If --prefix-storage CXL is used, a CXL branch appears showing device memory usage.
    • Power: If a power: block is configured, a cluster-wide Avg power consumption line is appended at the end of the heartbeat.