DOCSEARCHbyAlgolia
Submit Docs

RoleBasedGroup (RBG)

repository·main·Indexed 42 days ago

https://github.com/sgl-project/rbg

A Kubernetes-native orchestration layer for distributed, stateful AI inference workloads. RBG enables complex multi-role topologies, such as disaggregated prefill/decode, with built-in coordination, service discovery, and hardware-aware scheduling. It includes the RoleBasedGroupWarmup CRD (workloads.x-k8s.io/v1alpha2) to mitigate cold start latency for LLMs via image and volume pre-warming.

Tokens
119.3K
Snippets
226
Records
367
Agent score
54%

What's inside RBG

  1. Overview of KEP-171 Pod Port Allocation

    main

    KEP-171 provides a dynamic port allocation solution for RBG (RoleBasedGroup) applications. This feature is specifically designed to solve port conflicts that occur when deploying LLM services using the PD (Prefill-Decode) disaggregated approach with hostNetwork: true.

    When multiple replicas of the same Role are scheduled to the same node using hostNetwork, they would normally conflict if they use the same port. KEP-171 automatically assigns unique port numbers to each replica during deployment, increasing deployment density and allowing more replicas to run on the same node.

  2. Overview of KEP-74 Mooncake Integration

    main

    KEP-74 introduces Mooncake integration into the RBG-deployed SGLang inference service. Mooncake is a distributed KVCache storage engine designed for LLM inference, acting as a central component in a KVCache-centric distributed architecture.

    By integrating Mooncake as a role within RBG, the system provides KVCache offload capabilities, allowing for a distributed memory pool to store reusable KVCache across different tiers. This is particularly beneficial for improving service SLOs in scenarios like multi-turn conversations.

  3. Image and Volume Warmup for RoleBasedGroup (KEP-129)

    main

    KEP-129 introduces image and volume warmup capabilities to RoleBasedGroup (RBG) to mitigate cold start latency when serving Large Language Models (LLMs). This feature allows users to pre-warm container images and model files (volumes) before a serving instance starts, which is critical for maintaining high availability and GPU utilization during upgrades.

    Key Capabilities

    • WarmUp Jobs: Users can submit jobs that target specific nodes to preload container images and execute customized warmup actions.
    • Reduced Upgrade Latency: By adding a 'warmup stage' before the actual upgrade process, the time required for RBG to perform rolling updates is significantly reduced.

    Scope

    • Included: Image preloading and customized warmup actions.
    • Excluded: Coordinated upgrades between WarmUp and RBG's core upgrade process, and dataCopy actions (copying model files between volumes) are currently out of scope.

  4. Explore the RoleBasedGroup (RBG) Ecosystem

    main

    RBG is supported by a suite of specialized projects designed for autoscaling, runtime management, CLI extensions, and AI Agent integration:

    • rbg-planner: An engine-agnostic, SLA-driven LLM inference autoscaler. It uses pluggable metric adapters (supporting SGLang, vLLM, NVIDIA Dynamo) and ARIMA-based load forecasting to scale prefill/decode roles to meet TTFT/ITL latency targets.
    • inference-engine-runtime: A Python sidecar runtime for inference engines. It provides LoRA adapter management, unified Prometheus metrics, and distributed topology management for SGLang and vLLM.
    • inference-ext-cli (llmctl): An RBG CLI extension for managing LLM inference workloads. It includes service/model management, benchmark orchestration, automated parameter search (via Optuna), convergence analysis, and an experimental visualization web panel.
    • rbg-agent-guide: A guide for AI Operations Agents (like Claude Code) to provide deployment skills, enabling users to deploy LLM models to Kubernetes using RBG CRDs and the CLI.
    • rolebasedgroup.github.io: The official documentation site built with Docusaurus.

  5. Explore the RBG Ecosystem Projects

    main

    The RBG ecosystem includes several companion projects designed to extend functionality for autoscaling, runtime management, CLI operations, and AI agent integration:

    • rbg-planner: An engine-agnostic, SLA-driven autoscaler for LLM inference on Kubernetes. It supports SGLang, vLLM, and NVIDIA Dynamo. It uses ARIMA-based load prediction and automatic SLA profiling to scale prefill/decode roles to meet TTFT (Time To First Token) and ITL (Inter-Token Latency) targets.
    • inference-engine-runtime: A Python-based sidecar runtime for AI inference engines. It manages LoRA adapters, provides unified Prometheus metrics, and handles distributed topology management for SGLang and vLLM.
    • inference-ext-cli (llmctl): An RBG CLI extension for managing LLM inference workloads. It supports service/model management, benchmark orchestration, automated parameter search (via Optuna), convergence analysis, and web dashboards.
    • rbg-agent-guide: Skill guides for AI agents (like Claude Code) to assist in deploying LLM models to Kubernetes using RBG CRDs and the CLI.
    • rolebasedgroup.github.io: The official documentation website.

  6. What is In-Place Scheduling in RBG?

    main

    In-Place Scheduling is a scheduling enhancement for RoleBasedGroup (RBG) designed to optimize AI inference workloads. It ensures that Pods being recreated during rolling upgrades or following a failure are preferentially scheduled to nodes where they (or their component peers) previously ran.

    This mechanism allows new Pods to reuse node-local cached resources, such as:

    • Model weights: Large files (tens to hundreds of GBs) that avoid expensive re-downloads.
    • GPU state: Warm-up state and inference framework dumps.
    • Per-Pod state: Model shards, KV-cache files, and runtime metadata dumped to local disk.

    By steering Pods toward their historical nodes, RBG reduces the time required for Pods to become ready and take over traffic.

  7. What is RoleBasedGroup (RBG)?

    main
    RoleBasedGroup (RBG) is a custom resource designed to model a group of roles, where each role represents a specific workload type and its associated set of pods. It manages multi-role applications by providing coordinated scheduling, lifecycle management, rolling updates, and optional gang-scheduling (PodGroup) support. This is particularly useful for complex distributed applications that require synchronized management across different types of workloads.

  8. What is Mooncake and how does it integrate with SGLang?

    main

    Mooncake is a high-performance L3 storage backend for SGLang HiCache. It enables cross-node distributed KV cache storage using RDMA-accelerated data transfer.

    When L1 and L2 caches miss, HiCache automatically retrieves the required KV cache from Mooncake's distributed cache. This provides virtually unlimited cache capacity by using a distributed memory pool, overcoming the limits of GPU-only or GPU+CPU caches. The system uses intelligent prefetching and zero-copy techniques to ensure high-bandwidth, low-latency transfers between SGLang instances and Mooncake storage nodes.

  9. What is RoleBasedGroup (RBG)?

    main
    RoleBasedGroup (RBG) is a Kubernetes API designed to orchestrate distributed, stateful AI inference workloads. Unlike traditional Kubernetes primitives (like StatefulSet or Deployment) which struggle with the complex topologies required for LLM inference (e.g., gateway → router → prefill → decode), RBG treats an inference service as a single logical unit composed of multiple coordinated roles. It provides built-in service discovery, hardware-aware scheduling (GPU/NVLink/RDMA), and atomic cross-role operations for deployment, scaling, and recovery.

  10. What is CoordinatedPolicy and when to use it

    main

    In multi-role inference architectures (like PD-disaggregated setups), roles like Prefill and Decode operate independently. Without coordination, they may scale or upgrade at different rates, leading to version mismatches, resource waste (idle GPUs), or service unavailability.

    CoordinatedPolicy is a standalone Custom Resource Definition (CRD) used to synchronize the deployment and upgrade progress of multiple roles within a RoleBasedGroup (RBG).

    It supports two primary coordination scenarios:

    1. Coordinated scaling: Ensures replica counts grow proportionally during initial deployment and subsequent scale-outs.
    2. Coordinated upgrade: Ensures consistent update progress across roles during rolling updates to prevent protocol incompatibilities between new and old versions.

    Key Behavior:

    • It binds to an RBG by having the same name and namespace as the RBG.
    • It is decoupled from the RBG lifecycle; deleting the policy does not affect the RBG.
    • It does not affect scale-down behavior.

  11. What is an Instance

    main

    An Instance (v1alpha1) is a collection of Pods with a tightly bound lifecycle. It is used to manage upgrades, status updates, and in-place updates for a group of Pods.

    Note on Versions: The Instance resource described here is part of the v1alpha1 API. In v1alpha2, RoleBasedGroup uses RoleInstanceSet or RoleInstance resources via the customComponentsPattern instead of this v1alpha1 Instance resource.

  12. How exponential backoff works in RBG

    main

    When using RecreateRoleInstanceOnPodRestart, RBG applies exponential backoff to subsequent crashes within a stability window to prevent rapid restart loops.

    Backoff Logic

    • First crash: Immediate recreation (0s delay).
    • Second crash: Delayed by baseDelaySeconds.
    • Subsequent crashes: The delay doubles each time, capped at maxDelaySeconds.
    • Formula: delay = min(baseDelaySeconds * 2^(restartCount-1), maxDelaySeconds).

    Stability Window

    The stability window is defined as max(maxDelaySeconds * 2, 10 minutes). If an instance remains stable (no crashes) for longer than this window, the restartCount resets to 0, and the next crash will trigger an immediate recreation.