DOCSEARCHbyAlgolia
Submit Docs

slime LLM Post-Training Framework

repository·main·Indexed 27 days ago

https://github.com/thudm/slime

A high-performance LLM post-training framework for Reinforcement Learning (RL) scaling. slime integrates Megatron for training and SGLang for rollout to provide an efficient pipeline for large-scale model training. It supports GRPO and PPO training, provides NPU support for Ascend hardware, and allows for agentic RL workflows through custom rollout and generation interfaces.

Tokens
34.6K
Snippets
83
Records
171
Agent score
93%

What's inside slime

  1. Overview of the slime ecosystem

    main

    The slime framework serves as a reusable RL substrate for various specialized production and research systems:

    • Dressage: Agentic RL training for blackbox agents and white loops across sandboxes (e.g., bwrap, E2B, Kubernetes).
    • Miles: Enterprise-grade RL post-training with deep SGLang integration and operational tooling.
    • vime: vLLM-native RL post-training; users can switch to vLLM by adjusting rollout-related parameters in existing slime launch scripts.
    • Relax: Asynchronous omni-modal agentic RL using Ray Serve and a Distributed Checkpoint Service (DCS).
    • OpenClaw-RL: RL server for personalized agents supporting GRPO and on-policy distillation.
    • P1: Physics reasoning models trained via multi-stage RL.
    • RLVE: Scaling LM RL using adaptive, verifiable environments.
    • TritonForge: Training LLMs for optimized GPU kernel generation.
    • APRIL: System-level optimization for accelerating the rollout generation phase.
    • qqr (hilichurl): Extension for evolving open-ended agents using ArenaRL and Model Context Protocol (MCP).
    • ART (AgentCore RL Toolkit): SDK for adapting production agents for RL training on AWS Bedrock AgentCore Runtime using the @app.rollout_entrypoint decorator.

  2. Overview of slime RL Post-training Framework

    main

    slime is a high-performance LLM post-training framework designed for RL scaling. It provides two core capabilities:

    1. High-performance training: Connects Megatron with SGLang to support various efficient training modes.
    2. Flexible data generation: Uses custom data generation interfaces and server-based engines to implement arbitrary training data generation workflows.

    The framework integrates Megatron training, SGLang rollout, custom data generation, reward computation, verifier feedback, and environment interaction into a single unified training/rollout/Data Buffer path.

  3. Overview of slime LLM post-training framework

    main

    slime is an LLM post-training framework designed for Reinforcement Learning (RL) scaling. It provides two primary capabilities:

    1. High-Performance Training: Connects Megatron with SGLang to support efficient training in various modes.
    2. Flexible Data Generation: Enables arbitrary training data generation workflows through custom data generation interfaces and server-based engines.

    The framework is designed so that Megatron training, SGLang rollout, custom data generation, reward computation, verifier feedback, and environment interaction all flow through a unified training / rollout / Data Buffer path. This architecture supports agentic workflows (tool use, sandbox interaction, etc.) as part of the standard data generation pipeline.

  4. Overview of slime RL framework

    main
    slime is an LLM post-training framework designed for Reinforcement Learning (RL) scaling. It focuses on high-performance training by connecting Megatron with SGLang and provides flexible data generation through custom interfaces and server-based engines. The framework is designed to be lightweight and opinionated, optimizing the Megatron + SGLang path for large-scale RL workloads.

  5. Overview of slime customization interfaces

    main

    slime allows you to inject custom logic into the training and rollout pipeline using function path arguments. This enables customization of generation, reward computation, sampling, filtering, and loss calculation without modifying the core codebase.

    Key customization categories include:

    • Rollout & Generation: Override the entire rollout logic (--rollout-function-path) or just the generation step (--custom-generate-function-path).
    • Rewards: Implement custom reward logic (--custom-rm-path) or post-process rewards (--custom-reward-post-process-path).
    • Filtering: Filter samples during dynamic sampling (--dynamic-sampling-filter-path), in the buffer (--buffer-filter-path), or for loss calculation (--rollout-sample-filter-path).
    • Data Processing: Post-process rollout data (--rollout-data-postprocess-path), convert samples to training data (--custom-convert-samples-to-train-data-path), or override the data source (--data-source-path).
    • Loss & Training: Implement custom loss functions (--custom-loss-function-path), importance sampling (--custom-tis-function-path), or PG loss reduction (--custom-pg-loss-reducer-function-path).
    • Megatron Hooks: Inject logic during Megatron initialization or before log-prob/training steps.

  6. Rollout Buffer Workflow Overview

    main

    Rollout Buffer acts as an asynchronous agent trajectory generation component. It interfaces between the slime training process and the agent framework via HTTP APIs.

    Workflow Logic:

    1. slime Training Process communicates with the Rollout Buffer via HTTP API.
    2. Rollout Buffer sends HTTP requests to the Agent Framework.
    3. The Agent Framework interacts with the LLM Server (launched by slime).
    4. The LLM Server provides a Model Response.
    5. The Agent Framework uses that response to perform Trajectory Generation, which is sent back to the Rollout Buffer.

  7. Understand slime Docker image release types

    main

    slime publishes two types of Docker images:

    1. stable version: Based on official sglang releases. These versions include patches applied by the slime team.
    2. latest version: Aligns directly with lmsysorg/sglang:latest.

  8. Architecture Overview of slime

    main

    The slime architecture consists of three primary modules:

    • training (Megatron): Manages the main training process. It reads data from the Data Buffer and synchronizes parameters to the rollout module after training.
    • rollout (SGLang + router): Generates new data (including rewards/verifiers) and stores it in the Data Buffer. It supports custom generation for multi-turn loops, tool calls, environment/sandbox interactions, and verifier-based rewards.
    • data buffer: Acts as the bridge module. It manages prompt initialization, custom data, and rollout generation methods (including agentic workflows that produce samples via a unified interface).

  9. Architecture of slime

    main

    The slime architecture consists of three core modules:

    1. training (Megatron): Handles the main training process, reads data from the Data Buffer, and synchronizes parameters to the rollout module.
    2. rollout (SGLang + router): Generates new data (including rewards/verifier outputs) and stores it in the Data Buffer. This module can be wrapped with multi-turn loops, tool calls, or environment interactions.
    3. data buffer: Acts as a bridge managing prompt initialization, custom data, and rollout generation methods.

  10. Recommended production pattern for long-running RL jobs

    main

    For stable production workloads, follow this pattern:

    1. Enable --use-fault-tolerance.
    2. Save checkpoints regularly using --save-interval.
    3. Save rollout debug dumps, especially for new agentic or verifier-heavy workloads.
    4. Use the Trace Viewer to inspect long-tail samples and reward/model-call spans.
    5. Use Profiling to distinguish between rollout and training bottlenecks.
    6. Maintain explicit SGLang deployment using SGLang Config for complex topologies.

  11. Use Shared-Filesystem Visibility Hooks

    main

    For object-store-backed mounts that require explicit refresh/publish steps to make writes visible across hosts, you can provide custom hooks via import paths:

    Trainer Side (slime)

    Use --custom-update-weight-post-write-path to run code after a version's files are written but before engines are notified. Signature: hook(args, version_dir, rollout_engines)

    Engine Side (sglang)

    Use --sglang-custom-pull-weights-pre-read-hook to run code on each host before the engine reads the delta directory (e.g., to refresh a mount's view). Signature: hook(delta_dir, target_version)

  12. Use the automated profiling tool

    main

    The tools/profile_rollout.py script automates profiling across multiple engines simultaneously.

    Starting Profiling

    To start profiling on all workers for a specific number of steps, use the --action start command. By default, it will stop after 3 steps.

    Parameters:

    • --router-url: The URL of the Router.
    • --num-steps: Number of steps to record (defaults to 3).
    • --output-dir: Directory where trace files will be saved.
    • --activities: Activities to monitor (e.g., GPU, CPU).
    • --profile-by-stage: Whether to profile by stage (prefill/decode).

    Stopping Profiling Manually

    If you did not specify --num-steps or need to stop the profiling session early, use the --action stop command.