DOCSEARCHbyAlgolia
Submit Docs

rLLM

repository·main·Indexed 26 days ago

https://github.com/rllm-org/rllm

An open-source framework for training language agents using reinforcement learning. rLLM allows developers to wrap agent harnesses or sandboxes and switch between multiple training backends, including verl, tinker, and fireworks. The framework includes cookbooks for specific agents such as Deepcoder, FinQA, and FrozenLake, and provides CLI tools for dataset management, agent evaluation, and training.

Tokens
137.5K
Snippets
391
Records
677
Agent score
90%

What's inside rLLM

  2. Overview of SETA (Scaling Environment for Terminal Agents)

    main
    SETA, developed by CAMEL-AI, provides scalable environment infrastructure specifically designed for training terminal agents using rLLM. It is used to create diverse and reproducible terminal environments necessary for Reinforcement Learning (RL) training at scale.

  3. Overview of V1: Parallel Self-Verification

    main
    V1 is a framework designed to improve model performance during inference by unifying generation and pairwise self-verification. Instead of scoring individual solution candidates in isolation, V1 uses a pairwise approach where the model compares two candidates head-to-head. This is combined with a tournament-based ranking algorithm to optimize the allocation of verification compute. The framework is designed to jointly develop generation and verification capabilities, which has shown improvements of up to 10% on math reasoning and code generation benchmarks.

  4. Overview of LLM-in-Sandbox

    main
    LLM-in-Sandbox is a research project focused on building general-purpose agents by running Large Language Models (LLMs) inside sandboxed virtual computer environments. It leverages rLLM to perform Reinforcement Learning (RL) training for agents that interact with full desktop and terminal interfaces.

  5. Use Terminal-Bench-RL for long-horizon terminal agent training

    main
    Terminal-Bench-RL is a benchmark and training framework designed for long-horizon task completion within terminal environments. It provides a suite of tasks specifically for evaluating and training terminal agents using the rLLM reinforcement learning (RL) pipeline.

  6. FinQA Agent Flow Pattern

    main

    The FinQA agent follows a multi-turn ReAct-style loop (up to 20 turns). It uses native OpenAI function calling to dispatch tools.

    Termination: The loop breaks when the model emits a FINAL ANSWER: block without tool calls, or when the maximum number of turns is reached.

    Artifacts: The resulting Episode contains:

    • answer: The full text response.
    • accessed_tables: A list of tables accessed during the run (used for table-access bonuses).
    • turns: The number of steps taken.

  7. Geo3K Agent Architecture and Pattern

    main

    The Geo3K agent is a single-turn vision-language agent (VLM) that solves geometry problems using the AgentFlow protocol.

    Pattern Details

    • Loop shape: Single-turn (one VLM call per task).
    • Tools: None; the answer is parsed directly from the model response.
    • Inputs: Multimodal (text question + base64-encoded diagram image).
    • Termination: The single LLM call returns, and the evaluator extracts the answer using \boxed{...}.
    • Reward shape: 1.0 if the boxed answer matches the ground truth (symbolic math), otherwise 0.0.

    Multimodal Content-Block Pattern

    When implementing similar agents in AgentFlow, the messages list should use the multimodal content-block pattern, including an image_url block with a base64-encoded string:

    # Example pattern for multimodal messages
messages = [
    {"type": "system", "content": "system_prompt"},
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "question_text"},
            {
                "type": "image_url",
                "image_url": {"url": f"data:image/png;base64,{base64_string}"}
            }
        ]
    }
]

  8. Compare rLLM training backends: verl vs tinker

    main

    rLLM provides two training backends depending on your scale and complexity requirements:

    • verl: A distributed, production-ready backend designed for large-scale training. It uses a Ray-based architecture, supports multi-node/multi-GPU clusters, and is suitable for Vision-Language Models (VLM) and advanced distributed training (FSDP, tensor parallel).
    • tinker: An async-first, service-based backend designed for rapid development and prototyping. It is best suited for single-node training, LoRA fine-tuning, and has a gentler learning curve with simpler configuration.

  9. Understand the rLLM data model: Step, Trajectory, and Episode

    main

    The solver-judge workflow relies on three nested data structures:

    1. Step: The atomic unit representing a single LLM interaction. It captures input messages, generated output, token IDs, log-probabilities, and a parsed action.
    2. Trajectory: An ordered list of Steps from a single role (e.g., "solver" or "judge"). The name attribute is used by the trainer to group trajectories for advantage computation.
    3. Episode: The top-level container returned by an AgentFlow. It bundles all trajectories from a single rollout execution along with metadata like is_correct and artifacts.

  10. Understand the rLLM Trainer Capability Matrix

    main

    rLLM uses a UnifiedTrainer that supports different backends, launch methods, execution flows, and dataset types. Capabilities are not uniform across these dimensions.

    Key Dimensions:

    • Backend: tinker (single-machine, LoRA-only, async-native) vs verl (Ray-distributed, multi-GPU/node, FSDP/Megatron).
    • Launch Method: rllm train CLI (a strict subset of features, hardcoded to tinker backend) vs Python script/Hydra (full surface, supports verl and all config keys).
    • Execution Flow: regular, sandboxed, or remote runtime (decides if gateway, hooks, and sandbox warm pool engage).
    • Dataset Type: rLLM-native rows vs harbor task dirs (decides how rows become Tasks).

    Critical Rule: The CLI is a subset of the Python API. If a feature is not available via rllm train flags, you must use a Python script with AgentTrainer to access it.

  11. Understand rLLM Cookbooks

    main

    A cookbook is a self-contained Python package that bundles an AgentFlow and an Evaluator together. It includes data preparation scripts (prepare_data.py) and training scripts (train_tinker.sh for single-machine LoRA or train_verl.sh for distributed multi-GPU training). Cookbooks are discovered by the rLLM CLI via Python entry points, allowing you to run evaluations and training using simple CLI commands.

    Each cookbook's AgentFlow is an async function that calls an OpenAI-compatible endpoint and returns an Episode object.

  12. Understand the rLLM execution pipeline

    main

    The rLLM training workflow follows a four-step pipeline:

    1. Run Agent: The agent runs as-is; rLLM's SDK intercepts LLM calls.
    2. Collect Traces: LLM calls are structured into Episodes (one task), Trajectories (one agent run), and Steps (one LLM call).
    3. Compute Rewards: A reward function scores the results.
    4. Update Model: An RL algorithm (such as GRPO, REINFORCE, or RLOO) updates the model weights.

    Internal components involved:

    • Workflow Engine: Runs parallel agent instances for rollouts.
    • LiteLLM Proxy: Routes requests and captures token IDs and logprobs.
    • Transform Pipeline: Groups trajectories for advantage computation.
    • Training Backend: (verl or tinker) executes the policy update.