DOCSEARCHbyAlgolia
Submit Docs

verl-agent

repository·master·Indexed 24 days ago

https://github.com/langfengq/verl-agent

A specialized extension of the veRL (Volcano Engine Reinforcement Learning for LLM) framework designed for training LLM agents via reinforcement learning. It focuses on long-horizon, multi-turn interactions using a step-wise input construction method. The repository includes integrations for various environments such as ALFWorld, gym-cards, SkyRL-Gym, and WebShop.

Tokens
69.9K
Snippets
157
Records
310
Agent score
81%

What's inside verl-agent

  1. Overview of verl-agent for LLM Agent Training

    master

    What is verl-agent?

    verl-agent is an extension of veRL designed specifically for training Large Language Model (LLM) agents using Reinforcement Learning (RL).

    Core Capabilities

    • Step-Independent Multi-turn Rollouts: Unlike methods that concatenate full interaction histories, verl-agent uses a step-independent mechanism. This allows for fully customizable per-step input structures, history management, and memory modules.
    • Scalability: Because it constructs inputs step-by-step rather than growing a single massive context window, it is highly scalable for long-horizon tasks (e.g., 30–50 steps in ALFWorld) without hitting token limits.
    • Memory Management: Features a customizable memory module that lets developers define what to include in each step (e.g., recent steps, key events, or summaries).
    • Environment Support: Supports parallelized Gym-style environments and "Group Environments" (where multiple rollouts share the same initial state), which is critical for algorithms like GRPO and DAPO.
    • Model & Modality Support: Compatible with models like Qwen3, Qwen3-VL, Qwen2.5, and LLaMA3.2. It supports both text-only and multi-modal (text + image) agents.
    • Efficient Training: Supports LoRA (Low-Rank Adaptation) to reduce computational costs (e.g., training 7B models on 2 H100 GPUs).

  2. Overview of verl RL training framework

    master

    verl is a flexible, efficient, and production-ready Reinforcement Learning (RL) training framework specifically designed for Large Language Model (LLM) post-training. It is an open-source implementation of the HybridFlow paper.

    Key capabilities include:

    • Flexible RL Algorithms: Uses a hybrid programming model (combining single-controller and multi-controller paradigms) to allow users to build complex RL dataflows with minimal code.
    • Modular LLM Integration: Decouples computation and data dependencies, allowing seamless integration with frameworks like PyTorch FSDP, Megatron-LM, vLLM, and SGLang.
    • Efficient Resource Utilization: Supports flexible device mapping and parallelism, allowing models to be placed on different sets of GPUs for scalability.
    • High Throughput: Achieves state-of-the-art generation and training throughput by integrating SOTA LLM infra and using a 3D-HybridEngine for efficient actor model resharding, which reduces communication overhead during transitions between training and generation phases.

  4. Overview of SkyRL-Gym features

    master

    SkyRL-Gym is a library of Reinforcement Learning (RL) environments for Large Language Models (LLMs) built using the Gymnasium API.

    Key capabilities include:

    • Gymnasium API Compliance: Uses a standard Environment interface.
    • Ready-built Environments: Includes environments for math, code, search, and text-to-SQL tasks.
    • Reusable Tool Interface: Developers can implement a tool once and reuse it across different environments.
    • Multi-tool Support: Environments can support the use of multiple tools simultaneously.

  5. Use the PyTorch FSDP Backend

    master

    The PyTorch FSDP Backend is recommended for algorithm research and prototyping due to its simplicity and ease of organizing forward/backward computations. It supports various model types (actor, critic, reference, rollout, and reward) by implementing specialized workers.

    Key Components:

    • Workers: Specialized implementations for different model roles.
    • FSDPVLLMShardingManager: A context manager used to reshard weights between FSDP (for training/actor) and vLLM (for generation/rollout).
    • Weight Loaders:
      • hf_weight_loader: Supports any model available in both Hugging Face and vLLM without code changes.
      • dtensor_weight_loader: Required for custom weight synchronization between FSDP and vLLM.

    Trade-offs:

    • Pros: Readily supports various models; easy to organize computation.
    • Cons: Poor scalability for very large models (e.g., Llama 70B/405B); higher resharding overhead between actor and rollout compared to the Megatron-LM backend.

  6. Supported Environments in SkyRL-Gym

    master

    SkyRL-Gym supports several environment types designed for testing agent capabilities across different domains:

    • Basic Environments: Single-domain tasks such as Search, SQL, Math, and Simple Code (LCB).
    • Multi-Tool Environment: An environment where the model must decide which tool to invoke at each step (e.g., combining Search and Python Code). Note that these environments use different tool parsing logic compared to basic environments.
    • Mix Dataset Environment: Multi-domain datasets that require switching between different environment initializations. These environments reuse the environment instance but require a different agent loop structure.

  7. DeepSeek R1 Reproduction Recipe Overview

    master

    This recipe provides the framework and results for reproducing DeepSeek R1. It includes evaluation benchmarks for both the distilled models (e.g., DS-R1-Distill-Qwen2.5-1.5B) and the original DS-R1 model across various datasets like GPQA Diamond, LiveCodeBench, AIME 2024, and CNMO 2024.

    Note: This recipe is currently under development. Developers interested in contributing can find the task list at the project's issue tracker.

  8. What is Sandbox Fusion and how does it benefit LLM training?

    master

    Sandbox Fusion is a remote code sandbox service that provides a secure environment for running and evaluating code generated by Large Language Models (LLMs).

    Using Sandbox Fusion for code verification offers two main benefits:

    1. Security: Provides a secure, isolated environment for executing untrusted LLM-generated code.
    2. Performance: By leveraging remote CPU resources for concurrent code verification, it can reduce the reward stage time by 10-30%, depending on the quality of the generated code.

  9. What is the SPIN recipe and how does it work?

    master

    The SPIN (Self-Play Fine-Tuning) recipe in verl implements an iterative self-improvement algorithm inspired by game theory. Instead of relying on fixed external preference datasets, the model learns by playing against itself.

    The Online Loop Algorithm:

    1. Generation: The current model generates multiple responses for each prompt in a batch.
    2. Preference Labeling: Responses are evaluated (e.g., using rule-based ranking for math problems) to identify a chosen response and a rejected response.
    3. Update: The model is updated using compute_online_dpo_loss, comparing the actor's policy against a reference model using the generated preference tuples (prompt, chosen_response, rejected_response).

    This approach allows the target data distribution to change dynamically as the model improves, potentially surpassing the performance ceiling of static human-annotated data.

  10. What is Self-Play Fine-Tuning (SPIN)?

    master

    SPIN is a language model finetuning algorithm implemented in verl that enables iterative self-improvement through a self-play mechanism. Instead of relying on fixed external preference datasets, the model generates its own training data by playing against itself.

    The SPIN Workflow:

    1. Synthetic Data Generation: The current model generates responses, creating its own training data from previous iterations.
    2. Two-Player Game Setup: A game involving two players is acted by a single LLM.
    3. Iterative Training: The model progressively improves by refining its policy, with each iteration's model becoming the opponent for the next iteration.

    In this implementation, SPIN is realized by using DPO (Direct Preference Optimization) loss iteratively, where preference data is dynamically updated during training.

  12. How the Online DPO implementation works in SPIN

    master

    The SPIN recipe in verl adapts the existing PPO infrastructure for iterative online DPO. Key architectural differences from standard PPO include:

    • No Critic: The value function critic is omitted.
    • Dynamic Reference Model: An explicit reference policy (ref_policy_wg) is used. The reference model's weights can be periodically updated from the actor using the ref_update_freq parameter.
    • Online Preference Generation: The compute_onlineDPO_pref function (in core_algos.py) dynamically creates chosen/rejected pairs based on a reward source (e.g., rule-based ranking for math problems).
    • DPO Loss Integration: The compute_online_dpo_loss function (in core_algos.py) is used within the actor update (dp_actor.py) to optimize the policy directly using generated preferences.
    • Orchestration: The SpinTrainer (in spin_trainer.py) manages the loop of generation, preference labeling, reference model updates, and policy updates.