DOCSEARCHbyAlgolia
Submit Docs

verl (Volcano Engine Reinforcement Learning)

repository·main·Indexed 10 days ago

https://github.com/verl-project/verl

A high-performance RL training library for LLMs featuring HybridFlow, a flexible programming model that decouples computation and data dependencies. It supports various infrastructures including NVIDIA GPUs, AMD ROCm, and Huawei Ascend NPUs, with integrations for vLLM, SGLang, Megatron-LM, and TransformerEngine.

Tokens
233.3K
Snippets
520
Records
868
Agent score
97%

What's inside verl

  1. Overview of verl RL training framework

    main

    verl is a flexible, efficient, and production-ready Reinforcement Learning (RL) training framework designed for Large Language Model (LLM) post-training. It implements the HybridFlow paradigm, which combines single-controller and multi-controller models to enable efficient execution of complex RL dataflows.

    Key Capabilities:

    • Flexible RL Algorithms: Build complex dataflows with minimal code using a hybrid programming model.
    • Modular LLM Integration: Decouples computation from data dependencies, allowing seamless use of frameworks like PyTorch FSDP, Megatron-LM, vLLM, and SGLang.
    • Efficient Resource Utilization: Supports flexible device mapping and parallelism to scale across different cluster sizes.
    • High Throughput: Achieves state-of-the-art performance through SOTA LLM integration and the 3D-HybridEngine, which optimizes actor model resharding to reduce memory redundancy and communication overhead during training-to-generation transitions.

  2. Overview of the Fully Async Policy Trainer

    main

    The fully_async_policy is an experimental PPO training system in verl designed to decouple the Trainer and the Rollouter. This architecture allows for asynchronous sample generation and training, which improves GPU utilization and training efficiency by addressing the 'long-tail' problem (where training waits for slow rollout tasks to finish).

    Key features include:

    • Resource Isolation: Rollouter and Trainer use separate computing resources, which must be specified independently.
    • Parallelism: The Rollouter generates new samples while the Trainer is simultaneously performing training steps.
    • Flexible Asynchrony: Supports multi-step asynchronous settings (from 0.x steps to multiple steps) and freshness control via async_training.staleness_threshold.
    • Stream Inference: Rollouter generates data sample-by-sample, using a single sample as the minimum transmission unit.
    • Partial Rollout: Supports sleep() and resume() logic during parameter synchronization to save ongoing rollouts and reduce waiting time.

    Supported Mode: Currently supports megatron/fsdp+vllm. Note that vllm must be used in server mode based on AgentLoop.

  3. Overview of verl: Volcano Engine Reinforcement Learning for LLMs

    main

    verl is a flexible, efficient, and production-ready Reinforcement Learning (RL) training library specifically designed for Large Language Models (LLMs). It is the open-source implementation of the HybridFlow framework.

    Key capabilities include:

    • Flexible RL Algorithms: Uses a hybrid-controller programming model to easily implement complex dataflows like GRPO and PPO.
    • Modular Infrastructure Integration: Decouples computation and data dependencies, allowing seamless use of existing LLM frameworks such as FSDP, Megatron-LM, vLLM, and SGLang.
    • Efficient Resource Utilization: Supports flexible device mapping for placing models on different GPU sets and features a 3D-HybridEngine to reduce memory redundancy and communication overhead during transitions between training and generation phases.
    • HuggingFace Compatibility: Ready integration with popular HuggingFace models.

  4. Explore projects built with verl

    main

    The verl platform is used as a foundation for a wide variety of reinforcement learning (RL) research and production projects. These include:

    Developers can use these projects as reference implementations or inspiration for building their own RL training pipelines.

  5. Hardware Support: NVIDIA, AMD, and Ascend

    main

    verl is designed for high-performance hardware:

    • NVIDIA: Full support.
    • AMD: Supports ROCm GPUs (MI300X / MI325X / MI355X) with FSDP, FSDP2, and Megatron trainer backends. vLLM is the validated inference engine (SGLang support is in progress).
    • Ascend: Supported.

  6. What is the Entropy Mechanism in RL?

    main

    The Entropy Mechanism addresses entropy collapse in Reinforcement Learning (RL) for Large Language Models (LLMs). Entropy collapse occurs when policy entropy drops sharply during training, causing the model to become overconfident and leading to performance saturation.

    Core Concepts

    • Entropy-Performance Relationship: Performance ($R$) is bottlenecked by entropy exhaustion, following the relationship $R = -a \exp(H) + b$.
    • Entropy Drivers: Entropy changes are driven by the covariance between action probability and logit updates. High-probability, high-advantage actions reduce entropy, while rare, high-advantage actions increase it.

    Mitigation Strategies

    To prevent entropy collapse and allow for better exploration, two strategies are proposed that restrict updates for high-covariance tokens:

    1. CLIP-Cov: Restricts updates based on covariance to maintain entropy.
    2. KL-Cov: An alternative approach to restrict updates, which has shown significant performance gains in maintaining higher entropy levels compared to standard GRPO.

  7. What is the Checkpoint Engine and its core APIs

    main

    The Checkpoint Engine is a unified abstraction layer designed to synchronize weights between different training and inference backends. It provides three primary streaming APIs for weight transfer:

    • send_weights: Retrieves named tensors from a generator and sends them in a streaming manner.
    • receive_weights: Returns a tensor generator that yields named tensors in a streaming manner.
    • get_weights: Returns a tensor generator that yields named tensors in a streaming manner. This is typically used when each inference instance needs to update its weights independently from a local cache (such as shared memory or disk).

    These APIs allow for efficient weight synchronization in both on-policy (colocated actor/rollout) and off-policy (disaggregated actor/rollout) training scenarios.

  8. What is Self-Play Fine-Tuning (SPIN)?

    main

    SPIN is a language model finetuning algorithm implemented in verl that enables iterative self-improvement through a self-play mechanism. Instead of relying on fixed external preference datasets, the model learns by playing against itself.

    The SPIN Process:

    1. Synthetic Data Generation: The current model generates responses, creating its own training data from previous iterations.
    2. Two-Player Game Setup: A game involving two players is acted out by a single LLM.
    3. Iterative Training: The model progressively improves by refining its policy, with each iteration's model becoming the opponent for the next iteration.

    In verl, this is realized by using Online DPO (Direct Preference Optimization), where preference data is dynamically updated during training, allowing the model to learn from its own generated data.

  9. What is GSPO (Group Sequence Policy Optimization)

    main

    GSPO is a policy-loss variant within the GRPO family. It is designed to aggregate the Importance Sampling (IS) ratio at the sequence level using the seq-mean-token-mean mode and utilizes a very tight clipping window. This optimization is particularly effective when training large Mixture-of-Experts (MoE) models.

    For theoretical background, refer to the original paper: Group Sequence Policy Optimization.

  10. Overview of ReMax algorithm in verl

    main
    ReMax is a lightweight policy-gradient reinforcement learning method implemented in verl. It is designed to reduce variance without requiring a critic by using a single greedy-decoded baseline response per prompt. This makes it a simple and efficient method for aligning Large Language Models (LLMs).

  11. Overview of On-Policy Distillation Trainer

    main
    The On-Policy Distillation trainer is designed to jointly train a student model using policy-gradient on-policy rollouts and a distillation loss against a frozen teacher model. The teacher model is served by a separate Ray cluster. This approach is intended to close the gap between teacher and student models more effectively than pure Supervised Fine-Tuning (SFT) from teacher generations within the same compute budget.

  12. Overview of RLOO (REINFORCE Leave-One-Out) in verl

    main
    RLOO is a policy gradient baseline implementation that eliminates the need for a critic model. It reduces variance per prompt by computing the advantage for each sample against the average of its sibling samples (the 'leave-one-out' approach). This method is based on the paper Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs.