DOCSEARCHbyAlgolia
Submit Docs

verl: Volcano Engine Reinforcement Learning for LLM

repository·main·Indexed 25 days ago

https://github.com/wyf3/llm_related

A flexible, production-ready reinforcement learning training library for large language models, implementing the HybridFlow framework. It supports RL algorithms like PPO and GRPO, integrates with training backends (FSDP, FSDP2, Megatron-LM) and rollout engines (vLLM, SGLang, HF Transformers), and scales up to 671B parameter models. Compatible with models such as Qwen, Llama, Gemma, and DeepSeek, as well as VLMs like Qwen2.5-vl.

Tokens
164.3K
Snippets
362
Records
745
Agent score
84%

What's inside verl

  1. Overview of pdf2markdown

    main
    pdf2markdown is a tool developed as a secondary development of gptpdf. It replaces the original PDF processing logic with a layout analysis model to identify tables and images. It then utilizes a multimodal large language model (MLLM) to generate content in Markdown format.

  2. Overview of the Entropy Mechanism for LLM Reasoning

    main

    The Entropy Mechanism addresses the "entropy collapse" issue in Reinforcement Learning (RL) for Large Language Models (LLMs). During training, policy entropy often drops sharply, causing overconfidence and performance saturation.

    To mitigate this, the project proposes two strategies to restrict updates for high-covariance tokens:

    1. CLIP-Cov: Restricts updates based on covariance.
    2. KL-Cov: Restricts updates based on KL divergence and covariance.

    These methods help maintain higher entropy levels, allowing the model to explore more freely and achieve better performance on reasoning benchmarks like AIME, MATH, and OlympiadBench.

  3. Overview of the S1 Model implementation

    main
    The S1 model is a lightweight implementation designed to achieve performance comparable to top-tier models like OpenAI's o1 and DeepSeek's R1 in mathematical reasoning and code generation tasks. It is notable for its efficiency, requiring less than $50 in cloud computing costs for training.

  4. Overview of verl: Volcano Engine Reinforcement Learning for LLMs

    main

    verl is a flexible, efficient, and production-ready reinforcement learning (RL) training library designed for large language models (LLMs). It is the open-source implementation of the HybridFlow framework.

    Key capabilities include:

    • Flexible RL Algorithms: Easily implement dataflows like GRPO and PPO using a hybrid-controller programming model.
    • Modular Infrastructure Integration: Seamlessly integrates with existing LLM frameworks such as FSDP, Megatron-LM, vLLM, and SGLang.
    • Efficient Resource Utilization: Supports flexible device mapping for various GPU placements and uses a 3D-HybridEngine to reduce communication overhead during transitions between training and generation phases.
    • Scalability: Capable of scaling up to 671B parameter models across hundreds of GPUs using expert parallelism.

  5. Overview of Self-Play Fine-Tuning (SPIN) Recipe

    main

    The SPIN recipe in verl implements an iterative self-improvement algorithm inspired by the paper "Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models".

    Unlike standard PPO, SPIN uses an online DPO (Direct Preference Optimization) approach where the model learns by playing against itself. The process involves:

    1. Synthetic Data Generation: The current model generates its own responses.
    2. Preference Labeling: Responses are evaluated (e.g., via rule-based ranking for math) to create chosen/rejected pairs.
    3. Iterative Training: The model is updated using compute_online_dpo_loss, with the model from previous iterations potentially serving as the reference baseline.

  6. Overview of verl Megatron-Core (mcore) Models

    main

    verl has migrated from Megatron-LM 0.4 to Megatron-Core (mcore) version 0.11.0. This migration allows the use of the recommended GPTModel class for language models, enabling advanced features like context parallel, expert parallel, and dist_checkpointing.

    Currently supported models include:

    • LlamaForCausalLM
    • Qwen2ForCausalLM

    Supported parallel strategies:

    • tensor parallel
    • pipeline parallel
    • sequence parallel
    • virtual pipeline parallel
    • context parallel

  8. Overview of Entropy Mechanism strategies: CLIP-Cov and KL-Cov

    main

    The Entropy Mechanism addresses entropy collapse in Reinforcement Learning (RL) for Large Language Models (LLMs). Entropy collapse occurs when policy entropy drops sharply, leading to overconfidence and performance saturation.

    To mitigate this, two strategies are proposed to restrict updates for high-covariance tokens:

    1. CLIP-Cov: Restricts updates based on covariance.
    2. KL-Cov: Restricts updates based on KL divergence and covariance.

    These methods help maintain higher entropy levels during training, allowing for better exploration and improved performance on reasoning benchmarks.

  9. Overview of the Char Count task

    main
    Char Count is a simple NLP task designed to help beginners understand Reinforcement Learning from Verifiable Rewards (RLVR). The task involves counting occurrences of a specific character within a word. It is lightweight enough to be trained using a tiny model (e.g., HuggingFaceTB/SmolLM2-135M) on a consumer GPU with only 8GB of VRAM.

  10. Overview of Entropy Mechanism for LLM Reasoning

    main

    This project implements methods to prevent entropy collapse in Reinforcement Learning (RL) for Large Language Models. Entropy collapse occurs when policy entropy drops sharply during training, leading to overconfidence and performance saturation.

    Proposed Strategies

    To alleviate entropy collapse, the project proposes two strategies that restrict updates for high-covariance tokens:

    1. CLIP-Cov: Restricts updates based on covariance.
    2. KL-Cov: Restricts updates based on KL divergence and covariance.

    Key Findings

    • Performance is bottlenecked by entropy exhaustion ($R = -a \exp(H) + b$).
    • Entropy changes are driven by the covariance between action probability and logit updates.
    • These methods allow models to maintain higher entropy, enabling better exploration and improved performance on reasoning benchmarks (e.g., AIME, MATH, OlympiadBench).

  11. Overview of Embedding Distillation

    main
    This project implements embedding distillation where a student model learns from a teacher model. The process involves passing queries, positive samples, and negative samples through both models to obtain scores. The distillation loss is calculated using the KL divergence between the Softmax distributions of the teacher's scores and the student's scores to update the student model.