DOCSEARCHbyAlgolia
Submit Docs

EasyVideoR1

repository·main·Indexed 41 days ago

https://github.com/cyuq1n/easyvideor1

A specialized Reinforcement Learning (RL) framework for video understanding post-training of multimodal LLMs. It provides research-friendly interfaces for mixed-modality training, optimized video data training efficiency, and a comprehensive evaluation pipeline including support for Qwen-VL models and LLM-based judging for open-ended results.

Tokens
25.3K
Snippets
51
Records
94
Agent score
99%

What's inside EasyVideoR1

  1. Configure batch balancing and micro-batching parameters

    main

    To optimize GPU utilization and prevent idle time during RL training, EasyVideoR1 uses specific mechanisms for batching:

    • Sequence Balancing: The framework uses the Karmarkar-Karp algorithm to ensure each GPU receives a similar total token count, even when sequence lengths vary significantly.
    • Micro-batching: During the Actor update phase, sequences are further divided into micro-batches based on the max_token_len_per_gpu parameter. This ensures that the memory footprint of the forward and backward passes remains within GPU limits.

    Key Parameters for Tuning:

    • rollout_batch_size: The number of prompts used for generation.
    • n: The number of responses generated per prompt (used for GRPO group calculation).
    • max_token_len_per_gpu: The threshold used to split sequences into micro-batches during training (e.g., 32768).

  2. Calculate Qwen3-VL visual tokens

    main

    To estimate the number of visual tokens generated by the Qwen3-VL encoder, use the following formulas based on the fixed architecture parameters:

    Fixed Architecture Parameters (Do not modify):

    • patch_size: 16
    • spatial_merge_size: 2
    • temporal_patch_size: 2

    Formulas:

    • Tokens per frame: (H / 32) * (W / 32)
    • Total video tokens: (frames / 2) * (H / 32) * (W / 32)

    Note: The factor of 32 comes from patch_size (16) * spatial_merge_size (2). The factor of 2 in the frame count comes from temporal_patch_size.

    # Example calculation
# Configuration: video_max_pixels: 262144 (512x512), video_max_frames: 128

# Tokens per frame = (512 / 32) * (512 / 32) = 16 * 16 = 256 tokens/frame
# Total video tokens = (128 / 2) * 256 = 64 * 256 = 16384 tokens

  3. Understand the RL training step lifecycle in EasyVideoR1

    main

    A single training step in EasyVideoR1 follows an 8-phase lifecycle managed by the fit() method in verl/trainer/ray_trainer.py. The process transitions between vLLM for high-throughput generation and FSDP for gradient-based optimization.

    Training Phases:

    1. Rollout (Generation): Loads vLLM, synchronizes Actor FSDP weights to the vLLM engine, and generates n responses per prompt using sampling (e.g., temperature=1.0).
    2. Balance Batch: Uses the Karmarkar-Karp partitioning algorithm to distribute sequences across GPUs such that the total token count per GPU is approximately equal, preventing the 'bottleneck effect'.
    3. Reward Calculation: Executes reward functions asynchronously via Ray remote() calls to avoid blocking GPU computation. Rewards are determined by problem_type (e.g., math, multiple choice, code, or open-ended).
    4. Old Log Probs Calculation: Performs a forward pass using the FSDP Actor model to get precise log probabilities for importance sampling (since vLLM sampling is stochastic).
    5. Reference Log Probs Calculation: (Optional) Performs a forward pass using a frozen reference model to calculate KL divergence, preventing the policy from drifting too far from the initial model.
    6. Advantage Calculation: Computes GRPO advantages on the Driver. For a group of n responses to the same prompt, advantages are calculated as (rewards - mean) / std.
    7. Actor Update (PPO): The only phase involving gradient computation. It uses micro-batching (controlled by max_token_len_per_gpu) and performs forward/backward passes to update the Actor via PPO clip loss.
    8. Validation and Checkpointing: Periodically runs validation (with n=1 and lower temperature) and saves Actor weights, optimizer states, and DataLoader states.
    # Entry point in verl/trainer/ray_trainer.py
self.fit()

  4. How the FSDP and vLLM memory rotation mechanism works

    main

    Because the Actor (FSDP) and Rollout (vLLM) share the same GPUs, the framework implements a rotation mechanism to switch between generation and training phases, preventing VRAM overflow.

    Phase 1: vLLM Generation (Rollout)

    1. Offload: Actor FSDP parameters are offloaded to CPU.
    2. Sync: Actor weights are synchronized to vLLM.
    3. Initialize: vLLM initializes the KV Cache.
    4. Generate: $n$ answers are generated. VRAM usage: ~80-100 GB

    Phase 2: FSDP Training (Update)

    1. Release: vLLM weights are offloaded (release_rollout_engine()).
    2. Reload: Actor FSDP parameters are loaded back into GPU.
    3. Compute: Forward/backward propagation occurs.
    4. Optimize: Optimizer updates the weights. VRAM usage: ~5-10 GB

    Key Implementation Hooks

    • prepare_rollout_engine(): Handles steps 1-3 (found in fsdp_workers.py:611).
    • release_rollout_engine(): Handles step 5 (found in fsdp_workers.py:615).
    • Automatic Loading: Actor parameters are automatically reloaded during compute_log_probs() or update_actor() calls.

  5. Identify the model roles in RL training

    main

    During RL training, the framework manages four distinct roles. In EasyVideoR1, the Actor, Rollout, and Reference roles are colocated on the same GPUs to optimize resource usage.

    RoleDescriptionTrainable?
    Actor (Policy Model)The model currently being trained; responsible for generating answers.Yes
    Rollout EngineUses vLLM to efficiently generate answers using Actor weights.No (shares weights with Actor)
    Reference ModelA frozen snapshot of the model from the start of training, used to calculate KL divergence.No (frozen)
    Reward (Reward Function)Assigns scores to answers, typically via rule-based functions.No

  6. Understand the GRPO algorithm in EasyVideoR1

    main

    EasyVideoR1 uses GRPO (Group Relative Policy Optimization) as its core Reinforcement Learning algorithm. Unlike traditional PPO, GRPO does not require a Critic model (value network), which significantly reduces VRAM usage.

    How GRPO works:

    1. Group Generation: For a single question, the model generates $n$ different answers (a Group).
    2. Reward Scoring: Each answer in the group is assigned a reward score.
    3. Relative Advantage: Advantages are calculated based on the group's relative performance: Advantage_i = (reward_i - mean(rewards_in_group)) / std(rewards_in_group)
    4. Policy Update: The model is updated to increase the probability of high-advantage answers and decrease the probability of low-advantage ones.

    Key Loss Formula (Simplified): Loss = -E[ min(ratio × advantage, clip(ratio, 1-ε, 1+ε) × advantage) ] where ratio = exp(new_log_prob - old_log_prob).

  7. Understand the Qwen3-VL Multimodal Data Flow

    main

    The Qwen3-VL multimodal processing pipeline is split across three distinct stages to manage memory efficiency. Instead of passing heavy frame data through the entire pipeline, the system passes video paths and metadata, re-processing the actual frames at each stage.

    1. Dataset Stage (verl/utils/dataset.py): Pre-processes videos using process_video and performs tokenization. It outputs input_ids (containing <|video_pad|> tokens) and multi_modal_data which contains only the video paths, not the frame data. This prevents excessive memory usage during data loading.
    2. Rollout Stage (verl/workers/rollout/vllm_rollout_spmd.py): Uses vLLM for inference. It retrieves configuration from meta_info, re-processes the video paths into (frames, metadata) tuples, and passes them to vLLM. It sets do_resize=False and do_sample_frames=False to avoid redundant processing.
    3. Training Stage (verl/workers/fsdp_workers.py): Uses FSDP for training. It re-processes video paths into frames and metadata, then uses the video_processor with do_resize=False and do_sample_frames=False to ensure the frames match the pre-processed dimensions.

  8. Understand memory distribution and bottlenecks

    main

    Memory usage in EasyVideoR1 is split between CPU (Ray Object Store) and GPU VRAM, with different profiles depending on the training phase.

    CPU Memory (Ray Object Store)

    • Bottleneck: Video frame data (multi_modal_data) is the primary driver of CPU memory usage (~200-400 MB per video sample).
    • Scaling: CPU memory usage is directly related to rollout_batch_size. Large batches combined with Ray serialization during dispatch can lead to massive memory spikes (e.g., 200+ GB).

    GPU VRAM

    • Rollout Phase (vLLM): This is the highest memory consumption phase (~80-100 GB per GPU). It is driven by max_token_len_per_gpu and micro_batch_size, not by rollout_batch_size.
    • Inference Phases (compute_log_probs / compute_ref_log_probs): Much lower usage (~3-5 GB per GPU). Consists of FSDP parameter shards, all-gather temporary weights, batch data, and activations.
    • Training Phase (update_actor): Low usage (~4-6 GB per GPU). Consists of FSDP shards, all-gather weights, forward activations (with gradient checkpointing), and gradients. Optimizer states are offloaded to CPU to save VRAM.

  9. Ensure consistency in Qwen3-VL multimodal data processing

    main

    When training Qwen3-VL models in the EasyVideoR1 framework, you must ensure that the video/image processing parameters are identical across all three stages of the training pipeline. Failure to do so will result in a mismatch between the number of tokens in the text sequence and the number of visual features produced by the encoder.

    The three stages are:

    1. Dataset Stage: Reads video/images, performs preprocessing + tokenization, and generates input_ids.
    2. Rollout Stage: vLLM inference generates responses (requires re-processing video).
    3. Training Stage: FSDP training updates weights (requires re-processing video).

    Critical Requirement: You must use the exact same min_pixels, max_pixels, and video_fps in all three stages.

  10. Configure FSDP for distributed training

    main

    The framework uses Fully Sharded Data Parallel (FSDP) for training phases (e.g., compute_log_probs, update_actor). FSDP shards model parameters across all available GPUs (e.g., 128 cards) to reduce per-card memory requirements.

    FSDP Lifecycle per layer:

    1. All-gather: GPUs exchange shards to reconstruct the full layer parameters.
    2. Forward: Computation is performed using the full layer parameters.
    3. Release: The reconstructed parameters are discarded, leaving only the local shard to save memory.

    Note: Unlike the generation phase which uses Tensor Parallel (TP), the training phase relies on FSDP for parameter sharding and Data Parallel (DP) for processing batches.

  11. Calculate Qwen3-VL video tokens

    main

    The number of tokens generated for a video depends on the resolution (H, W), the number of frames (T), and the model's patch/merge settings.

    Model Constants:

    • patch_size = 16 (16x16 patches)
    • spatial_merge_size = 2 (2x2 patches merged into 1 token)
    • temporal_patch_size = 2 (2 consecutive frames merged into 1 token)

    Calculation Formulas:

    • Tokens per frame = (H / 32) * (W / 32)
    • Total video tokens = (Frames / 2) * (H / 32) * (W / 32)

    Example Calculation: If max_pixels = 131072 (approx. 362x362), and a video has 36 frames at 256x480 resolution:

    1. Tokens per frame: (256 / 32) * (480 / 32) = 8 * 15 = 120
    2. Total video tokens: (36 / 2) * 120 = 18 * 120 = 2160

  12. Understand EasyVideoR1 project structure

    main

    The repository is organized into several functional areas:

    • verl/: The core RL training framework, including trainers, workers (Actor, rollout, reward, critic), model support (Qwen series), and FSDP utilities.
    • examples/: Contains ready-to-use pipelines:
      • video_rl/: Video-only RL training with a single-file reward.
      • unified_rl/: Mixed image-video training with modular rewards.
    • eval/: The evaluation toolkit supporting 25+ benchmarks.
    • scripts/: Utility scripts for checkpoint merging and video preprocessing.
    • docs/: Detailed technical documentation.