Dynamic Fine-Tuning (DFT)

The verl interaction system enables dynamic, multi-turn conversational feedback during reinforcement learning (RL) training. It allows models to engage in iterative problem-solving where interaction agents provide corrective feedback, guidance, or evaluation based on model responses.

Key capabilities include:

• Multi-Interaction Support: Multiple named interactions (e.g., different solvers or verifiers) can be managed in a single session, allowing for curriculum learning or domain-specific feedback at the sample level.
• SGLang Integration: Seamlessly integrates with the SGLang rollout system for multi-turn conversations.
• Async Architecture: Uses non-blocking, async-based processing to support distributed training.
• Sample-Level Selection: Each training sample can specify which interaction strategy to use via its configuration.

The PyTorch FSDP Backend provides specialized workers for actor, critic, reference, rollout, and reward models. It is highly recommended for algorithm research and prototyping due to its simplicity and ease of organizing forward/backward computations.

Key Features:

• Model Support: Supports various models. If using hf_weight_loader, any model supported by both Hugging Face and vLLM works without code changes. For other models, you must implement a dtensor_weight_loader for weight synchronization between FSDP and vLLM.
• Sharding Management: Uses FSDPVLLMShardingManager to handle weight resharding between the actor and rollout models.

Limitations:

• Poor scalability for very large models (e.g., Llama 70B or 405B).
• Higher resharding overhead between actor and rollout compared to the Megatron-LM backend.

In Reinforcement Learning for Large Language Models, a common issue is entropy collapse, where policy entropy drops sharply during training. This leads to overconfidence and performance saturation because the model stops exploring.

Research shows that performance ($R$) is bottlenecked by entropy ($H$) following the relationship $R = -a \exp(H) + b$. Entropy changes are driven by the covariance between action probability and logit updates.

To mitigate this, two strategies are proposed to restrict updates for high-covariance tokens:

1. CLIP-Cov: Restricts updates based on covariance.
2. KL-Cov: Restricts updates based on covariance to maintain higher entropy levels.

These methods allow the model to maintain higher entropy throughout training, enabling better exploration and superior performance on reasoning benchmarks compared to standard GRPO.

Sandbox Fusion is a remote code sandbox service that provides a secure environment for running and evaluating code generated by Large Language Models (LLMs).

Using Sandbox Fusion for code verification offers two main benefits:

1. Security: Provides a secure, isolated environment for executing LLM-generated code.
2. Performance: By leveraging remote CPU resources for concurrent code verification, you can reduce the reward stage time by 10-30%, depending on the quality of the generated code.

Dynamic Fine-Tuning (DFT) is a method designed to improve the generalization of Supervised Fine-Tuning (SFT) for Large Language Models (LLMs). It addresses the limited generalization of standard SFT by rectifying the implicit reward structure in SFT gradients.

DFT works by dynamically rescaling the objective function for each token by its predicted probability (using a detached probability to avoid gradient flow). This single-line change allows SFT to achieve performance closer to Reinforcement Learning (RL) while maintaining the simplicity of SFT. It is particularly effective for tasks with non-deterministic solution trajectories, such as mathematical chain-of-thought (CoT) reasoning, complex coding, and multimodal reasoning.

loss = loss * torch.softmax(shift_logits, dim=-1).gather(1, shift_labels.unsqueeze(-1)).squeeze(-1).detach()

SPIN (Self-Play Fine-Tuning) is a language model finetuning algorithm that enables iterative self-improvement through a self-play mechanism. Instead of relying on fixed external preference datasets, the model learns by playing against itself.

The SPIN Loop:

1. Generation: The current model generates multiple responses for each prompt in a batch.
2. Preference Labeling: Responses are evaluated (e.g., using rule-based ranking for math problems) to identify 'chosen' and 'rejected' pairs.
3. Update: The model is updated using compute_online_dpo_loss, comparing the new policy against a reference model.

This process creates a dynamic target data distribution that can potentially allow the LLM to exceed the performance ceiling of fixed human-annotated data.

SPIN is a language model finetuning recipe in verl inspired by the paper "Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models". It enables iterative self-improvement through a self-play mechanism, reducing reliance on external preference datasets.

Core Workflow:

1. Synthetic Data Generation: The current model generates responses to create its own training data.
2. Two-Player Game: A game setup where a single LLM acts as both players.
3. Iterative Training: The model progressively improves by refining its policy, using the model from the previous iteration as the opponent.

OPO is an on-policy reinforcement learning algorithm designed to reduce training instability, such as large policy shifts and entropy collapse. It is particularly useful for encouraging diverse and less repetitive responses.

Key characteristics include:

• Exact On-Policy Training: It always generates responses from the current policy, avoiding the use of pre-generated or off-policy data.
• Optimal Reward Baseline: Instead of using the mean reward of a group (as seen in GRPO), OPO uses a length-weighted reward of the group as the baseline for normalizing rewards.
• Simplified Objective: It omits standard deviation normalization, allowing for the training of a single policy model with the objective of maximizing only the expected reward.

The verl framework supports different backends for training and inference (rollout generation):

Training Backends

• FSDP: Recommended for research, prototyping, and investigating different models/algorithms.
• Megatron-LM: Recommended for users requiring high scalability. Currently supports Megatron-LM v0.11.

Inference (Rollout) Backends

• vLLM: Recommended for stability. For optimal performance, set the environment variable VLLM_USE_V1=1.
• SGLang: Offers advanced features and optimizations; currently under active development.
• TGI (HuggingFace): Primarily used for debugging and single-GPU exploration.