AgentGym-RL

verl is a flexible, efficient, and production-ready Reinforcement Learning (RL) training framework designed for Large Language Model (LLM) post-training. It is an open-source implementation of the HybridFlow paper.

Key Capabilities

• Algorithm Extension: Uses a Hybrid programming model to combine single-controller and multi-controller paradigms, allowing users to build complex RL dataflows with minimal code.
• Modular Integration: Decouples computation and data dependencies to integrate seamlessly with existing LLM infrastructure like PyTorch FSDP, Megatron-LM, and vLLM.
• Scalable Parallelism: Supports flexible device mapping to place models on different GPU sets for efficient resource utilization across various cluster sizes.
• High Throughput: Achieves state-of-the-art throughput by integrating SOTA training/inference frameworks and utilizes a 3D-HybridEngine for efficient actor model resharding, which reduces memory redundancy and communication overhead during transitions between training and generation phases.

The verl library implements the HybridFlow architecture, which treats Reinforcement Learning (RL) as a two-level dataflow problem. This design decouples the high-level algorithm logic from the low-level neural network computations to balance performance and flexibility.

The Two Levels of Dataflow

1. Control Flow: Defines the high-level execution of the RL algorithm (e.g., in PPO: Rollout $\rightarrow$ Advantage Computation $\rightarrow$ Training). This defines the core logic of the RL algorithm.
2. Computation Flow: Defines the neural network operations (e.g., model forward, backward, optimizer steps). This is the standard tensor-level computation.

The Hybrid Approach

Unlike unified multi-controller systems that couple control and computation flows, verl uses a separated flow strategy:

• Single-process Control Flow: The controller (the RL algorithm logic) runs on a single process. This makes it simple to implement new RL algorithms by just defining a new control flow.
• Multi-process Computation Flow: The heavy lifting (generator/actor workers, critic workers) runs on multiple processes across specific resource groups. This allows for high-performance, scalable neural network training (e.g., using FSDP or Megatron).

Trade-off: While this separation introduces some data communication overhead when the controller interacts with workers, it provides significantly higher reusability. You can swap computation engines (e.g., from FSDP to Megatron) without rewriting your RL algorithm's control logic.

In a verl reinforcement learning job, execution is distributed between a central controller and specialized workers:

1. Controller (Single Process): Orchestrates the entire RL algorithm. It manages the sequence of operations and coordinates data movement between different worker groups.
2. Workers (Multi-process): Specialized groups of processes assigned to specific tasks:
   • Generator/Actor Workers: Perform rollout/sample generation.
   • Critic Workers: Perform value estimation/critic updates.

Data Flow Pattern:

• The controller sends data/instructions to a worker group (e.g., sending prompts to the Generator for rollout).
• The worker group performs the high-level computation (e.g., generating samples).
• The worker group sends the resulting data back to the controller.
• The controller then uses that data to trigger the next step in the algorithm (e.g., passing samples to the Critic or Actor for training).

To save redundant CUDA and distributed context overhead, you can merge different roles (e.g., actor, critic, reference, reward model) into the same process using create_colocated_worker_cls.

Important Considerations:

• Resource Pools: If you need different resource pools (and thus different parallel sizes) for each role, do not use create_colocated_worker_cls. Instead, pass different resource pools directly to different worker groups.
• Megatron Backend: When colocating roles in the same process, all roles will share the same 3D parallel size. If you require different 3D parallel sizes for different roles, you must initialize each role's worker_group separately (as shown in the non-colocated example) rather than using the colocation optimization.
• Rollout Initialization: It is recommended to create the actor_rollout worker group last to allow vllm to better estimate KV cache memory requirements.

# initialize WorkerGroup
all_wg = {}
for resource_pool, class_dict in self.resource_pool_to_cls.items():
    worker_dict_cls = create_colocated_worker_cls(class_dict=class_dict)
    wg_dict = self.ray_worker_group_cls(resource_pool=resource_pool, ray_cls_with_init=worker_dict_cls)
    spawn_wg = wg_dict.spawn(prefix_set=class_dict.keys())
    all_wg.update(spawn_wg)

if self.use_critic:
    self.critic_wg = all_wg['critic']
    self.critic_wg.init_model()

if self.use_reference_policy:
    self.ref_policy_wg = all_wg['ref']
    self.ref_policy_wg.init_model()

if self.use_rm:
    self.rm_wg = all_wg['rm']
    self.rm_wg.init_model()

# we should create rollout at the end so that vllm can have a better estimation of kv cache memory
self.actor_rollout_wg = all_wg['actor_rollout']
self.actor_rollout_wg.init_model()