DOCSEARCHbyAlgolia
Submit Docs

OpenRLHF Documentation

repository·main·Indexed 27 days ago

https://github.com/openrlhf/openrlhf

A high-performance, production-ready open-source RLHF framework combining Ray and vLLM distributed architecture. It supports scalable reinforcement learning from human feedback using algorithms such as PPO, REINFORCE++, GRPO, and RLOO. The framework features a unified agent-based design for single-turn and multi-turn execution, supports SFT, DPO, and reward model training, and includes a Molt backend for scaling to hundreds of billions of parameters.

Tokens
16.2K
Snippets
16
Records
76
Agent score
92%

What's inside OpenRLHF

  1. Overview of OpenRLHF Architecture

    main

    OpenRLHF is a high-performance, production-ready open-source RLHF framework designed for scalable human feedback reinforcement learning. It is built on a distributed architecture combining Ray, vLLM, and DeepSpeed to efficiently orchestrate multiple components across GPUs.

    Core Infrastructure Components

    • Ray: Acts as the distributed scheduler and controller, distributing Actor, Reward, Reference, and Critic models across GPUs. It supports training models with 70B+ parameters.
    • vLLM: A high-performance inference engine used for sample generation. It utilizes Auto-Tensor Parallelism (AutoTP) and Pipeline Parallelism (PP) to provide high-throughput, memory-efficient generation.
    • DeepSpeed: Provides memory-efficient training via ZeRO-3, deepcompile, AutoTP, and RingAttention, allowing for large model training with seamless HuggingFace integration.
    • Transformers: Native integration with HuggingFace Transformers for model loading, state management, and fine-tuning.
    • NCCL / CUDA IPC: Enables high-speed inter-GPU communication for distributed training and inference.

  2. Agent Execution Modes: Single-turn vs Multi-turn

    main

    OpenRLHF provides two primary agent execution modes:

    1. Single-turn Agent (Default): Best for reinforcement fine-tuning using custom reward functions without requiring a pre-trained reward model.
    2. Multi-turn Agent: Designed for tasks requiring multi-step interactions, such as chain-of-thought reasoning, coding with feedback, or gameplay.

  3. New Backend: Molt for Large-Scale Training

    main
    OpenRLHF supports Molt as a new backend. Molt brings an Automodel-powered backend that is more powerful than DeepSpeed, allowing users to scale RL training to hundreds of billions of parameters while maintaining the standard OpenRLHF workflow.

  4. Understand OpenRLHF Agent-Based Execution Modes

    main

    OpenRLHF uses a unified agent-based paradigm where RL algorithms are decoupled from execution modes. You can combine any supported algorithm (PPO, GRPO, etc.) with either Single-Turn or Multi-Turn execution.

    Single-Turn Mode (Default)

    • Use Case: Standard RLHF and custom reward functions.
    • Interface: Uses an optional reward_func().
    • Complexity: Low (Default for 99% of use cases).

    Multi-Turn Mode

    • Use Case: Multi-step reasoning or interactive environments.
    • Interface: Uses reset() and step() methods.
    • Complexity: Advanced.

  5. Implement DAPO (Dynamic Adaptive Preference Optimization)

    main

    DAPO uses dynamic sampling to filter responses based on scores returned by a reward function or agent.

    Requirements:

    • --rollout.n_samples_per_prompt > 1
    • Must specify either --reward.remote_url (reward function) or --train.agent_func_path (agent).

    Configuration:

    • Enable: --algo.dynamic_filtering_enable
    • Set score range: --algo.dynamic_filtering_range 0.0 1.0

  6. Use Single-turn and Multi-turn Agent RL training modes

    main

    OpenRLHF features an Agent-based execution paradigm that decouples algorithms from execution modes.

    Single-turn Mode (Default)

    Used for 99% of use cases where each prompt is generated once. Compatible with all RL algorithms.

    • Custom Reward Functions: Use --reward.remote_url to point to a remote reward service.
    • Hybrid Engine: Use hybrid engine scripts (e.g., train_ppo_ray_hybrid_engine.sh) to maximize GPU utilization.

    Multi-turn Mode (Advanced)

    Used for interactive tasks requiring multi-step interaction with an environment.

    • Custom Agent Functions: Define your logic in a Python file and pass it via --train.agent_func_path.
    • OpenAI Compatible Servers: You can wrap vLLM as a local OpenAI server and use it as an agent executor.
    • Asynchronous Pipeline: Enable --train.async_enable to improve throughput during multi-turn training.

  7. Implement Custom Reward Functions and Agents

    main

    You can extend the training pipeline by providing custom logic for rewards or agent interactions:

    • Custom Reward Functions: Use the --reward.remote_url flag to point to a remote reward service.
    • Custom Agent Functions: For multi-turn/interactive tasks, use the --train.agent_func_path flag to specify the path to your custom agent function.
    • OpenAI-compatible Server: You can wrap vLLM as a local OpenAI Agent Server using the executor found in examples/python/agent_func_openai_server_executor.py.

  9. Install OpenRLHF

    main

    You can install OpenRLHF via pip or from source. Using Docker is recommended for a hassle-free setup. For optimal performance, it is recommended to use vLLM 0.22.1+.

    # Recommended: Docker setup
docker run --runtime=nvidia -it --rm --shm-size="10g" --cap-add=SYS_ADMIN \
  -v $PWD:/openrlhf nvcr.io/nvidia/pytorch:26.03-py3 bash

# Clean conflicting packages
sudo pip uninstall xgboost transformer_engine flash_attn pynvml -y

# Install OpenRLHF (choose one)
pip install openrlhf                    # Basic
pip install openrlhf[vllm]              # + vLLM 0.22.1 (recommended)
pip install openrlhf[vllm_latest]       # + Latest vLLM
pip install openrlhf[vllm,ring,liger]   # + All optimizations

# Alternative: Install from source
git clone https://github.com/OpenRLHF/OpenRLHF.git
cd OpenRLHF
pip install -e .

  10. Configure Single-Turn RLHF training

    main

    For standard RLHF tasks, use Single-Turn mode. You can implement custom reward functions by providing a --reward.remote_url. To maximize GPU utilization, use the hybrid engine.

    Common scripts for single-turn:

    • PPO: examples/scripts/train_ppo_ray_hybrid_engine.sh
    • REINFORCE++/GRPO/RLOO: examples/scripts/train_reinforce_baseline_hybrid_engine.sh
    • Custom Reward: examples/scripts/train_ppo_with_reward_fn.sh

  11. Understand the Agent-Based Execution Paradigm

    main

    OpenRLHF utilizes a unified Agent-Based Execution paradigm built on top of the Ray distributed architecture. This ensures that all training runs—whether standard PPO or complex multi-turn reasoning—follow a consistent execution flow using a Token-in-Token-out approach.

    Key Design Principles

    • Token-in-Token-out: All sampling produces token-level trajectories, ensuring zero text-level mismatch.
    • Unified Interface: All modes use the same AgentExecutorBase API, allowing users to switch modes easily.
    • Algorithm Agnostic: RL algorithms (e.g., PPO, REINFORCE++, GRPO, RLOO) are decoupled from the Agent executors, meaning any algorithm can be paired with any execution mode.
    • Scalable: Supports easy insertion of custom rewards or external environments.

    Execution Modes

    Execution modes are orthogonal to the RL algorithms. You can combine any algorithm with either of these modes:

    ModeUse CaseInterfaceComplexity
    Single-TurnStandard RLHF, custom reward functionsOptional reward_func()⭐ Default (99% of use cases)
    Multi-TurnMulti-step reasoning, interactive environmentsreset() + step()⭐⭐ Advanced

  12. Choose between Single-Turn and Multi-Turn execution modes

    main

    OpenRLHF uses an agent-based execution paradigm that is orthogonal to the chosen RL algorithm. You can switch between two primary modes:

    1. Single-Turn Mode (Default, 99% of use cases): Used for standard RLHF and custom reward functions. It performs one generation per prompt.
    2. Multi-Turn Mode (Advanced): Used for multi-step reasoning or interactive environments. It uses reset() and step() interfaces to interact with environments (e.g., OpenAI Agent Server).