DOCSEARCHbyAlgolia
Submit Docs

OpenR Framework

repository·main·Indexed 23 days ago

https://github.com/openreasoner/openr

An open-source framework for advanced reasoning with Large Language Models (LLMs), focusing on process supervision, reinforcement learning, and test-time computation. It includes tools for process-supervision data generation via OmegaPRM, online policy training (APPO, GRPO, TPPO), and various search strategies such as MCTS, Beam Search, and rStar. The framework also provides utilities for LaTeX to SymPy conversion and supports training and deploying Process Reward Models (PRMs).

Tokens
10.9K
Snippets
24
Records
65
Agent score
83%

What's inside OpenR

  1. Overview of OpenR Framework

    main

    OpenR is an open-source framework designed for advanced reasoning with Large Language Models (LLMs). It provides tools for process-supervision data generation, online policy training, and various search strategies to enhance mathematical and logical reasoning capabilities.

    Key Capabilities

    • Process-supervision Data Generation: Uses OmegaPRM to improve mathematical reasoning through automated process supervision.
    • Online Policy Training: Supports multiple reinforcement learning algorithms including APPO, GRPO, and TPPO (located in train/mat/trainers).
    • PRM Training: Supports both supervised training for Process Reward Models (in prm/code) and Generative Reward Model training (via gen_rm/).
    • Search Strategies: Implements multiple reasoning strategies such as Greedy Search, Best-of-N, Beam Search, MCTS, rStar, and Critic-MCTS.

  2. Overview of OpenR

    main

    OpenR is an open-source framework focused on advancing the reasoning capabilities of Large Language Models (LLMs). It provides tools for process supervision, online policy training, and various search strategies to improve mathematical and logical reasoning.

    Key Features:

    • Process Supervision Data Generation: Includes implementations like OmegaPRM to improve mathematical reasoning via automated process supervision.
    • Online Policy Training: Uses Process Reward Models (PRM) for online Reinforcement Learning (RL) training.
    • PRM Training: Supports supervised training for PRMs and Direct GenRM (Generative Reward Model) training.
    • Search Strategies: Supports multiple strategies including Greedy Search, Best-of-N, Beam Search, MCTS, rStar, and Critic-MCTS.
    • Test-time Computation: Research and tools related to scaling laws and test-time compute.

  3. Understand OpenR benchmark evaluation dimensions

    main

    OpenR evaluates model performance using a multi-dimensional matrix. When reviewing benchmark results, the following dimensions are used to categorize the performance of different reasoning configurations:

    • Generator: The base language model being evaluated (e.g., Qwen2.5-Math-1.5B-Instruct, Skywork-o1-Open-Llama-3.1-8B).
    • Reward Model: The model used to score or guide the reasoning process (e.g., Math-Sphered-Mistral-7B-PRM, Math-psa-7B, Skywork-o1-PRM-7B).
    • Search: The algorithm used to navigate the reasoning space (e.g., Greedy, Majority Vote, Best-of-N, Beam Search, MCTS).
    • Budget: The computational resource allocated to the search process, often expressed in powers of 2 (e.g., $2^0$, $2^6$).
    • MATH: The specific benchmark score achieved (e.g., performance on the MATH dataset).

  4. Understand OmegaPRM_v2 generated data formats

    main

    OmegaPRM_v2 supports two output formats depending on the --save_data_tree flag:

    Flat Format

    Trigger: --save_data_tree=False Provides a linear view of the reasoning process. Each entry follows this structure:

    { "solution_prefix": [Q, x_1:x_i], "mc_value": 0.5 }

    where i represents the number of reasoning steps.

    Tree Format

    Trigger: --save_data_tree=True Organizes data into a hierarchical tree structure. Each node contains:

    • text: The cumulative reasoning from the root node up to that specific step.
    • mc_value: The Monte Carlo score for the reasoning progression up to that step.
    • children: A list of child nodes branching from the current node.

  6. Configure and start LM & RM services

    main

    To run inference, you must first configure the service scripts located in reason/llm_service/.

    Modify the following variables in the scripts to match your local setup:

    • $MODEL_BASE: The directory where your models are stored.
    • $POLICY_MODEL_NAME: The name of the policy model.
    • $VALUE_MODEL_NAME: The name of the value model.
    • $NUM_LM_WORKER: Number of language model (LM) workers to start.
    • $NUM_RM_WORKER: Number of reward model (RM) workers to start.

    To start the services (e.g., for Math Shepherd), run the provided shell script. To stop the services, use tmux kill-session on the session name (default is FastChat).

  7. Start LM & RM Services

    main

    Before running inference, you must start the Language Model (LM) and Reward Model (RM) services.

    1. Configure the number of workers and model paths by editing reason/llm_service/create_service_math_shepherd.sh.
    2. Set the NUM_LM_WORKER and NUM_RM_WORKER environment variables.
    3. Set the model_path configuration.
    4. Execute the service script.

    Note: Ensure PYTHONPATH is set to the current directory.

    export PYTHONPATH=$(pwd)
sh reason/llm_service/create_service_math_shepherd.sh

  9. Fine-tune PRM (Process Reward Model)

    main

    Navigate to prm/code to perform PRM learning. You can run fine-tuning on a single GPU or using multiple GPUs via torchrun.

    Single GPU usage: Requires --model_path, --train_data_path, and --test_data_path.

    Multi-GPU usage: Requires --model_path and --data_path. Use the --datasets both flag to specify dataset types.

    # single gpu
python finetune_qwen_single_gpu.py --model_path $YOUR_MODEL_PATH \
                                   --train_data_path $TRAIN_DATA_PATH \
                                   --test_data_path $TEST_DATA_PATH

# multi gpu
torchrun --nproc_per_node=2 finetune_qwen.py --model_path $YOUR_MODEL_PATH \
                                             --data_path $YOUR_DATA_FOLDER_PATH \
                                             --datasets both

  10. Run OmegaPRM on multiple GPUs

    main

    To process large datasets in parallel, you can split your input data and run OmegaPRM across multiple GPUs using a shell script.

    1. Split the input data: Use process_json.py to divide your JSON file into segments corresponding to the number of GPUs you intend to use.
    2. Configure the runner: Edit run_omegaprm_multi_gpu.sh to set your MODEL_NAME (path to Hugging Face or vLLM model), MODEL_TYPE (hf or vllm), and other hyperparameters like MAX_NEW_TOKENS and TEMPERATURE.
    3. Execute: Run the shell script to start processing. Results are saved in the output_results directory.

  11. Run LLM training

    main

    To run training, you must first modify the following variables in train/mat/scripts/train_llm.sh:

    • $dataset_path
    • $model_name_or_path
    • $prm_name_or_path

    Then, execute the training script from the train/mat/scripts directory.

    cd train/mat/scripts
bash train_llm.sh

  12. Configure and Start LM and RM Services

    main

    To run inference, you must first configure the environment variables in the scripts located in reason/llm_service/.

    Required Variables:

    • $MODEL_BASE: The directory path where your models are stored.
    • $POLICY_MODEL_NAME: The name of the policy model you wish to use.
    • $VALUE_MODEL_NAME: The name of the Value model you wish to use.
    • $NUM_LM_WORKER: The number of Language Model (LM) processes to start.
    • $NUM_RM_WORKER: The number of Reward Model (RM) processes to start.

    Starting Services: To start the Math Shepherd model's LM and RM services, run the provided shell script.

    Stopping Services: If you are using tmux, you can kill the session (the default session name is FastChat) using: tmux kill-session -t {Your Session Name}