RewardBench

RewardBench provides two primary YAML configuration files for managing experiment environments and model evaluations:

1. beaker_eval.yaml: Used for setting up the compute environment when using internal AI tooling (Beaker).
2. eval_configs.yaml: Contains the specific model configurations required to reproduce results when running run_rm.py or run_dpo.py scripts.

The repository is organized as follows:

• rewardbench/: Core utilities and modeling files.
  • rewardbench/models/: Standalone files for running existing reward models.
  • rewardbench/*.py: Core RewardBench tools and utilities.
• analysis/: Tools for analyzing RewardBench results or other reward model properties.
• scripts/: Scripts and configuration files used to evaluate reward models.
• tests/: Unit tests for the package.
• setup.py: Allows for editable installation via pip install -e ..

Two Docker images are provided for different research environments:

1. rewardbench: Best for evaluating reward models and API-based LLM judges. Uses Dockerfile and builds quickly (~5-10 min).
2. rewardbench-vllm: Best for local LLM inference via vLLM. Uses Dockerfile.vllm and takes longer to build (~45 min) because it builds flash-attn from source.

Note: The base image uses torch <= 2.8, while the vLLM image uses torch 2.9.

# Base image (fast)
docker build -t rewardbench . --platform linux/amd64

# vLLM image (slow, includes local LLM inference)
docker build -f Dockerfile.vllm -t rewardbench-vllm . --platform linux/amd64

To allow others to compare results directly and to enable result verification, you can contribute custom model loading or pipeline code to RewardBench. This is preferred over submitting raw JSON results.

To contribute, follow these steps:

1. Implement Model Logic: Provide a model loading function (handling quantization and the forward implementation) and/or a pipeline function. The pipeline must follow HuggingFace's style: it should accept raw, un-tokenized text and return a scalar output (or a supported dict containing the classifier).
2. Register the Model: Add your model configuration to the REWARD_MODEL_CONFIGS dictionary located in rewardbench/models/__init__.py.
3. Verify Implementation: Run your model using the --debug flag on run_rm.py to ensure inference works correctly.

Note: Please include links to the original implementation in your source code.

The CLI supports advanced logging to upload model outputs and accuracy scores to the HuggingFace Hub. This is useful for tying results to your models for rejection sampling.

Upload scores and metadata: Use --push_results_to_hub to upload a dataset of scores/correctness and --upload_model_metadata_to_hf to add results directly to the model card.

rewardbench --model vwxyzjn/reward_modeling__EleutherAI_pythia-14m --batch_size 128 --tokenizer=EleutherAI/pythia-14m --push_results_to_hub --upload_model_metadata_to_hf --chat_template raw

Note: Full metadata functionality currently works best with DPO models for preference datasets.

Generative models (local via vLLM or API-based like OpenAI/Anthropic) can be evaluated using the rewardbench-gen CLI or the scripts/run_generative.py script.

Using the CLI:

rewardbench-gen --model={yourmodel}

Using the script (Local vs API):

• API (e.g., OpenAI): python scripts/run_generative.py --model=gpt-3.5-turbo-0125
• Local (via vLLM): python scripts/run_generative.py --model=meta-llama/Llama-3-70b-chat-hf --force_local
• Together API: python scripts/run_generative.py --model=meta-llama/Llama-3-70b-chat-hf

Generative Ensembles: You can run an ensemble of generative models (must be an odd number > 1) via API:

python scripts/run_generative.py --model gpt-3.5-turbo-0125 claude-3-sonnet-20240229 meta-llama/Llama-3-70b-chat-hf

rewardbench-gen --model={}

To run evaluations specifically for RewardBench 2, use the scripts/run_v2.py script. This script includes special handling for best-of-4 and Ties data.

Standard V2 evaluation:

python scripts/run_v2.py --model={yourmodel}

Generative models in V2: Generative models can use a rankings-based prompt (default, comparing 4 responses) or a ratings-based prompt (scoring responses separately). Use the --score_w_ratings flag for the latter. For the Ties subset, the code enforces the ratings setting.

python scripts/run_generative_v2.py --model={yourmodel}

You can install RewardBench using uv (recommended) or pip. If you intend to use generative models (LLM-as-judge, vLLM, or API providers), you must install the [generative] extra.

Using UV:

uv pip install rewardbench
# For generative models
uv pip install rewardbench[generative]

Using pip:

pip install rewardbench
# For generative models
pip install rewardbench[generative]

For development (install from source):

git clone https://github.com/allenai/reward-bench.git
cd reward-bench
uv sync                      # base install
uv sync --extra generative   # with generative support

For standard reward models (classifiers), RewardBench loads the model using a pipeline_builder (typically transformers.pipeline with text-classification) and evaluates it on the dataset.

Key behaviors:

• Quantization: If the model configuration specifies quantization, it uses load_in_8bit. Note that device_map="auto" is not supported for bitsandbytes quantized models; the tool handles this by mapping to the current device.
• Padding: The tokenizer's padding_side is set to left for determinism. If --force_truncation is passed, truncation_side is also set to left.
• Scoring: The tool extracts scores from the pipeline output. If the pipeline returns a dictionary (e.g., {'score': 0.68}), it extracts the score key; otherwise, it converts the raw tensor to a list of floats.

When using run_judge_pair, you can pass a model_modifier to use specialized prompt templates designed for specific evaluation frameworks. This ensures the LLM judge follows the expected output format for that framework.

Supported modifiers:

• prometheus: Uses REL_SYSTEM_PROMPT and RELATIVE_PROMPT (requires single-turn).
• Con-J: Uses a Chinese-language JSON-based prompt.
• RISE-Judge: Uses a specific Chinese-language analysis prompt.
• offsetbias: Uses a prompt designed to mitigate position bias.
• Atla: Uses a specific reasoning/result format.
• gemini: Special handling for Gemini models that prepends instructions to the user prompt and removes the system prompt.