ScienceAgentBench

ScienceAgentBench supports OpenAI models (gpt-4o, o1-preview) and Amazon Bedrock models (llama-3.1, mistral, claude).

For OpenAI:

export OPENAI_API_KEY={YOUR_OPENAI_KEY}

For Amazon Bedrock: Configure your ~/.aws/config file:

[default]
aws_access_key_id = {YOUR_AWS_ID}
aws_secret_access_key = {YOUR_AWS_KEY}
region=us-west-2

You can evaluate an agent's generated programs using the evaluation.harness.run_evaluation module. This module uses Docker to run each task in an independent container, preventing package conflicts and allowing for parallel execution.

Before running, ensure you have set your OPENAI_API_KEY environment variable. If you have raw logs and need to extract predicted programs first, use python recover_pred_from_log.py.

export $OPENAI_API_KEY=YOUR_API_KEY
python -m evaluation.harness.run_evaluation \
--benchmark_path benchmark \
--pred_program_path pred_programs \
--log_fname self_debug_eval.jsonl \
--run_id test_run_1 \
--cache_level base \
--max_workers 4 \
--force_rebuild False \
--instance_ids 1 2 3

To prevent data contamination, only the annotation sheet (containing inputs) is available on Huggingface. To evaluate agent outcomes (generated code), you must download the full benchmark artifacts separately.

1. Download the full benchmark here.
2. Unzip the file using the password: scienceagentbench.

Warning: Do not redistribute the unzipped data files online.

Expected directory structure:

|-- ScienceAgentBench/
|---- benchmark/
|------ datasets/
|------ eval_programs/
|------ gold_programs/
|------ scoring_rubrics/
|---- engine/
|---- agent.py

You need two separate conda environments: one for running the agents (sci-agent) and one for evaluation (sci-agent-eval) to ensure a clean environment for task-specific package installations.

1. Create the Agent Environment

conda create -n sci-agent python=3.10 pip setuptools wheel
conda activate sci-agent
pip install -r requirements.txt
export PYTHONPATH=.

2. Create the Evaluation Environment

This environment uses pip-tools to automatically update package installations for different tasks. After setup, deactivate this environment and perform all work within sci-agent.

conda create -n sci-agent-eval python=3.10 pip setuptools wheel
conda activate sci-agent-eval
pip install pip-tools
conda deactivate

If Docker is unavailable, you can run evaluation directly. This method runs one instance at a time.

Note: A valid OPENAI_API_KEY is required because the evaluation uses GPT-4o to judge output visualizations.

export OPENAI_API_KEY={YOUR_OPENAI_KEY}
python -u run_eval.py \
    --log_fname {EVAL_LOG_FNAME}

Arguments:

• --log_fname: Custom JSONL filename for evaluation results.

The max_workers argument determines the number of parallel Docker containers running task instances.

Recommendations:

• Do not exceed the number of available CPU cores on your machine.
• For a 16-core machine with 32GB RAM, max_workers=8 completes the evaluation in ~22 minutes.

Latency Reference Table:

The cache_level argument controls how much disk space is used for Docker images during and after evaluation:

• base (Default): Stores only the base image (containing common dependencies like numpy and pandas). Requires ~5 GB for the base image and approximately 5 + max_workers * 25 GB of free space during runtime.
• instance: Caches all instance images (task-specific packages). Warning: This can consume up to 2,600 GB of disk space. Use this only if you need to reuse specific task environments.
• none: Removes all created images after evaluation completes. Requires 5 + max_workers * 25 GB of free space during runtime.

Use run_infer.py to execute agents. By default, it uses direct prompting. You can enable self-debugging or expert knowledge via flags.

python -u run_infer.py \
    --llm_engine_name {MODEL_NAME} \
    --log_fname {LOG_FNAME} \
    [--use_knowledge] \
    [--use_self_debug]

Arguments:

• --llm_engine_name: Name of the base LLM on OpenAI or Amazon Bedrock.
• --log_fname: Custom JSONL filename to store agent trajectories and costs (e.g., claude_self_debug.jsonl).
• --use_knowledge: (Optional) Whether to use expert-provided knowledge.
• --use_self_debug: (Optional) Whether to use self-debug mode.

If you are evaluating OpenHands agents or need to re-run evaluation for other agents, use recover_pred_from_log.py to automatically extract the generated Python code files from the JSONL trajectory logs.

python -u recover_pred_from_log.py \
    --log_fname {LOG_FNAME} \
    [--is_opendevin]

Arguments:

• --log_fname: The JSONL log file containing agent trajectories.
• --is_opendevin: (Optional) Set this if the log file is specifically from an OpenHands (formerly OpenDevin) agent.

To report final metric numbers, use calculate_metrics.py. The program automatically selects the best trajectory out of three runs (as described in the paper) to calculate metrics.

python calculate_metrics.py \
    --run_logs {LOG_FNAME_1} \
    --run_logs {LOG_FNAME_2} \
    --run_logs {LOG_FNAME_3} \
    --eval_logs {EVAL_LOG_FNAME_1} \
    --eval_logs {EVAL_LOG_FNAME_2} \
    --eval_logs {EVAL_LOG_FNAME_3}

Arguments:

• --run_logs: JSONL files containing agent trajectories and costs.
• --eval_logs: JSONL files containing evaluation results.

Note: You can adjust the number of runs by adding or removing --run_logs or --eval_logs arguments.

The evaluation.harness.run_evaluation module accepts the following arguments:

Mandatory Arguments:

• --benchmark_path: Path to the benchmark folder.
• --pred_program_path: Path to the predicted program folder.
• --log_fname: Path to the customized JSONL log file for results (e.g., my_eval.jsonl).
• --run_id: An arbitrary string identifier for the run. If omitted, all unevaluated instances are processed.

Optional Arguments:

• --cache_level: Controls Docker image caching. Options: none, base, instance. Default is base.
• --max_workers: Number of CPU workers for parallel execution. Default is 4.
• --force_rebuild: Boolean (True/False) to force rebuilding all images. Default is False.
• --instance_ids: Specific instances to run (e.g., 1 2 3). If not set, all instances are run.