Introduction to PaperBench
mainPaperBench is an evaluation framework designed to test an AI agent's ability to replicate research from 20 Spotlight and Oral papers from ICML 2024 from scratch. Each evaluation sample consists of a research paper and a rubric defining the requirements for successful replication.
The evaluation process follows a three-stage lifecycle:
- Agent Rollout: The agent runs in an Ubuntu container to create a codebase that replicates the paper.
- Reproduction: The submitted codebase is executed in a fresh, second container (with GPU access) to generate results (the executed submission).
- Grading: A third container runs a judge to grade the executed submission against the paper's rubric.