DOCSEARCHbyAlgolia
Submit Docs

DeepResearch Bench

repository·main·Indexed 21 days ago

https://github.com/ayanami0730/deep_research_bench

A benchmark for evaluating Deep Research Agents (DRAs) using 100 PhD-level research tasks across 22 fields. It features two evaluation frameworks: RACE (Reference-based Adaptive Criteria-driven Evaluation) for report quality across comprehensiveness, insight, instruction-following, and readability; and FACT (Framework for Factual Abundance and Citation Trustworthiness) for verifying information retrieval and grounding via statement-URL extraction and support verification.

Tokens
1.7K
Snippets
3
Records
9
Agent score
24%

What's inside DeepResearch Bench

  1. Overview of DeepResearch Bench

    main
    DeepResearch Bench is a comprehensive benchmark designed to systematically evaluate Deep Research Agents (DRAs). It consists of 100 PhD-level research tasks across 22 distinct fields, including Science & Technology, Finance & Business, Software, and more. The benchmark is designed to reflect real-world research demands by mirroring the topical distribution found in large-scale user query datasets.

  2. How the FACT evaluation methodology works

    main

    FACT (Framework for Factual Abundance and Citation Trustworthiness) evaluates an agent's information retrieval and grounding capabilities through the following pipeline:

    1. Statement-URL Extraction: Automatically extracts factual claims and their cited sources from the generated reports.
    2. Deduplication: Removes redundant statement-URL pairs to focus on unique factual claims.
    3. Support Verification: Uses web scraping and LLM judgment to verify if the cited sources actually support the claims.
    4. Citation Metrics Calculation: Computes:
      • Citation Accuracy: Percentage of correctly supported citations.
      • Effective Citations: Average number of verifiably supported citations per task.

  3. How the RACE evaluation methodology works

    main

    RACE (Reference-based Adaptive Criteria-driven Evaluation) evaluates the quality of generated research reports using a multi-step process:

    1. Dynamic Criteria Generation: Automatically generates task-specific evaluation criteria across four dimensions:
      • Comprehensiveness: Coverage breadth and depth of the research topic.
      • Insight/Depth: Quality of analysis and insight generation.
      • Instruction-Following: Adherence to specific task requirements.
      • Readability: Clarity, organization, and presentation quality.
    2. Reference-Based Scoring: Compares the agent's target reports against high-quality reference reports.
    3. Weighted Assessment: Applies dynamic weights adapted to the specific requirements of each task.

  4. Install DeepResearch Bench

    main

    To install the benchmark, clone the repository and install the required Python dependencies.

    Prerequisites:

    • Python 3.9+
    • OpenRouter or OpenAI API key (for LLM evaluation)
    • Jina API key (for web scraping in FACT evaluation)

    Installation steps:

    git clone https://github.com/your-username/deep_research_bench.git
cd deep_research_bench
pip install -r requirements.txt

  5. Submit to the DeepResearch Bench Leaderboard

    main

    To obtain an official leaderboard entry, email your materials to dumingxuan@mail.ustc.edu.cn and imlrz@mail.ustc.edu.cn.

    Required Materials:

    1. A temporary key with access to GPT-5.5 (OpenAI or OpenRouter) for verification.
    2. Raw generated articles: Model outputs in the benchmark's raw data format (e.g., .jsonl files).
    3. Reproducibility link: A repository link (for open-source) or a product/API link (for closed-source).
    4. Model metadata: Model name, project link, and license (or indication that it is proprietary).

    Recommended Files:

    • results/race/<model_name>/race_result.txt
    • results/fact/<model_name>/fact_result.txt

  6. Run the benchmark evaluation

    main

    Follow these steps to evaluate your Deep Research Agent using the benchmark:

    1. Prepare Model Data: Run your agent using the queries found in data/prompt_data/query.jsonl. Save the outputs to data/test_data/raw_data/<model_name>.jsonl in the following JSONL format:
      {"id": "task_id", "prompt": "original_query_text", "article": "generated_research_article_with_citations"}
    2. Configure Target Models: Edit run_benchmark.sh and add your model name to the TARGET_MODELS array:
      TARGET_MODELS=("your-model-name")
    3. Execute: Run the benchmark script:
      bash run_benchmark.sh

    Output Locations:

    • RACE evaluation: results/race/<model_name>/race_result.txt
    • FACT evaluation: results/fact/<model_name>/fact_result.txt
    # 1. Save outputs to data/test_data/raw_data/<model_name>.jsonl
# 2. Edit run_benchmark.sh: TARGET_MODELS=("your-model-name")
# 3. Run:
bash run_benchmark.sh

  7. Submit your agent to the DeepResearch Bench Leaderboard

    main

    To evaluate your deep research agent and submit results to the leaderboard, follow these requirements:

    • Contact: Reach out to dumingxuan@mail.ustc.edu.cn and imlrz@mail.ustc.edu.cn.
    • Evaluator Version: Note that the official evaluator has transitioned from Gemini-2.5-Pro to GPT-5.5 (with GPT-5.4-mini used for the FACT pipeline).
    • Migration Window: During the dual-acceptance window (ending 31 May 2026), submissions evaluated under both the legacy (Gemini-2.5-Pro) and new (GPT-5.5) evaluators are accepted and displayed on separate leaderboards. After 1 June 2026, only the GPT-5.5 leaderboard is maintained.
    • Configuration: New submitters must follow the specific keys and configuration defined in the API Configuration section (refer to the second segment of the documentation for details).

  8. Configure API environment variables

    main

    The benchmark requires specific environment variables to function. You must select an LLM_BACKEND (either openrouter or openai). OpenRouter is the default.

    Required Variables:

    • LLM_BACKEND: Set to openrouter or openai.
    • OPENROUTER_API_KEY: Required if using OpenRouter.
    • OPENAI_API_KEY: Required if using OpenAI direct.
    • JINA_API_KEY: Required for the FACT evaluation pipeline (web scraping).

    Default Judge Models:

    BackendRACE judge (Model)FACT judge (FACT_Model)
    openrouteropenai/gpt-5.5openai/gpt-5.4-mini
    openaigpt-5.5gpt-5.4-mini

    You can override these defaults using RACE_MODEL or FACT_MODEL environment variables.

    # Pick one backend. OpenRouter is the default.
export LLM_BACKEND="openrouter"               # or "openai"

# OpenRouter (default):
export OPENROUTER_API_KEY="sk-or-v1-xxxxx"

# Or OpenAI direct:
# export LLM_BACKEND="openai"
# export OPENAI_API_KEY="sk-xxxxx"

# Set Jina API key for web scraping (FACT pipeline only)
export JINA_API_KEY="your_jina_api_key_here"