DOCSEARCHbyAlgolia
Submit Docs

Paper2Agent Documentation

repository·main·Indexed 25 days ago

https://github.com/jmiao24/paper2agent

A multi-agent AI system that transforms research paper codebases into interactive AI agents. It extracts tools from tutorials and exposes them via Model Context Protocol (MCP) servers, compatible with Claude Code. The system utilizes a series of orchestrator agents—including Environment Setup & Tutorial Discovery, Tutorial Execution, and Tool Extraction & Testing—to automate the conversion of scientific code into validated, production-ready function libraries.

Tokens
7.4K
Snippets
14
Records
41
Agent score
81%

What's inside Paper2Agent

  1. Understand the combined quality scoring model

    main

    The system calculates a weighted Overall Quality Score (out of 100) based on the following components:

    • Coverage metrics (40% weight): Based on overall coverage percentages from JSON.
    • Code style score (30% weight): Based on average pylint score from parsed scores.
    • Test completeness score (20% weight): Based on the test-to-code ratio and function coverage.
    • Code structure score (10% weight): Based on issue density and quality distribution.

    The final output is reports/coverage_and_quality_report.md, which provides a per-tutorial breakdown and actionable recommendations for improving coverage, style, and structure.

  2. Understand the Code Quality & Coverage Analysis Coordinator role

    main

    The Code Quality & Coverage Analysis Coordinator is a quality assurance role designed to generate comprehensive code coverage reports and quantitative code quality metrics. It analyzes pre-generated reports from pytest-cov and pylint to provide actionable insights into test completeness, code style, and overall code quality for extracted tools.

    Its core mission is to:

    1. Extract quantitative metrics from coverage and pylint reports.
    2. Identify gaps in testing and style issues.
    3. Compile comprehensive quality assessment reports.

  3. MCP Server Template Structure

    main

    The MCP server must follow a specific structure to ensure compatibility and documentation completeness. The server file should be named src/${github_repo_name}_mcp.py.

    Template Requirements:

    • Docstring: Must include a three-sentence description of the codebase and a detailed list of all tutorial files and their associated tools/descriptions.
    • Imports: from fastmcp import FastMCP must come first, followed by tool imports in alphabetical order.
    • Mounting: Each tool module must be explicitly mounted using mcp.mount().
    • Execution: Include the if __name__ == "__main__": block to call mcp.run().
    """
Model Context Protocol (MCP) for ${github_repo_name}

[Three-sentence description of codebase functionality]

This MCP Server contains tools extracted from the following tutorial files:
1. tutorial_file_1_name
    - tool1_name: tool1_description
    - tool2_name: tool2_description
2. tutorial_file_2_name
    - tool1_name: tool1_description
    ...
"""

from fastmcp import FastMCP

# Import statements (alphabetical order)
from tools.tutorial_file_1_name import tutorial_file_1_name_mcp
from tools.tutorial_file_2_name import tutorial_file_2_name_mcp

# Server definition and mounting
mcp = FastMCP(name="${github_repo_name}")
mcp.mount(tutorial_file_1_name_mcp)
mcp.mount(tutorial_file_2_name_mcp)

if __name__ == "__main__":
    mcp.run()

  4. Guiding Principles for Tool Extraction and Testing

    main

    When using the Paper2Agent coordination system to extract tools from tutorials, follow these core principles to ensure scientific rigor and tutorial fidelity:

    Scientific Rigor & Tutorial Fidelity

    • Exact Tutorial Preservation: Function calls must exactly match the tutorial. Do not add extra parameters or generalize patterns. Preserve exact tutorial data structures.
    • No Fabrication: Never invent inputs, defaults, or examples. If it is not in the tutorial, do not assume it exists.
    • Conservative Approach: Explicitly surface all assumptions, limitations, and uncertainties.
    • Minimal Error Handling: Implement only basic input file validation to avoid over-engineering.

    Parallel Execution Coordination

    • Phase Dependency: Testing cannot begin until all parallel extractions are complete.
    • Output Validation: Verify that each parallel phase produces the required inputs for the subsequent phase.
    • Cross-File Independence: Ensure that parallel tutorial files do not interfere with each other.

    File Structure Standards

    • Naming Convention: Use snake_case for file names (e.g., Data-Processing-Tutorial becomes data_processing_tutorial).
    • Path Management: Use absolute paths in all artifacts and references to ensure reliability.

  5. Understand the Environment Setup & Tutorial Discovery Coordinator role

    main

    The Environment Setup & Tutorial Discovery Coordinator is an orchestrator agent designed to transform scientific research codebases into reusable tools. It manages two specialized subagents working in parallel:

    1. environment-python-manager: Handles Python environment setup using uv, configures pytest, and manages dependencies.
    2. tutorial-scanner: Identifies, classifies, and assesses the quality of tutorials within the codebase for potential tool extraction.

    The coordinator is responsible for launching these agents, monitoring progress, handling errors, and validating that the resulting outputs meet specific quality and schema standards.

  6. How tutorial extraction and testing are organized by file

    main

    Tutorial extraction and testing follow a file-based organization rather than an individual tutorial-based one. This means:

    • Consolidated Implementation: If a single source file (like README.md or a notebook) contains multiple tutorial sections, all extracted tools are implemented in a single file: src/tools/${tutorial_file_name}.py.
    • Unified Testing: All tools from that same source file are tested together within the directory tests/code/${tutorial_file_name}/.

    Example: If README.md contains 6 tutorial sections, all tools go into src/tools/readme.py, and their tests are located in tests/code/readme/.

  7. Understand the Tool Extraction & Testing Coordinator role

    main

    The Tool Extraction & Testing Coordinator is an orchestrator agent designed to transform executed tutorial notebooks into production-ready, tested function libraries. It manages two specialized subagents:

    1. tutorial-tool-extractor-implementor: Extracts reusable functions from tutorials with scientific rigor.
    2. test-verifier-improver: Creates and iteratively improves test suites until a 100% pass rate is achieved.

    The process is split into two main phases: Parallel Tool Extraction (Phase 1) followed by Parallel Testing, Verification & Improvement (Phase 2).

  8. Understand the Tutorial Execution Coordinator role

    main

    The Tutorial Execution Coordinator is an orchestrator agent responsible for managing the tutorial-executor subagent. Its primary mission is to transform tutorial materials into validated, executable Jupyter notebooks with gold-standard outputs.

    It manages the entire lifecycle of tutorial execution, including:

    • Pre-execution validation: Checking input files, environments, and dependencies.
    • Execution coordination: Launching the tutorial-executor and monitoring progress.
    • Error recovery: Handling environment issues, data dependencies, and version conflicts.
    • Quality assurance: Validating output integrity, file organization, and image extraction.
    • Reporting: Generating a final execution summary and validating the reports/executed_notebooks.json file.

  9. Configure API keys for tutorial execution

    main

    If tutorials require external API access (e.g., OpenAI, Anthropic, Gemini, AlphaGenome, ESM), you can provide an api_key. The coordinator instructs the tutorial-executor to detect these requirements and inject the key at the beginning of the notebooks using common patterns.

    Example injection patterns:

    # API Configuration
api_key = "${api_key}"
openai.api_key = api_key  # For OpenAI
# client = anthropic.Anthropic(api_key=api_key)  # For Anthropic
# etc.

  10. Analyze coverage metrics and gaps

    main

    The analysis process extracts several key metrics from reports/coverage/coverage.json and reports/coverage/coverage_summary.txt:

    • Line Coverage: Percentage of lines executed by tests.
    • Branch Coverage: Percentage of branches (if/else, try/except) tested.
    • Function Coverage: Percentage of functions/methods called.
    • Statement Coverage: Percentage of statements executed.
    • Per-File Coverage: Individual file coverage percentages.
    • Missing Coverage: Identification of functions/lines with 0% coverage.

    These metrics are used to generate reports/coverage/coverage_report.md, which includes per-tutorial coverage analysis and quality recommendations.

  11. Execution workflow for the Coordinator

    main

    The Coordinator operates in three distinct phases:

    Phase 1: Parallel Agent Launch

    Both subagents are executed simultaneously using the Task tool:

    • environment-python-manager: Sets up ${github_repo_name}-env with Python $\ge$ 3.10 in the current working directory (not the repo/ subfolder).
    • tutorial-scanner: Scans repo/${github_repo_name}/ for tutorials, applying the tutorial_filter if provided.

    Phase 2: Progress Monitoring & Error Recovery

    • Timeouts: A 10-minute timeout is applied per agent.
    • Environment Recovery: If setup fails, the coordinator attempts alternative Python versions (3.10, 3.11, 3.12).
    • Tutorial Recovery: If scanning fails, it attempts a partial scan with error reporting.

    Phase 3: Output Validation

    • Environment: Verifies environment-manager_results.md exists, contains required sections, and confirms Python $\ge$ 3.10.
    • Tutorials: Validates JSON schema compliance, cross-references paths with the repository structure, and ensures no legacy/deprecated content is marked for inclusion.

  12. Install Paper2Agent

    main

    Follow these steps to set up the Paper2Agent environment.

    Prerequisites

    Installation Steps

    1. Clone the repository:
      git clone https://github.com/jmiao24/Paper2Agent.git
cd Paper2Agent
    2. Install Python dependencies:
      pip install fastmcp
    3. Install and configure Claude Code:
      npm install -g @anthropic-ai/claude-code
claude
    git clone https://github.com/jmiao24/Paper2Agent.git
cd Paper2Agent
pip install fastmcp
npm install -g @anthropic-ai/claude-code
claude