DOCSEARCHbyAlgolia
Submit Docs

Paper2Code (PaperCoder)

repository·master·Indexed 26 days ago

https://github.com/going-doer/paper2code

A multi-agent LLM system that transforms scientific machine learning papers into functional code repositories via a planning, analysis, and generation pipeline. Includes the Paper2Code Benchmark dataset featuring 90 papers from ICML, NeurIPS, and ICLR 2024, and supports integration with OpenAI API and open-source models via vLLM.

Tokens
1.8K
Snippets
7
Records
7
Agent score
39%

What's inside Paper2Code

  1. Run PaperCoder using Open Source models with vLLM

    master

    To use open-source models, ensure vllm is installed. The default model is deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct. You can run the pipeline using either the PDF-based JSON format or the LaTeX source.

    # Using the PDF-based JSON format
cd scripts
bash run_llm.sh

# Using the LaTeX source
cd scripts
bash run_latex_llm.sh

  2. Access the Paper2Code Benchmark dataset

    master

    The Paper2Code Benchmark is a dataset of 90 papers from ICML 2024, NeurIPS 2024, and ICLR 2024, designed to evaluate the ability to reproduce methods and experiments from scientific papers.

    To use the dataset:

    1. Unzip the paper2code_data.zip file.
    2. For the full dataset including JSON files paired with original PDFs, download paper2code_full_data.zip from the provided Google Drive link.
    3. Alternatively, access the dataset on Hugging Face via the Paper2Code dataset page.
    unzip paper2code_data.zip

  3. Convert PDF to JSON using s2orc-doc2json

    master

    If you do not have the LaTeX source, you can convert a paper PDF into a structured JSON format using the s2orc-doc2json tool. This requires cloning the tool and running a Grobid processing service.

    1. Clone the repository: git clone https://github.com/allenai/s2orc-doc2json.git
    2. Run the Grobid service: ./gradlew run inside the grobid-0.7.3 directory.
    3. Process the PDF using process_pdf.py.
    # 1. Clone
git clone https://github.com/allenai/s2orc-doc2json.git

# 2. Run Grobid service
cd ./s2orc-doc2json/grobid-0.7.3
./gradlew run

# 3. Convert PDF to JSON
# Replace ${PDF_PATH} with your actual file path
python ./s2orc-doc2json/doc2json/grobid2json/process_pdf.py \
    -i ${PDF_PATH} \
    -t ./s2orc-doc2json/temp_dir/ \
    -o ./s2orc-doc2json/output_dir/paper_coder

  4. Install Paper2Code dependencies

    master

    You can install the necessary dependencies for Paper2Code using pip. It is recommended to use a Python virtual environment. You can install all dependencies at once or selectively based on your model provider.

    • For OpenAI API: Install openai.
    • For Open Source Models: Install vllm.
    • Full Installation: Install everything via requirements.txt.
    # Install all dependencies
pip install -r requirements.txt

# Or install selectively
pip install openai
pip install vllm

  5. Run PaperCoder using OpenAI API

    master

    To use PaperCoder with OpenAI models (e.g., o3-mini), set your OPENAI_API_KEY environment variable. You can run the pipeline using either a PDF-based JSON format or the LaTeX source of the paper.

    Note: The default example runs the 'Attention Is All You Need' paper.

    # Using the PDF-based JSON format
export OPENAI_API_KEY="<OPENAI_API_KEY>"
cd scripts
bash run.sh

# Using the LaTeX source
export OPENAI_API_KEY="<OPENAI_API_KEY>"
cd scripts
bash run_latex.sh

  6. Evaluate generated repositories with eval.py

    master

    PaperCoder supports model-based evaluation of generated repositories using eval.py. You can perform either Reference-free evaluation (critiquing the repo based on the paper) or Reference-based evaluation (comparing the repo against an official/gold repository).

    Setup: Install tiktoken and export your OPENAI_API_KEY.

    Arguments:

    • --paper_name: Name of the paper.
    • --pdf_json_path: Path to the paper's JSON file.
    • --data_dir: Directory containing data.
    • --output_dir: Directory for output artifacts.
    • --target_repo_dir: The directory of the generated repository to evaluate.
    • --eval_result_dir: Directory to save results.
    • --eval_type: Either ref_free or ref_based.
    • --gold_repo_dir: (Required for ref_based) Path to the official/gold repository.
    • --papercoder: Flag to indicate PaperCoder evaluation.
    # Reference-free Evaluation
cd codes/
python eval.py \
    --paper_name Transformer \
    --pdf_json_path ../examples/Transformer_cleaned.json \
    --data_dir ../data \
    --output_dir ../outputs/Transformer \
    --target_repo_dir ../outputs/Transformer_repo \
    --eval_result_dir ../results \
    --eval_type ref_free \
    --generated_n 8 \
    --papercoder

# Reference-based Evaluation
cd codes/
python eval.py \
    --paper_name Transformer \
    --pdf_json_path ../examples/Transformer_cleaned.json \
    --data_dir ../data \
    --output_dir ../outputs/Transformer \
    --target_repo_dir ../outputs/Transformer_repo \
    --gold_repo_dir ../examples/Transformer_gold_repo \
    --eval_result_dir ../results \
    --eval_type ref_based \
    --generated_n 8 \
    --papercoder

  7. Understand the Paper2Code data structure

    master

    The dataset is organized by conference folders (iclr2024, icml2024, nips2024). Each folder contains:

    • [PAPER].json: The parsed version of the paper.
    • [PAPER]_cleaned.json: The preprocessed version specifically formatted for use with PaperCoder.
    • pdfs/[PAPER].pdf: The original paper PDF (Note: This directory is only included in the paper2code_full_data.zip distribution).
    ├── iclr2024 
├── icml2024
└── nips2024
    ├── adaptive-randomized-smoothing.json
    ├── adaptive-randomized-smoothing_cleaned.json
    ├── ... 
    ├── YOLA.json
    ├── YOLA_cleaned.json
    └── pdfs  # only available in `paper2code_full_data.zip`
        ├── adaptive-randomized-smoothing.pdf
        ├── ... 
        └── YOLA.pdf