DOCSEARCHbyAlgolia
Submit Docs

SciPDF Parser

repository·master·Indexed 19 days ago

https://github.com/titipata/scipdf_parser

A Python-based tool for extracting structured data—including titles, abstracts, sections, references, and figures—from scientific PDF documents using the GROBID service. Features include converting PDFs to dictionaries via parse_pdf_to_dict(), extracting XML content with parse_pdf(), and extracting figures using pdffigures2 via parse_figures().

Tokens
758
Snippets
5
Records
5
Agent score
16%

What's inside scipdf_parser

  1. Set up the GROBID service

    master

    SciPDF Parser is based on GROBID. You must have a GROBID service running on your machine before parsing.

    While the project provides a script to run GROBID, it is highly recommended to run GROBID via Docker for better stability and to ensure you are using the latest version.

    To use the provided script, run:

    bash serve_grobid.sh

    This script runs GROBID at the default port 8070.

  2. Install SciPDF Parser

    master

    Install the package directly from the GitHub repository using pip:

    pip install git+https://github.com/titipata/scipdf_parser

    Additionally, you must download the en_core_web_sm model for spacy to ensure full functionality:

    python -m spacy download en_core_web_sm
    pip install git+https://github.com/titipata/scipdf_parser
python -m spacy download en_core_web_sm

  3. Extract figures with parse_figures()

    master

    To extract figures from a collection of PDFs using pdffigures2, use scipdf.parse_figures().

    Parameters:

    • folder: A path to a folder containing only PDF files.
    • output_folder: The directory where extracted figures will be saved.
    import scipdf

scipdf.parse_figures('example_data', output_folder='figures')

  4. Parse PDF to dictionary with parse_pdf_to_dict()

    master

    Use scipdf.parse_pdf_to_dict() to convert a local PDF file or a remote URL into a structured Python dictionary.

    Options:

    • as_list: If set to True, the text field of parsed sections will be returned as a list of paragraphs instead of a single string. Defaults to False (single string).

    Output Schema: Returns a dictionary containing:

    • title: String
    • abstract: String
    • sections: List of {'heading': str, 'text': str}
    • references: List of {'title': str, 'year': str, 'journal': str, 'author': str}
    • figures: List of {'figure_label': str, 'figure_type': str, 'figure_id': str, 'figure_caption': str, 'figure_data': str}
    • doi: String
    import scipdf

# Parse local file
article_dict = scipdf.parse_pdf_to_dict('example_data/futoma2017improved.pdf')

# Parse from URL with text as a list of paragraphs
article_dict = scipdf.parse_pdf_to_dict('https://www.biorxiv.org/content/biorxiv/early/2018/11/20/463760.full.pdf', as_list=True)

  5. Parse PDF to XML with parse_pdf()

    master

    Use scipdf.parse_pdf() to extract the full XML content from GROBID. Setting soup=True allows you to parse the full XML structure.

    import scipdf

xml = scipdf.parse_pdf('example_data/futoma2017improved.pdf', soup=True)