DOCSEARCHbyAlgolia
Submit Docs

paperetl

repository·master·Indexed 20 days ago

https://github.com/neuml/paperetl

An ETL (Extract, Transform, Load) library for processing medical and scientific papers. It supports ingesting formats such as PDFs, PubMed XML, ArXiv XML, TEI XML, and CSVs, and loading them into datastores including SQLite, Elasticsearch, JSON, or YAML. PDF parsing requires a running instance of GROBID.

Tokens
9.3K
Snippets
38
Records
49
Agent score
72%

What's inside paperetl

  1. Configure GROBID for PDF parsing

    master

    PDF parsing in paperetl requires a running instance of GROBID. It is assumed to be running locally on the ETL server.

    If you encounter 503 errors due to engine pool exhaustion, you may need to increase concurrency and/or poolMaxWait in your GROBID configuration file.

  4. Install paperetl using Docker

    master

    You can build and run a Docker container containing paperetl and all its dependencies using the provided Dockerfile in the repository.

    wget https://raw.githubusercontent.com/neuml/paperetl/master/docker/Dockerfile
docker build -t paperetl -f Dockerfile .
docker run --name paperetl --rm -it paperetl

  5. How the Execute engine manages parallel processing

    master

    The Execute class implements a producer-consumer pattern using Python's multiprocessing.Process and Queue to parallelize the parsing of scientific files.

    1. Scanning (Producer): Execute.scan() walks the input directory and populates an inputs queue with file metadata (path, extension, compression status).
    2. Parsing (Workers): Multiple worker processes run Execute.process(). Each worker pulls from the inputs queue, calls Execute.parse() to extract articles, and groups them into batches of size batchsize.
    3. Serialization: To avoid passing large objects through queues, workers use Execute.serialize() to write batches to temporary files using pickle. The file path is then sent to the outputs queue.
    4. Saving (Consumer): The main process runs Execute.save(), which reads the temporary file paths from the outputs queue, loads the articles, and saves them to the database via the db.save() method. It also handles the Execute.COMPLETE signal to shut down gracefully.
    5. Cleanup: Execute.close() ensures all processes and queues are properly closed.

  6. How CSV sections are constructed

    master

    The CSV.sections method determines how text content is structured within an Article.

    By default, the CSV processor creates a single section for each article. This section is constructed by concatenating the title and the abstract (if an abstract is present in the row) into a single block of text.

    Section Format:

    • The resulting sections list contains a single tuple: [(None, text)], where None represents the section name/header and text is the combined title and abstract content.

  8. Understand the PMB article processing logic

    master

    The PMB.process method converts a single PubmedArticle XML element into a structured Article object. It extracts several categories of data:

    • Metadata: Includes PMID (as string), source name, publication date, journal title, authors (semicolon-separated), affiliations, primary affiliation, article title, tags (including MeSH codes), and a PubMed URL.
    • Sections: A list of tuples (section_name, text) representing the article's content (e.g., ('ABSTRACT', '...'), ('TITLE', '...')).
    • Citations: A list of PubMed IDs found in the article's reference list, filtered by the provided ids list if applicable.

    Section Parsing Formats Supported: PMB handles three different abstract formats found in PubMed XML:

    1. Raw Text: Single text block parsed into sentences.
    2. HTML Formatted: Uses tags like <b> or specific section headers (e.g., 'aim', 'introduction') to split the text into named sections.
    3. Labeled/Parsed: Uses the Label attribute on AbstractText elements to define section names.

  9. Use the FileSystem base class for custom file storage

    master

    The FileSystem class provides the base logic for directory management and file naming. It automatically creates the outdir if it does not exist. To implement a custom format, you must subclass FileSystem and implement the extension() and write(self, output, article) methods.

    Key behaviors:

    • Directory Creation: Uses os.makedirs(outdir, exist_ok=True) during initialization.
    • Filename Generation: The filename is constructed as {article.uid()}.{extension()}. If article.source() is present, it is prefixed: {source_stem}-{uid}.{extension()}.
    from paperetl.filesystem import FileSystem

class MyFormat(FileSystem):
    def extension(self):
        return "txt"

    def write(self, output, article):
        output.write(article.build())

fs = MyFormat(outdir="./my_files")
fs.save(article)

  10. Install paperetl and dependencies

    master

    To use paperetl, install it directly from the GitHub repository. Additionally, you must download the NLTK punkt tokenizer data for text processing. If you intend to process PDF articles, you must also install GROBID.

    # Install paperetl
pip install git+https://github.com/neuml/paperetl

# Download NLTK data
python -c "import nltk; nltk.download('punkt')"

  11. Install the Elasticsearch extra for paperetl

    master

    The Elastic class requires the elasticsearch Python package. If it is not installed, you will encounter an ImportError. Ensure you have installed the elasticsearch dependency to enable Elasticsearch indexing capabilities.

    # Note: The specific install command is inferred from the error message
# pip install paperetl[elasticsearch]

  12. Load articles into SQLite

    master

    To load a set of medical/scientific articles from a local directory into a SQLite database, use the paperetl.file module.

    Example workflow:

    1. Place articles in a directory (e.g., paperetl/data).
    2. Run the command to build the database pointing to a target directory for the models.
    python -m paperetl.file paperetl/data paperetl/models