DOCSEARCHbyAlgolia
Submit Docs

pubmed_parser

repository·master·Indexed 20 days ago

https://github.com/titipata/pubmed_parser

A Python library for parsing PubMed Open-Access Subset and MEDLINE XML repositories into structured Python dictionaries. It provides specialized functions for extracting article metadata, authors, affiliations, references, figure captions, paragraphs, and table data. The library supports processing large datasets using PySpark 2.1 and includes utilities for parsing data retrieved from the PubMed website via E-utils.

Tokens
7.6K
Snippets
24
Records
41
Agent score
72%

What's inside pubmed_parser

  1. Overview of Pubmed Parser capabilities

    master

    Pubmed Parser is a Python-based utility designed to efficiently mine PubMed Open-Access XML subsets and MEDLINE XML datasets. Unlike some other parsers, it provides deep structural extraction including:

    • Article and journal metadata
    • Authors and affiliations
    • References
    • Figure captions
    • Paragraphs

    The output is provided as Python dictionaries, making it easy to integrate into machine learning pipelines (e.g., scikit-learn, tensorflow, pytorch) or to export as JSON files. The implementation is designed to be scalable within MapReduce-like infrastructures such as PySpark.

  2. Overview of Pubmed Parser

    master
    Pubmed Parser is a Python library designed to parse the PubMed Open-Access (OA) subset and MEDLINE XML repositories. It leverages the lxml library to convert complex XML data into structured Python dictionaries. This makes the data readily available for research tasks such as text mining and natural language processing (NLP) pipelines.

  3. Understanding the supported datasets

    master

    Pubmed Parser supports two primary data sources:

    1. PubMed Open-Access (OA) subset: Contains XML files of full submitted papers. This provides more structured information than a standard PDF article.
    2. MEDLINE XML: Contains approximately 30 million biomedical articles. The parser can access information up to the abstracts from compressed XML files.

    Note: For information not included in these XML files (such as citation counts), you should use the Entrez Programming Utilities (E-utils) to query data in XML format.

  4. Set up PySpark for processing MEDLINE XML data

    master

    You can use PySpark to process MEDLINE XML data into Spark DataFrames. This approach significantly reduces parsing time for large datasets (e.g., 25 million documents) when using multi-core processors.

    To use PySpark in environments like Jupyter Notebook, you may need to initialize findspark with your specific spark_home path.

    import os
import findspark
findspark.init(spark_home="/opt/spark-2.1.0-bin-cdh5.9.0/")

  6. Install pubmed_parser via pip

    master

    You can install pubmed_parser directly from the GitHub repository using pip. Alternatively, you can clone the repository locally and install it from the cloned directory.

    # Install directly from the repository
pip install git+git://github.com/titipata/pubmed_parser.git

# Or clone and install locally
git clone https://github.com/titipata/pubmed_parser
pip install ./pubmed_parser

  7. Configure a SparkSession for PubMed parsing

    master

    When setting up a SparkSession for heavy parsing tasks, you can use SparkConf to tune memory and executor settings. This is particularly useful when running on YARN or local clusters to ensure sufficient resources for processing large XML files.

    Note that the spark_home path and Python binary paths (PYSPARK_PYTHON, PYSPARK_DRIVER_PYTHON) must match your local environment.

    from pyspark.sql import SparkSession
from pyspark.conf import SparkConf

conf = SparkConf().\
    setAppName('map').\
    setMaster('local[5]').\
    set('spark.yarn.appMasterEnv.PYSPARK_PYTHON', '~/anaconda3/bin/python').\
    set('spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON', '~/anaconda3/bin/python').\
    set('executor.memory', '8g').\
    set('spark.yarn.executor.memoryOverhead', '16g').\
    set('spark.sql.codegen', 'true').\
    set('spark.yarn.executor.memory', '16g').\
    set('yarn.scheduler.minimum-allocation-mb', '500m').\
    set('spark.dynamicAllocation.maxExecutors', '3').\
    set('spark.driver.maxResultSize', '0')

spark = SparkSession.builder.\
    appName("testing").\
    config(conf=conf).\
    getOrCreate()

  9. Run and preprocess the PubMed Open-Access dataset to Spark Dataframe

    master

    To download and process the PubMed Open-Access (OA) subset, you must first modify the directory path within the pubmed_oa_spark.py script. Once configured, execute the script using spark-submit with PySpark version 2.1.

    ~/spark-2.1.0/bin/spark-submit pubmed_oa_spark.py

  11. Download PubMed OA and MEDLINE datasets

    master

    You can download the raw XML datasets for PubMed Open Access (OA) and MEDLINE from the following FTP and web locations:

    • PubMed Open-Access (OA): Available at http://www.ncbi.nlm.nih.gov/pmc/tools/ftp/. For bulk downloads, use the FTP link: ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/. The full set of available tar files is located in the oa_bulk folder: ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_bulk/.
    • MEDLINE XML (Baseline): ftp://ftp.nlm.nih.gov/nlmdata/.medleasebaseline/gz/.
    • MEDLINE XML (Weekly Updates): ftp://ftp.nlm.nih.gov/nlmdata/.medlease/gz/.
    • MEDLINE DTDs: To identify available tags within a MEDLINE XML, refer to the Document Type Definitions (DTDs) at https://www.nlm.nih.gov/databases/dtd/.

  12. Schedule PubMed Open-Access preprocessing with a Cronjob

    master

    You can automate the PubMed Open-Access preprocessing by adding a task to your crontab.

    1. Open the crontab editor: crontab -e (if needed, set your editor first via export EDITOR="your_editor").
    2. Add the following entry to run the script every Sunday at 08:00:
    #!/bin/bash

0 8 * * Sun source ~/.bash_profile;~/spark-2.1.0/bin/spark-submit pubmed_oa_spark.py