The paperscraper library provides a modular API for scraping scientific literature and analyzing citations. The main entry points are organized by source and functionality:

• paperscraper: The main package entry point.
• paperscraper.arxiv: Tools for ArXiv scraping and keyword-based searches.
• paperscraper.citations: Functions to retrieve (self-)citations and (self-)references for both specific papers and authors.
• paperscraper.get_dumps: Utilities for downloading metadata dumps from bioRxiv, medRxiv, and chemRxiv.
• paperscraper.pdf: Functionality to download publications as PDF files.
• paperscraper.pubmed: Tools for keyword searches on PubMed.
• paperscraper.scholar: Endpoints for interacting with Google Scholar.
• paperscraper.xrxiv: Shared utilities specifically for the {bio, med, chem}Rxiv ecosystem.

The paperscraper package provides several top-level utility modules for different stages of the research paper analysis pipeline. These modules are accessible directly under the paperscraper namespace.

Available modules include:

• async_utils: Asynchronous utility functions.
• impact: Tools for calculating or analyzing research impact.
• load_dumps: Functions for loading data dumps.
• plotting: Visualization and plotting utilities.
• postprocessing: Tools for cleaning or transforming scraped data.
• server_dumps: Utilities related to server-side data dumps.
• utils: General purpose helper functions.

paperscraper provides utilities to move beyond simple citation counts to more nuanced bibliometric analysis.

Scholar Metrics:

• Paper level: Uses Semantic Scholar to retrieve citation counts.
• Author level: Uses Semantic Scholar to retrieve publication counts, citation counts, and h-index values.
• Journal level: Uses a fuzzy-search wrapper around the impact-factor package to find journal metrics.

Self-Citation Analysis: The package distinguishes between two types of self-links to help identify metric gaming:

• Self-citation: A citation from a later paper that shares at least one author with the cited paper.
• Self-reference: A reference made by a paper to earlier work that shares at least one author.

These metrics are estimated at both the paper and author levels using Semantic Scholar metadata.

In paperscraper, search terms are represented as nested lists to implement Boolean logic.

• The outer list represents the AND operator.
• The inner lists represent synonyms using the OR operator.

Example: [["AI", "Machine Learning"], ["Quantum"], ["Chemistry"]] translates to (AI OR "Machine Learning") AND (Quantum) AND (Chemistry).

ai = ["Artificial intelligence", "Machine learning"]
qc = ["Quantum computing", "Quantum algorithm"]
chemistry = ["Chemistry", "Chemical"]
query = [ai, qc, chemistry]

The project maintainers provide support for the following areas:

• Package behavior and documented APIs.
• Metadata parsing.
• Retrieval fallbacks.
• Reproducible examples.

The following are NOT supported or guaranteed:

• Access to paywalled content.
• Bypassing publisher restrictions.
• Solving Google Scholar captchas.
• Controlling the availability of upstream APIs.

If direct PDF retrieval via DOI fails, paperscraper attempts several fallback mechanisms to find full text:

• BioC-PMC XML: For open-access papers in PubMed Central.
• eLife XML: From the eLife article XML repository.
• Publisher APIs: When matching credentials (e.g., Wiley, Elsevier) are provided.
• bioRxiv S3 access: When AWS requester-pays credentials (IAM key with AmazonS3ReadOnlyAccess) are provided.

Note: Fallbacks cannot bypass publisher paywalls or restrictions.

You can install paperscraper using pip or uv to perform reproducible searches over scholarly metadata, retrieve full-text PDFs/XML, and perform bibliometric analysis across PubMed, arXiv, bioRxiv, medRxiv, and ChemRxiv.

pip install paperscraper

or

uv add paperscraper

Never post API keys, publisher credentials, AWS credentials, or private data in public GitHub issues.

If a report requires private credentials or describes a sensitive security issue, contact the maintainers privately via their public GitHub or package metadata profiles. Provide only the minimum information necessary for triage.

To scrape data from bioRxiv, medRxiv, or chemRxiv, you must first download their local metadata dumps. These are stored in JSONL format (one paper per line) in the server_dumps folder. After downloading, restart your Python interpreter so paperscraper.load_dumps can discover the new files.

Use paperscraper.utils.get_server_dumps_dir() to inspect the directory where dumps are stored.

from paperscraper.get_dumps import biorxiv, medrxiv, chemrxiv

chemrxiv()  # ~50K papers
medrxiv()   # ~100K papers
biorxiv()   # ~450K papers

Questions regarding documented workflows are welcome on the public issue tracker. When asking questions, include enough context to reproduce the problem locally.

Note on external factors: Be aware that failures may be caused by factors outside of paperscraper's control, such as:

• External API availability or rate limits.
• Captchas.
• Publisher access rules and paywalls.
• Platform terms of service.