DOCSEARCHbyAlgolia
Submit Docs

Findpapers Documentation

repository·main·Indexed 18 days ago

https://github.com/jonatasgrosman/findpapers

A Python library for unified, multi-database access to academic papers. Findpapers allows researchers to perform single boolean queries across databases including arXiv, CrossRef, IEEE Xplore, OpenAlex, PubMed, Scopus, Semantic Scholar, and Web of Science. It features automated deduplication, metadata enrichment, PDF downloading, and citation snowballing to build research networks.

Tokens
20.7K
Snippets
61
Records
79
Agent score
61%

What's inside Findpapers

  1. Control data sources using the databases parameter

    main

    The databases parameter allows you to specify which backends engine.get() should consult. By default (None), it uses all available sources and merges the results.

    Use specific lists to optimize for speed or specific metadata types:

    • CrossRef only: Fast, authoritative structured metadata.
    • Web scraping + OpenAlex: Scrapes the landing page and enriches it via OpenAlex.
    • Web scraping only: Fetches from a URL without any API-based enrichment.
    # CrossRef only
paper = engine.get("10.1038/nature12373", databases=["crossref"])

# Web scraping + OpenAlex
paper = engine.get("10.1038/nature12373", databases=["web_scraping", "crossref", "openalex"])

# Web scraping only
paper = engine.get("https://arxiv.org/abs/1706.03762", databases=["web_scraping"])

  2. How Findpapers handles rate limiting

    main
    Findpapers automatically respects the rate limits of each connected database. If a rate limit is reached, the engine waits for the required cooldown period before retrying the request. Specifically, responses with HTTP status codes 429 (Too Many Requests) and 5xx (Server Errors) are retried using an exponential backoff strategy.

  3. Understand the Paper object and its data structure

    main

    Every paper retrieved via search(), get(), or snowball() is a Paper instance. You can convert a paper to a standard Python dictionary using paper.to_dict(), which is useful for JSON serialization.

    A fully-populated Paper object contains fields such as:

    • title, abstract, authors (with affiliations)
    • source (title, publisher, source_type, etc.)
    • publication_date, url, pdf_url, doi
    • citations (count), keywords, subjects, fields_of_study
    • is_open_access, is_retracted
    • references (list of DOIs/URLs)
    • cited_by (list of DOIs/URLs)
    • found_in (list of databases where it was found)
    {
  "title": "Attention Is All You Need",
  "abstract": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks...",
  "authors": [
    {"name": "Vaswani, A.", "affiliation": "Google Brain"},
    {"name": "Shazeer, N.", "affiliation": "Google Brain"}
  ],
  "source": {
    "title": "31st Conference on Neural Information Processing Systems",
    "publisher": "Curran Associates",
    "source_type": "conference"
  },
  "publication_date": "2017-12-06",
  "url": "https://arxiv.org/abs/1706.03762",
  "pdf_url": "https://arxiv.org/pdf/1706.03762",
  "doi": "10.48550/arXiv.1706.03762",
  "citations": 140000,
  "keywords": ["attention mechanism", "neural machine translation"],
  "found_in": ["arxiv", "semantic_scholar"],
  "is_open_access": true,
  "references": ["10.1162/neco.1997.9.8.1735"],
  "cited_by": ["10.18653/v1/2020.acl-main.703"]
}

  4. Supported identifier formats for engine.get()

    main

    The get() method automatically routes identifiers to the appropriate backend:

    1. Bare DOI: Queries configured databases via APIs and merges results (e.g., 10.1038/nature12373).
    2. DOI URL: Automatically strips doi.org or dx.doi.org prefixes and resolves via the multi-database path (e.g., https://doi.org/10.1038/nature12373).
    3. Landing-page URL:
      • Supported APIs: For arXiv, PubMed, IEEE Xplore, OpenAlex, and Semantic Scholar, the paper is fetched directly via their respective APIs.
      • HTML Scraping Fallback: For all other publisher URLs, the page is downloaded and metadata is extracted from HTML <meta> tags.

  5. Compare persistence formats (JSON vs BibTeX vs CSV)

    main

    Choose a format based on your requirements:

    • Use JSON if you need a lossless round-trip, want to preserve SearchResult or SnowballResult objects, or need to keep author affiliations and all metadata.
    • Use BibTeX if you need LaTeX compatibility for citations.
    • Use CSV if you need spreadsheet compatibility or want to view data in tools like Excel.

  6. How the snowball method works

    main

    The engine.snowball() method performs breadth-first citation traversal to discover related papers starting from one or more seed papers. It iteratively fetches references (backward links) and/or citing papers (forward links) to map the citation network around the seeds.

    The Two-Step Fetch Strategy:

    1. Seed enrichment: Seed papers are fetched using the union of databases and enrichment_databases to ensure they have full metadata (including references and cited_by) before traversal begins.
    2. BFS discovery: For each level, candidate DOIs found in the citation lists are fetched using the configured databases. After all levels complete, surviving non-seed papers are re-enriched using enrichment_databases that were not already used during discovery to fill metadata gaps (like abstracts or PDFs) without redundant API calls.

    Note: Papers without a DOI are silently skipped as they cannot be resolved by upstream APIs.

    import findpapers

engine = findpapers.Engine()
seed = engine.get("10.1038/nature12373")
result = engine.snowball(seed, max_depth=1, direction="both")

  7. Use CrossRef for DOI enrichment and backward snowballing

    main

    CrossRef is not used as a search database in Findpapers, but it is critical for metadata enrichment and backward snowballing.

    Key Details:

    • Authentication: Not required. However, providing your email enables the "polite pool," increasing response speed from ~10 to ~50 requests/s.
    • Functionality:
      • Enrichment: Adds abstracts, keywords, and citation counts to papers found in other databases.
      • Backward Snowballing: Follows the reference list of a paper by resolving cited DOIs.
    • Limitations: Does not support forward snowballing (cited-by). Reference lists are only available for references that carry a DOI.

  8. Understand the output files from engine.download()

    main

    When downloading, the following files are generated in the output_directory:

    1. PDF Files: Saved using a YEAR-title.pdf naming scheme.
    2. download_log.txt: A log file containing the status of each download (success or failure with the reason).
    pdfs/
├── 2023-attention-is-all-you-need.pdf
├── 2023-bert-pre-training-of-deep-bidirectional.pdf
├── 2024-vision-transformer-for-medical-imaging.pdf
└── download_log.txt

  9. Save and reload snowball results using JSON

    main

    You can persist the results of a snowball search to a file and reload them later using the save_to_json and load_from_json functions. This is useful for long-running searches or for sharing results with other tools.

    import findpapers

# Save as JSON
findpapers.save_to_json(result, "snowball_result.json")

# Reload later
result = findpapers.load_from_json("snowball_result.json")

  10. Construct basic queries using square brackets

    main

    The canonical way to define a search term in Findpapers is to enclose it in square brackets: [term]. Terms cannot be empty and cannot contain double quotes. Boolean connectors (case-insensitive) must have at least one whitespace before and after them to function correctly.

    Supported Boolean Connectors:

    • AND: Both terms must be present.
    • OR: At least one term must be present.
    • AND NOT: First term present, second excluded.
    [machine learning] AND [healthcare]
[deep learning] OR [neural network]
[reinforcement learning] AND NOT [game]

  11. Typical workflow: Search, Download, and Save

    main

    A standard workflow involves searching for papers, downloading their PDFs in parallel, and then saving the search results to a JSON file for later use.

    import findpapers

engine = findpapers.Engine()

# 1. Search
result = engine.search("[machine learning] AND [healthcare]")

# 2. Download
metrics = engine.download(result.papers, "./pdfs", num_workers=8)

print(f"Downloaded {metrics['downloaded_papers']} of {metrics['total_papers']} papers")

# 3. Save results
findpapers.save_to_json(result, "results.json")

  12. Use OpenAlex for large-scale bibliometric analysis and snowballing

    main

    OpenAlex is a massive, open index of scholarly works. It is the best source for both forward and backward snowballing.

    Setup:

    • Authentication: Highly recommended. A free API key increases your limit from ~10 requests/day to ~10,000 requests/day. Register at openalex.org/settings/api.

    Key Details:

    • Snowballing: Supports both forward snowballing (papers that cite a seed) and backward snowballing (references cited by a seed).
    • Limitations:
      • Wildcards (* or ?) are not supported and will cause errors.
      • key[] (keywords) and src[] (source) filter codes are not supported.