SciDownl

repository·main·Indexed 43 days ago

https://github.com/tishacy/scidownl

An unofficial API and CLI tool for downloading scientific papers from SciHub using DOI, PMID, or paper titles. It features automated SciHub domain management, proxy support, and a local SQLite database to track active mirror domains. Provides both a command-line interface and a Python module for programmatic access via the `scihub_download` function and `SciHub` class.

Tokens
2.8K
Snippets
13
Records
13
Agent score
54%

What's inside SciDownl

  1. Install SciDownl via pip or source

    main

    You can install SciDownl using pip for convenience or by cloning the repository and installing from source.

    Using pip:

    pip3 install -U scidownl

    From source:

    git clone https://github.com/Tishacy/SciDownl.git
    cd Scidownl && python3 setup.py install
  2. Download a list of DOIs in Python

    main

    To download multiple papers, iterate through a list of DOIs and instantiate the SciHub class for each.

    from scidownl.scihub import *
    
    DOIs = ["10.1021/ol9910114", "10.1021/example2"]
    out = 'paper'
    for doi in DOIs:
      SciHub(doi, out).download(choose_scihub_url_index=3)
  3. Use `scihub_download` in Python

    main

    For programmatic access, import the scihub_download function.

    Signature: scihub_download(paper, paper_type, out=None, proxies=None)

    Parameters:

    • paper (str): The identifier (DOI, PMID, or Title).
    • paper_type (str): The type of identifier. Use 'doi', 'pmid', or 'title'.
    • out (str, optional): Output directory or file path.
    • proxies (dict, optional): A dictionary of proxy mappings (e.g., {'http': 'socks5://...'}).
    from scidownl import scihub_download
    
    paper = "https://doi.org/10.1145/3375633"
    paper_type = "doi"
    out = "./paper/one_paper.pdf"
    proxies = {
        'http': 'socks5://127.0.0.1:7890'
    }
    
    scihub_download(paper, paper_type=paper_type, out=out, proxies=proxies)
  4. Update SciHub links via Python module

    main

    Use the update_link function from scidownl.update_link to refresh the list of available SciHub domains. You can choose between two update methods via the mod parameter:

    • mod='c': Uses the crawling method.
    • mod='b': Uses the brute-force search method.
    from scidownl.update_link import *
    
    # Use crawling method to update available Scihub links.
    update_link(mod='c')
    
    # Use brute force search method to update available Scihub links.
    update_link(mod='b')
  5. Download papers using the SciHub Python module

    main

    You can import SciHub from scidownl.scihub to integrate paper downloading into your Python workflows. This is useful for processing lists of DOIs.

    To download a single paper, instantiate SciHub(doi, output_dir) and call .download(). You can optionally pass choose_scihub_url_index to specify which mirror to use.

    from scidownl.scihub import *
    
    DOI = "10.1021/ol9910114"
    out = 'paper'
    # Download using the SciHub mirror at index 3
    sci = SciHub(DOI, out).download(choose_scihub_url_index=3)
  6. Download papers using the CLI

    main

    The scidownl download command allows you to download papers using a DOI, PMID, or Title. You can specify multiple identifiers by repeating the corresponding flag.

    Common usage patterns:

    • By DOI: scidownl download --doi <DOI_STRING>
    • By PMID: scidownl download --pmid <PMID_INTEGER>
    • By Title: scidownl download --title "<TITLE_STRING>"
    • Mixing identifiers: scidownl download --doi <DOI> --pmid <PMID>

    Options:

    • -d, --doi TEXT: DOI string. Supports multiple entries.
    • -p, --pmid INTEGER: PMID number. Supports multiple entries.
    • -t, --title TEXT: Title string. Supports multiple entries.
    • -o, --out TEXT: Output directory or file path.
      • If downloading a single paper, this can be a specific file path (e.g., --out ./paper.pdf).
      • If downloading multiple papers, this is always treated as a directory.
      • Note: When specifying a directory, ensure you include a trailing slash (e.g., --out ./downloads/) to prevent the last segment from being treated as a filename.
    • -u, --scihub-url TEXT: Use a specific SciHub domain. If omitted, SciDownl automatically selects the best one from its local database.
    • --proxy <SCHEME=PROXY_ADDRESS>: Use a proxy (e.g., --proxy http=socks5://127.0.0.1:7890).
    # Download with a DOI (filename becomes the paper's title)
    scidownl download --doi https://doi.org/10.1145/3375633
    
    # Download with a PMID to a specific file
    scidownl download --pmid 31395057 --out ./paper/paper-1.pdf
    
    # Download with a proxy
    scidownl download --pmid 31395057 --out ./paper/paper-1.pdf --proxy http=socks5://127.0.0.1:7890
  7. Update and manage SciHub domains

    main

    SciDownl maintains a local SQLite database of available SciHub domains to ensure reliable downloads. You can update this list using the domain.update command.

    Update modes (--mode or -m):

    • crawl (Default): Crawls the SciHub domain source website to find active domains. The source URL is configurable in the global config file under [scihub.domain.updater.crawl] using the key scihub_domain_source.
    • search: Generates combinations of potential SciHub domains and searches for them. This is more thorough but takes longer than crawl mode.

    Commands:

    • scidownl domain.update --mode <crawl|search>: Updates the local domain database.
    • scidownl domain.list: Lists all saved SciHub domains from the local database, showing their URL, SuccessTimes, and FailedTimes (used for priority calculation).
    # Update domains using crawl mode
    scidownl domain.update --mode crawl
    
    # List saved domains
    scidownl domain.list
  8. Use the SciDownl CLI

    main

    The SciDownl command-line tool allows you to download papers using a DOI, manage SciHub mirror URLs, and specify output directories.

    Arguments:

    • -D DOI: The DOI number of the paper (required for downloading).
    • -o OUTPUT: The directory where the PDF will be saved.
    • -c CHOOSE: Select a specific SciHub URL from the list by its index.
    • -u: Update the list of available SciHub links.
    • -l: List all currently saved SciHub URLs and show the active one.
    • -h, --help: Show the help message.
    # Download a paper to the current directory
    $ scidownl -D 10.1021/ol9910114
    
    # Download a paper to a specific directory named 'paper'
    $ scidownl -D 10.1021/ol9910114 -o paper
    
    # List available SciHub URLs
    $ scidownl -l
    
    # Choose a specific SciHub URL by index (e.g., index 5)
    $ scidownl -c 5
    
    # Update available SciHub links
    $ scidownl -u
  9. Manage global configuration with the `config` command

    main

    The config command allows you to inspect the global configuration settings used by scidownl.

    Options:

    • -l, --location: Displays the file path to the global configuration file.
    • -g, --get: Retrieves a specific configuration value. Requires a section and a key.
      • Usage: --get <section> <key>

    Valid sections and keys depend on your current configuration file.

    # Show where the config file is located
    scidownl config --location
    
    # Get a specific value (e.g., from the 'proxy' section, the 'http' key)
    scidownl config --get proxy http
  10. Download papers via the `download` command

    main

    Use the download command to retrieve scientific papers from SciHub using DOIs, PMIDs, or titles. You can specify multiple identifiers by repeating the corresponding flag.

    Options:

    • -d, --doi: DOI string(s). Supports multiple: --doi DOI1 --doi DOI2.
    • -p, --pmid: PMID numbers. Supports multiple: --pmid PMID1 --pmid PMID2.
    • -t, --title: Title string(s). Supports multiple: --title "Title 1" --title "Title 2".
    • -o, --out: Output directory or file path.
      • If --out is not specified, papers are saved to the current directory using the paper's title as the filename.
      • If multiple DOIs, PMIDs, or titles are provided, --out is always treated as an output directory.
    • -u, --scihub-url: Specific SciHub domain URL. If omitted, the tool automatically selects one from the local database.
    • -x, --proxy: Proxy in the format SCHEME=PROXY_ADDRESS (e.g., --proxy http=http://127.0.0.1:7890). This overwrites any proxy settings in the global configuration.
    # Download a single paper by DOI to a specific directory
    scidownl download --doi 10.1038/s41586-020-2012-7 --out ./papers/
    
    # Download multiple papers by DOI and PMID
    scidownl download --doi 10.1038/s41586-020-2012-7 --pmid 3201234 --out ./downloads/
    
    # Download using a specific proxy
    scidownl download --doi 10.1038/s41586-020-2012-7 --proxy http=http://127.0.0.1:7890
  11. Update SciHub domains with `domain.update`

    main

    Use this command to refresh the list of available SciHub domains and save them to the local database.

    Options:

    • -m, --mode: The update method to use.
      • crawl (default): Crawls for domains.
      • search: Searches for domains.

    If an invalid mode is provided, the command will fail and list the valid available modes.

    # Update domains using the default 'crawl' mode
    scidownl domain.update
    
    # Update domains using 'search' mode
    scidownl domain.update --mode search