pysradb Documentation

repository·develop·Indexed 18 days ago

https://github.com/saketkc/pysradb

A Python package and CLI for interacting with SRAdb to retrieve, manage, and download datasets from SRA, ENA, and GEO. It features tools for converting between biological database identifiers (SRP, GSE, GSM, SRR, SRX, SRS), searching for datasets, and downloading SRA/ENA projects. The library also supports metadata enrichment using optional LLM backends to infer attributes like age, sex, and disease from metadata.

Tokens
38.4K
Snippets
163
Records
195
Agent score
62%

What's inside pysradb

  1. Introduction to pysradb

    develop

    pysradb is a Python package designed to provide a programmatic interface for accessing metadata and downloading sequencing data from the following repositories:

    • NCBI's Sequence Read Archive (SRA)
    • European Bioinformatics Institute's European Nucleotide Archive (ENA)
  2. Overview of pysradb submodules

    develop

    The pysradb package is organized into several functional submodules that allow users to interact with SRA, ENA, and GEO metadata. Key modules include:

    • pysradb.cli: Provides command-line interface functionality.
    • pysradb.download: Handles the downloading of metadata.
    • pysradb.enrichment: Provides tools for enriching metadata with additional information.
    • pysradb.search: Used for searching metadata across different databases.
    • pysradb.sraweb and pysradb.geoweb: Interfaces for interacting with SRA and GEO web services.
    • pysradb.filter_attrs: Utilities for filtering metadata attributes.
    • pysradb.taxid2name: Utilities for mapping TaxIDs to names.
    • pysradb.exceptions: Defines custom exception classes for the package.
  3. Available pysradb learning notebooks

    develop

    The following notebooks are available to guide you through specific pysradb tasks:

    1. Python API quickstart: Basic introduction to using the Python library.
    2. Command-line download preview: Using the CLI to preview downloads.
    3. Parallel download planning with the Python API: Managing large-scale parallel downloads.
    4. Converting SRA to FASTQ using Conda: Workflow for format conversion.
    5. Selecting subsets of a project: How to download specific parts of a project.
    6. Metadata for multiple SRPs: Handling multiple SRA Project identifiers.
    7. Searching SRA, GEO, and ENA: Querying multiple biological databases.
    8. Extracting identifiers from PMC and DOI records: Working with publication identifiers.
    9. Metadata enrichment with optional LLM backends: Using Large Language Models to enhance metadata.
    10. Parsing BioScience search results: Processing results from BioScience searches.
  4. Important behaviors of metadata enrichment

    develop

    When using the enrichment feature, keep the following constraints and behaviors in mind:

    • Incompatibility with --detailed: The --enrich flag cannot be combined with the --detailed flag. Enrichment relies on the pysraweb API output and standardizes it before processing.
    • Row Limits: If your input dataframe contains more than 15 rows, the CLI will prompt you to either enrich all rows or just the first 15 rows for a quick review. This is to prevent long wait times for large datasets.
    • Execution: Enrichment runs in parallel across rows and displays a progress bar. A summary line with the elapsed time is printed upon completion.
  5. Perform ultrafast FASTQ downloads with Aspera

    develop

    If you have aspera-client installed on your system, pysradb can perform high-speed downloads using the Aspera protocol.

    CLI Usage: Use the --use_ascp flag to enable Aspera and the -t flag to specify the number of threads.

    Python Usage: Pass use_ascp=True and the desired number of threads to the client.download() method.

    $ pysradb download -t 8 --use_ascp -p SRP002605
    from pysradb.sraweb import SRAweb
    
    client = SRAweb()
    client.download("SRP098789", use_ascp=True, threads=8)
  6. Install pysradb in development mode

    develop

    To set up pysradb for local development, fork the repository, clone it, and install it in editable mode with the necessary development extras (test and docs). This allows you to make changes to the source code and have them reflected immediately in your environment.

    $ git clone git@github.com:your_name_here/pysradb.git
    $ cd pysradb/
    $ python -m pip install --upgrade pip
    $ python -m pip install --editable ".[test,docs]"
  7. Install pysradb via pip or conda

    develop

    You can install the stable version of pysradb using either pip or conda.

    Using pip:

    pip install pysradb

    Using conda:

    conda install -c bioconda pysradb

    Recommended: New Conda Environment To avoid dependency conflicts with existing packages, it is recommended to create a fresh environment specifically for pysradb:

    conda create -c bioconda -n pysradb PYTHON=3.13 pysradb
  8. Verify changes with linting, tests, and builds

    develop

    Before submitting a pull request, ensure your changes pass local linting, testing, packaging, and documentation builds.

    • Linting: Use flake8 on the pysradb directory and tests.
    • Testing: Use coverage to run pytest with source tracking and generate a report.
    • Packaging: Use python -m build to ensure the package builds correctly.
    • Documentation: Use make docs to build the documentation (which includes notebooks and uses the Furo Sphinx theme).
    $ flake8 pysradb tests
    $ coverage run --source pysradb -m pytest
    $ coverage report -m
    $ python -m build
    $ make docs
  9. Explore pysradb workflows via notebooks

    develop

    The pysradb project provides a series of curated Jupyter notebooks that demonstrate common workflows. These notebooks are useful for learning both the Python API and the Command Line Interface (CLI). Download-oriented notebooks are designed to preview commands and data subsets rather than downloading large sequencing files.

    Available notebook workflows:
    
    1. Python API quickstart
    2. Command-line download preview
    3. Parallel download planning with the Python API
    4. Converting SRA to FASTQ using Conda
    5. Selecting subsets of a project
    6. Metadata for multiple SRPs
    7. Searching SRA, GEO, and ENA
    8. Extracting identifiers from PMC and DOI records
    9. Metadata enrichment with optional LLM backends
    10. Parsing BioScience search results
  10. Retrieve SRA metadata by accession number

    develop

    You can retrieve metadata for a specific SRA accession using either the CLI or the Python API. When using Python, the results are returned as a pandas DataFrame, allowing for standard pandas query and selection operations.

    To get basic metadata, use the metadata command or sra_metadata method. To include more detailed information, such as download URLs, enable the detailed flag/parameter.

    from pysradb.sraweb import SRAweb
    
    client = SRAweb()
    # Basic metadata
    df = client.sra_metadata('SRP016501')
    
    # Detailed metadata including download URLs
    df_detailed = client.sra_metadata('SRP016501', detailed=True)
  11. Obtain detailed SRA metadata including file locations

    develop

    To obtain extended metadata, including file locations (e.g., .fastq or .sra URLs), aliases, and geographic information, use the detailed flag. This is useful for finding download links like ena_fastq_ftp or sra_url.

    # Using the CLI
    $ pysradb metadata SRP265425 --detailed
    
    # Using Python
    from pysradb.sraweb import SRAweb
    
    client = SRAweb()
    df = client.metadata("SRP265425", detailed=True)
    print(df)
  12. Install pysradb via conda

    develop

    You can install pysradb using conda from the bioconda channel. This method installs all dependencies except aspera-client (which is highly recommended but not strictly required).

    If you have an existing environment with many packages, conda may be slow. It is recommended to create a new dedicated environment for pysradb.

    # Install into current environment
    conda install -c bioconda pysradb
    
    # Recommended: Create a new environment named 'pysradb'
    conda create -c bioconda -n pysradb PYTHON=3 pysradb