DOCSEARCHbyAlgolia
Submit Docs

NCBI Datasets

repository·master·Indexed 19 days ago

https://github.com/ncbi/datasets

A resource for gathering sequence, annotation, and metadata for genes and genomes from NCBI databases. The suite includes the `datasets` CLI for downloading biological sequence data and the `dataformat` tool for converting metadata from JSON Lines to formats like TSV. It supports various data packages (Gene, Genome, Virus, and Taxonomy) and features advanced CLI capabilities such as dehydration/rehydration for large packages, batch processing via `--inputfile`, and specific filtering flags for assembly versions, release dates, and orthologs.

Tokens
21.8K
Snippets
78
Records
94
Agent score
64%

What's inside ncbi-datasets

  1. Overview of NCBI Datasets command-line tools

    master

    NCBI Datasets provides access to biological data and metadata via web, API, and command-line interfaces. For high-performance computing (HPC) environments or large-scale automated workflows, the CLI is the preferred method to download genomes, genes, ortholog sets, and virus genomes directly to the target system.

    The CLI ecosystem consists of two primary tools:

    1. datasets: The main tool used for retrieving and downloading data packages (genomes, genes, etc.).
    2. dataformat: A companion tool used to explore and convert metadata into formats like TSV or Excel.

  2. Use the NCBI Datasets CLI tools

    master

    The NCBI Datasets suite consists of two primary tools:

    • datasets: Used to download biological sequence data (genomes, genes, etc.) across all domains of life.
    • dataformat: Used to convert metadata included in data packages from JSON Lines format to other formats like TSV.

  3. What is a data package?

    master

    A data package is a compressed ZIP archive provided by NCBI Datasets that contains both biological data and metadata.

    Every data package includes at least one metadata file. In a default genome data package, you will find:

    • dataset_catalog.json: A list of all files in the package grouped by accession, including uncompressed sizes, filetypes, and the API version.
    • assembly_data_report: The main metadata report containing assembly statistics (e.g., BUSCO, ANI, CheckM), biosample info, annotation info, and organism details.

  4. Overview of NCBI Datasets data packages

    master

    NCBI Datasets provides biological data as zip archives. The available package types are:

    1. Gene Data Package
    2. Genome Data Package
    3. Virus Data Package (specialized)
    4. Taxonomy Data Package

  5. How dehydrated genome packages and rehydration work

    master

    For large datasets, NCBI Datasets provides dehydrated genome packages.

    1. Dehydrated Download: Instead of downloading all sequence data, you download a small ZIP archive containing metadata and a fetch.txt file. This fetch.txt file contains the API paths and locations for the actual data files.
    2. Rehydration: You use the datasets rehydrate command to download the actual data files listed in fetch.txt.

    Advantages of rehydration:

    • Faster initial download for large projects.
    • Ability to download files in gzip format to save space using the --gzip flag.
    • Supports resuming interrupted downloads.
    • Allows selective downloading of specific file types (e.g., only GFF files) using the --match flag.
    # 1. Download the dehydrated package
datasets download genome taxon 59204 --dehydrated --include genome,gff3 --filename 59204-dehydrated.zip

# 2. Unzip it
unzip 59204-dehydrated.zip -d 59204

# 3. Rehydrate only the GFF files
datasets rehydrate --directory 59204 --match gff

  6. Use cache packages for SARS-CoV-2 and Influenza

    master
    For SARS-CoV-2 and (Alpha)Influenza, NCBI Datasets provides 'cache packages'. Unlike regular 'made-to-order' packages that are assembled upon request, cache packages are pre-packaged with all available genomes for these specific taxa. They are faster to download because they use optimized NCBI download channels and do not require real-time assembly.

  7. Understand NCBI Datasets data reports

    master
    Data packages include data report files containing metadata about the requested records. These reports follow specific schemas that define available fields, descriptions, and examples.

  9. Understand the NCBI Datasets CLI command structure

    master
    The datasets command follows a hierarchical structure designed to make selection intuitive. Commands are constructed by choosing one option from each level of a hierarchy (e.g., selecting a data type, then a specific resource, then applying filters). This structure allows for granular control over which biological entities are retrieved and how they are filtered.

  10. Understand the difference between Virus and Genome endpoints

    master

    When working with viral data, NCBI Datasets provides two distinct service endpoints with different curation levels and data availability:

    1. Virus endpoint: Sourced from NCBI Virus. It includes highly curated data (both manual and automated) from the International Nucleotide Sequence Database Collaboration (INSDC). This endpoint typically contains a much larger volume of sequences (e.g., for SARS-CoV-2, millions of sequences).
    2. Genome endpoint: Provides access to a subset of virus sequences from NCBI Virus that have been specifically assembled and assigned an NCBI Assembly accession (GCA_/GCF_). This endpoint contains significantly fewer sequences than the Virus endpoint.

    You can compare the sequence counts between these endpoints using the summary command with --limit 0 to get the total count without downloading data.

    # Check number of sequences in the Virus endpoint
datasets summary virus genome taxon sars2 --limit 0

# Check number of sequences in the Genome endpoint
datasets summary genome taxon sars2 --limit 0

  11. Perform large genome downloads using rehydration

    master

    For very large datasets, use a dehydrated download followed by rehydration.

    A dehydrated package contains no genomic data; instead, it includes a fetch.txt file containing locations for the requested files.

    Advantages:

    • Faster initial download.
    • Supports downloading files as .gz during rehydration to save space.
    • Supports resuming interrupted downloads.
    • The fetch.txt file is small and easy to share.

    Workflow

    1. Download dehydrated package: Use the --dehydrated flag and specify which data to include (e.g., --include genome,cds).
    2. Unzip the package.
    3. Rehydrate: Use the datasets rehydrate command pointing to the unzipped directory.

    You can rehydrate all files or use the --match flag to retrieve only specific file types (e.g., --match cds).

    # 1. Download dehydrated package
datasets download genome accession PRJNA489243 --annotated --assembly-source refseq --assembly-level chromosome --include genome,cds --dehydrated --filename vgp_refseq_chrom_annot.zip

# 2. Unzip
unzip vgp_refseq_chrom_annot.zip -d vgp_refseq_chrom_annot

# 3. Rehydrate all files
datasets rehydrate --directory vgp_refseq_chrom_annot

# OR: Rehydrate only files matching 'cds'
datasets rehydrate --directory vgp_refseq_chrom_annot --match cds