DOCSEARCHbyAlgolia
Submit Docs

nf-core/fetchngs

repository·master·Indexed 42 days ago

https://github.com/nf-core/fetchngs

A bioinformatics pipeline used to automate the retrieval of metadata and raw FastQ sequencing data from public repositories including SRA, ENA, DDBJ, and GEO. It resolves identifiers to experiment-level IDs, fetches metadata via the ENA API, and supports multiple download methods including FTP, Aspera, and sra-tools. The pipeline produces collated samplesheets compatible with various nf-core pipelines such as rnaseq, sarek, and mag.

Tokens
6.1K
Snippets
24
Records
33
Agent score
51%

What's inside nf-core/fetchngs

  1. Overview of nf-core/fetchngs

    master
    nf-core/fetchngs is a bioinformatics pipeline designed to fetch metadata and raw FastQ files from public databases. It currently supports SRA, ENA, DDBJ, and GEO identifiers. The pipeline resolves these IDs to experiment-level IDs, fetches metadata via the ENA API, downloads FastQ files using various methods, and produces a collated samplesheet containing metadata and file paths.

  2. Understand nf-core/fetchngs output structure

    master

    The nf-core/fetchngs pipeline generates several directories within your specified results directory. The output is categorized into downloaded sequencing data, metadata, and pipeline execution reports.

    Sequencing Data and Metadata

    • fastq/: Contains the downloaded *.fastq.gz files (paired-end or single-end).
    • fastq/md5/: Contains *.md5 checksum files for files downloaded from the ENA.
    • samplesheet/: Contains files for downstream pipeline integration:
      • samplesheet.csv: An auto-created samplesheet containing metadata and paths to the FastQ files.
      • id_mappings.csv: A mapping file used to rename samples using specific metadata fields.
      • multiqc_config.yml: A configuration file for MultiQC to facilitate bulk renaming of sample names from database IDs.
    • metadata/: Contains ENA metadata files (*.runinfo_ftp.tsv and *.runinfo.tsv).

    Pipeline Execution Information

    • pipeline_info/: Contains execution and software reports:
      • Nextflow Reports: execution_report.html, execution_timeline.html, execution_trace.txt, and pipeline_dag.dot/pipeline_dag.svg.
      • Pipeline Reports: pipeline_report.html, pipeline_report.txt, and software_versions.yml. Note: pipeline_report* files are only generated if the --email or --email_on_fail parameters are used.
      • Configuration: params.json contains the parameters used during the run.

  3. Supported identifier types for nf-core/fetchngs

    master

    The pipeline automatically downloads and processes raw FastQ files from public repositories using various identifier types. You can provide these identifiers in a file (one per line) via the --input parameter. Supported identifier types include:

    • SRA: SRR, SRX, SRS, SAMN, SRP, SRA, PRJNA
    • ENA: ERR, ERX, ERS, SAMEA, ERP, ERA, PRJEB
    • DDBJ: DRR, DRX, DRS, SAMD, DRP, DRA, PRJDB
    • GEO: GSM, GSE

    Note: If SRR/ERR/DRR run IDs are provided, the pipeline resolves them to their corresponding SRX/ERX/DRX experiment IDs to allow merging multiple runs from the same experiment (similar to merging multiple libraries from the same sample).

  4. Handle samples with more than 2 FastQ files

    master

    The ENA API may only list primary FastQ files, but some samples (e.g., 10x or UMI-based preps) contain additional technical FastQ files that can only be retrieved via sra-tools.

    If you anticipate that a sample has more than 2 FastQ files, you must use the sratools download method to ensure all files (including technical reads) are captured.

    --download_method sratools

  5. Use GitHub Codespaces for development

    master
    You can contribute to nf-core/fetchngs without a local environment by using GitHub Codespaces. The repository includes a devcontainer configuration that provides a browser-based VS Code environment with Nextflow, nf-core/tools, and other utilities pre-installed via Docker.

  6. Use samplesheet and MultiQC outputs for sample renaming

    master

    When running nf-core/fetchngs, the pipeline produces a samplesheet/ directory designed to help you manage sample identities in downstream workflows.

    To customize how samples are named from their database IDs, use the --sample_mapping_fields parameter. This affects the generation of:

    1. id_mappings.csv: Used to map database IDs to more informative names.
    2. multiqc_config.yml: A config file you can pass to other nf-core pipelines using the --multiqc_config parameter to perform bulk renaming in MultiQC reports.

    Example workflow:

    1. Run fetchngs with --sample_mapping_fields to define your preferred naming metadata.
    2. Use the resulting multiqc_config.yml in your next nf-core pipeline to ensure consistent sample naming in reports.

  8. Contribution quick start for nf-core/fetchngs

    master

    To contribute code to nf-core/fetchngs, follow these steps:

    1. Prerequisites: Ensure Nextflow, nf-core tools, and nf-test are installed.
    2. Issue Tracking: Check for an existing GitHub issue regarding your change. If none exists, create one.
    3. Fork & Branch: Fork the nf-core/fetchngs repository and create a new branch for your changes.
    4. Development: Follow pipeline conventions for code and logic.
    5. Documentation: Update files in docs/, update nextflow_schema.json using nf-core/tools, and update CITATIONS.md.
    6. Testing: Run and/or update tests using nf-test.
    7. Linting: Use nf-core/tools to lint your code.
    8. Pull Request: Submit a PR against the dev branch.

  9. How to add a new pipeline step

    master

    When adding a new step to the pipeline, follow these requirements:

    1. Channels: Define the input channel from the previous process using the naming scheme ch_output_from_<process>. Define intermediate/terminal channels using ch_<previousprocess>_for_<nextprocess>.
    2. Modules: Install a module via nf-core/tools or write a local module. Specify resource requirements (CPUs, memory, time) in conf/base.config using standard withLabel: selectors.
    3. Outputs: Define output channels. Mix version outputs into ch_versions and relevant files into ch_multiqc.
    4. Parameters: Add new parameters to nextflow.config with default values. Use nf-core pipelines schema build to update nextflow_schema.json.
    5. Validation: Add parameter validation to the utils_nfcore_pipeline/main.nf subworkflow.
    6. Documentation: Update usage.md, output.md, and citation.md.
    7. Testing & Linting: Perform local tests, add new tests in tests/, and run linting.

  10. Ensure reproducibility with specific pipeline versions

    master

    To guarantee that the exact same code and software versions are used for future runs, specify a specific pipeline version using the -r flag. You can find available versions on the nf-core/fetchngs releases page.

    Example using version 1.3.1:

    nextflow run nf-core/fetchngs -r 1.3.1 --input ./ids.csv --outdir ./results -profile docker
    nextflow run nf-core/fetchngs -r 1.3.1 --input ./ids.csv --outdir ./results -profile docker

  11. Add a new pipeline to fetchngs

    master

    To add support for a new pipeline in fetchngs, complete these three steps:

    1. Update Extras: Add an entry to pipeline_extras in subworkflows/local/channel_sra_create_csv/main.nf. Map the pipeline name to its extra samplesheet columns (e.g., [strandedness: strandedness]).
    2. Add Tests: Create a new test file in workflows/tests/ named sra_nf_core_pipeline_<pipeline>.nf.test. Set input[5] to the pipeline name and include appropriate tags.
    3. Update Schema: If the new pipeline introduces new parameters, run the schema builder:
      nf-core pipelines schema build
    nf-core pipelines schema build

  12. Use a parameters file for pipeline settings

    master

    To avoid long CLI commands and ensure reproducibility, you can specify pipeline parameters in a yaml or json file using the -params-file flag.

    Warning: Do not use -c <file> to specify pipeline parameters; this will cause errors. The -c flag is reserved for Nextflow configuration files used for tuning resource specifications or module arguments.

    Example usage with a YAML file:

    nextflow run nf-core/fetchngs -profile docker -params-file params.yaml

    Example params.yaml content:

    input: './ids.csv'
outdir: './results/'
    nextflow run nf-core/fetchngs -profile docker -params-file params.yaml