DOCSEARCHbyAlgolia
Submit Docs

Nextflow Documentation

repository·master·Indexed 25 days ago

https://github.com/nextflow-io/nextflow

Nextflow is a workflow system for creating scalable, portable, and reproducible workflows based on the dataflow programming model. It abstracts parallel and distributed execution across various platforms. The system supports a variety of plugins for cloud integration, including nf-amazon (AWS Batch, S3, Fusion), nf-azure (Azure Batch, Blob Storage), nf-google (Google Cloud Batch, GCS), nf-k8s (Kubernetes), nf-cloudcache for cloud-based caching, nf-codecommit for AWS CodeCommit integration, and nf-console for an interactive Groovy console.

Tokens
176.8K
Snippets
586
Records
1.2K
Agent score
86%

What's inside Nextflow

  1. Overview of static typing features

    master

    Nextflow static typing allows you to model and validate data structures as they flow through a pipeline. You can apply type annotations at various stages of the pipeline independently or together:

    • Typed parameters: Type annotations for the params block.
    • Typed processes: Type annotations for process inputs/outputs, stage: and topic: sections, records, and tuples.
    • Typed workflows: Type annotations for workflow inputs/outputs, dataflow types, and syntax restrictions.
    • Typed outputs: Type annotations for the output block.
    • Operators: Using core operators specifically designed for static typing.

  2. Overview of Nextflow capabilities

    master

    Nextflow is a workflow system based on the dataflow programming model designed for creating scalable, portable, and reproducible workflows.

    Execution Platforms:

    • Local machine
    • HPC schedulers
    • AWS Batch
    • Azure Batch
    • Google Cloud Batch
    • Kubernetes

    Software Dependency Management:

    • Conda
    • Spack
    • Docker
    • Podman
    • Singularity

  3. Use the nextflow.cloud.azure package for Azure Batch execution

    master
    The nextflow.cloud.azure package provides the implementation for the Azure Batch executor in Nextflow. It allows you to run Nextflow workflows using Azure Batch services by leveraging the AzBatchExecutor class. This package handles task submission, script launching via AzBatchScriptLauncher, and file copying using AzFileCopyStrategy.

  4. Use the nextflow.cloud.google package for Google Batch execution

    master

    The nextflow.cloud.google package provides the implementation for the Google Batch executor in Nextflow. It allows you to run workflows using Google Batch as the compute backend. The package includes the following core components:

    • GoogleBatchExecutor: The primary executor class.
    • GoogleBatchTaskHandler: Handles task submission.
    • GoogleBatchScriptLauncher: A BashWrapperBuilder implementation used for launching scripts.
    • BatchConfig: Configuration object used by the GoogleBatchExecutor.

  5. Define Processes and Workflows

    master
    • Processes: Define a task that runs in an isolated environment. Each call to a process adds a node to the dataflow graph. Nextflow executes these tasks asynchronously as inputs become available.
    • Workflows: Describe a dataflow graph consisting of processes and operators connected by channels.

  6. Understand the Nextflow Module System capabilities

    master

    The Nextflow module system provides a standardized mechanism for package management, versioning, and distribution of reusable process definitions. It enables:

    1. Remote module inclusion via a registry.
    2. Semantic versioning with dependency resolution.
    3. Unified Nextflow Registry access.
    4. First-class CLI support for managing modules (commands include install, publish, search, list, remove, and run).

  7. Understand the `nextflow.secret` package

    master
    The nextflow.secret package provides the interface for secrets providers and includes a built-in implementation for a local secrets store. It is used by the SecretsLoader to retrieve secrets during the execution of scripts or when building configurations. The default implementation, LocalSecretsProvider, stores secrets as key-value pairs in a local JSON file.

  8. Use the nextflow.cloud.aws package for AWS Batch execution

    master
    The nextflow.cloud.aws package provides the implementation for the AWS Batch executor in Nextflow. It allows you to run workflows using AWS Batch as the compute engine, managing tasks via AwsBatchExecutor, AwsBatchTaskHandler, and AwsBatchScriptLauncher. It also includes specialized configuration for AWS S3 and AWS Batch-specific settings.

  9. Understand the Multi-Revision Asset Management structure

    master

    The new asset management system uses a bare repository to store shared Git objects, which allows multiple revisions of the same pipeline to be checked out concurrently without duplicating the entire object database. This is achieved using Git alternates.

    Storage Layout:

    • Bare Repository: Located at ~/.nextflow/assets/.repos/<org>/<project>/bare/. This is the source of truth for all objects and refs.
    • Shared Clones: Located at ~/.nextflow/assets/.repos/<org>/<project>/clones/<commit-sha>/. Each revision has its own directory, but its .git/info/alternates file points back to the bare repository's objects to save space.
    • Working Directories: The actual pipeline files are checked out into the project's working directory.

    Key Benefits:

    • Concurrency: Run different versions (e.g., v1.0 and v2.0-dev) of the same pipeline simultaneously.
    • Efficiency: Disk usage is minimized because objects are shared via the bare repository.
    • Atomicity: Downloading a new revision does not interfere with currently running pipelines.

  10. Understand Nextflow core concepts: Processes and Dataflow

    master

    Nextflow pipelines are composed of processes connected by asynchronous dataflow structures called channels and values.

    • Processes: Define a specific task using a script (Bash, Python, etc.). Each process specifies its input and output files/values.
    • Dataflow: Execution order is determined by data dependencies rather than script order. A process executes as soon as its required inputs are provided by a channel.
    • Channels: Used to move data between processes. For example, channel.fromPath() can turn files into data streams for processes to consume.
    // Script parameters
params.query = "/some/data/sample.fa"
params.db = "/some/path/pdb"

// Define the blast_search process
process blast_search {
  input:
  path query
  path db

  output:
  path "top_hits.txt"

  script:
  """
  blastp -db $db -query $query -outfmt 6 > blast_result
  cat blast_result | head -n 10 | cut -f 2 > top_hits.txt
  """
}

// Define the extract_top_hits process
process extract_top_hits {
  input:
  path top_hits
  path db

  output:
  path "sequences.txt"

  script:
  """
  blastdbcmd -db $db -entry_batch $top_hits > sequences.txt
  """
}

// Define the workflow
workflow {
  def query_ch = channel.fromPath(params.query)
  blast_search(query_ch, params.db)
  extract_top_hits(blast_search.out, params.db).view()
}

  11. Understand Seqera Intelligent Compute run identifier propagation

    master

    When running workflows using the seqera executor (Seqera Intelligent Compute scheduler), Nextflow captures a scheduler-assigned run identifier and propagates it to the Platform. This allows the Platform to associate the workflow record with authoritative cost and resource-usage metrics (such as settled cost and VM types) provided by the scheduler.

    Key details for users:

    • The identifier is stored within the PlatformMetadata under the schedRunId field (workflow.platform.schedRunId).
    • Propagation is handled via a best-effort PATCH /workflow/{workflowId} request sent by the TowerObserver as soon as the ID is assigned (typically on the first task submission).
    • The presence of schedRunId in the Platform metadata is the indicator that a run is scheduler-managed; no separate 'scheduler enabled' flag is required.

  12. Understand Nextflow plugin types

    master

    Nextflow supports two types of plugins to extend functionality:

    1. Core plugins: These do not require manual configuration. The latest versions are automatically installed when a pipeline requests them. Examples include:

      • nf-amazon: AWS support.
      • nf-azure: Microsoft Azure support.
      • nf-cloudcache: Cloud cache support.
      • nf-console: Nextflow REPL console implementation.
      • nf-google: Google Cloud support.
      • nf-tower: Seqera Platform support.
      • nf-wave: Wave containers service support.

    2. Third-party plugins: These must be explicitly configured via configuration files or at runtime.

    To disable the automatic retrieval of core plugins, set the environment variable NXF_PLUGINS_DEFAULT=false.