DOCSEARCHbyAlgolia
Submit Docs

cwltool Reference Implementation

repository·main·Indexed 18 days ago

https://github.com/common-workflow-language/cwltool

The reference implementation of the Common Workflow Language (CWL) standards. cwltool provides comprehensive validation of CWL files and tools for working with CWL workflows. It can be used as a command-line tool or imported as a Python module to create custom runners. It supports local execution on Linux, OS X, and Windows, and includes utilities for generating input templates, extracting document dependencies, and managing job order overrides.

Tokens
48.8K
Snippets
116
Records
242
Agent score
59%

What's inside cwltool

  1. What is Semantic Annotations for Linked Avro Data (SALAD)?

    main

    SALAD is a schema language designed to describe structured linked data documents in JSON or YAML formats. It acts as a bridge between the record-oriented data modeling of Apache Avro and the Semantic Web (JSON-LD/RDF).

    Key capabilities include:

    • Structural Validation: Ensuring documents follow a defined grammar.
    • Link Checking: Validating object references and URIs.
    • Semantic Enrichment: Providing annotations that allow JSON/YAML documents to be interpreted as Linked Data (JSON-LD).
    • Rich Modeling: Supporting inheritance, template specialization, object identifiers, and object references.
    • Schema Generation: Enabling the generation of JSON-LD contexts and RDF schemas from the Salad schema.

  2. Overview of CWL v1.2 Specifications

    main

    The CWL v1.2 specifications are composed of several distinct documents that define different aspects of the language:

    • User Guide: Provides an introduction to writing CWL command line tools and workflows (currently covers v1.0).
    • Command Line Tool Description Specification: Defines the schema and execution semantics for wrapping and executing command line tools.
    • Workflow Description Specification: Defines the schema and execution semantics for composing workflows from tools and other workflows.
    • Semantic Annotations for Linked Avro Data (SALAD) Specification: Defines the preprocessing steps for loading CWL documents and the schema language used for the specifications.

  3. Introduction to Common Workflow Language (CWL)

    main
    Common Workflow Language (CWL) is a specification for describing analysis workflows and tools. It is designed to make workflows portable and scalable across various environments, including workstations, clusters, cloud, and high-performance computing (HPC) environments. CWL is particularly suited for data-intensive sciences like Bioinformatics, Medical Imaging, Astronomy, Physics, and Chemistry. It leverages technologies like JSON-LD for data modeling and Docker for portable runtime environments.

  4. Access the unreleased CWL v1.2 specification

    main

    The cwltool/schemas/v1.2.0-dev2/ directory contains the unreleased v1.2 version of the Common Workflow Language standard and its development history.

    To view the specification in a rendered format, you can use the website.sh script from the cwl-website repository. Alternatively, you can browse the online version of the v1.2 spec at the official CWL website.

    https://www.commonwl.org/v1.2/

  6. Recommended software for cwltool

    main

    To ensure full compatibility with CWL tutorials and features, the following software is strongly suggested:

    • Container Engines: One of Podman, Docker, Singularity/Apptainer, or udocker.
    • Node.js: Recommended for evaluating CWL Expressions quickly. This is a requirement for udocker users and highly recommended for all other container engine users.

  7. Understand the CWL Specification structure

    main

    The Common Workflow Language (CWL) specifications are organized into several distinct documents, each covering a different aspect of the language:

    • User Guide: Provides an introduction to writing CWL command line tools and workflows.
    • Command Line Tool Description Specification: Defines the document schema and execution semantics for wrapping and executing command line tools.
    • Workflow Description Specification: Defines the document schema and execution semantics for composing workflows from components like command line tools and other workflows.
    • Semantic Annotations for Linked Avro Data (SALAD) Specification: Defines the preprocessing steps required when loading CWL documents and the schema language used for the specifications.

  8. What is a ProcessGenerator and how does it work?

    main

    A ProcessGenerator is an experimental, unofficial CWL extension used for bootstrapping or pre-processing steps. It is a subtype of cwl:Process that executes a concrete CWL process (like a CommandLineTool, Workflow, or ExpressionTool) to generate a new CWL file, and then immediately executes that generated CWL file.

    Workflow Lifecycle:

    1. Execution of run: The ProcessGenerator runs the tool specified in its run field. This tool is expected to produce a CWL description as an output (e.g., via stdout or a file).
    2. Loading: The ProcessGenerator loads the newly generated CWL file.
    3. Execution: The ProcessGenerator executes the generated CWL process using the same input object provided to the original ProcessGenerator (including any additional parameters intended for the generated tool).
    4. Output: The output of the generated script is returned as the final output of the ProcessGenerator.
    - name: ProcessGenerator
     type: record
     inVocab: true
     extends: cwl:Process
     documentRoot: true
     fields:
       - name: class
         jsonldPredicate:
           "_id": "@type"
           "_type": "@vocab"
         type: string
       - name: run
         type: [string, cwl:Process]
         jsonldPredicate:
           _id: "cwl:run"
           _type: "@id"
           subscope: run
         doc: |
           Specifies the process to run.

  9. What is Schema Salad?

    main

    Schema Salad is a schema language designed for describing JSON or YAML structured linked data documents. It is based on JSON-LD and Apache Avro.

    Key Features:

    • Data Modeling: Supports inheritance, template specialization, object identifiers, and object references.
    • Validation: Provides rules for preprocessing, structural validation, and link checking.
    • Semantic Web Integration: Acts as a bridge between document/record-oriented data and the Semantic Web by enabling transformation to RDF, generation of JSON-LD contexts, and RDF schema generation.
    • Documentation: Provides robust support for inline documentation and automated documentation generation.

  10. Format of a conformance test entry

    main

    A conformance test is defined in a YAML document. Each entry represents a single test case consisting of a CWL document, an input object, and expected outputs.

    - doc: Test command line with optional input (missing)
   id: cl_optional_inputs_missing
   tool: tests/cat1-testcli.cwl  
   job: tests/cat-job.json
   output:
     args: [cat, hello.txt]
   tags: [ required, command_line_tool ]

  11. Use $import to embed external Salad documents

    main

    The $import directive allows you to embed an external Salad document (YAML/JSON) into your current document. During preprocessing, the $import node is replaced by the object or array contained in the target resource.

    Key Rules:

    • The directive must be an object with exactly one field: "$import": "URI".
    • Supported protocols: file, http, and https.
    • The imported document does not inherit the context of the importing document; it uses its own URI as the base URI.
    • You can use document fragments (e.g., file.yml#fragment) to import only a specific object from the target document.
    • If the referenced resource or fragment does not exist, it is a fatal error.
    # parent.yml
{
  "form": {
    "bar": {
      "$import": "import.yml"
      }
    }
  }
}

# import.yml
{
  "hello": "world"
}

# Resulting structure:
{
  "form": {
    "bar": {
      "hello": "world"
    }
  }
}

  12. Extend CWL with metadata and implementation extensions

    main

    You can add additional information to CWL objects using extensions and metadata:

    • Metadata: Information about workflows, tools, or input items. It is recommended to use schema.org concepts where possible.
    • Implementation Extensions: Fields used for non-execution purposes (e.g., GUI presentation or authorship) must use a namespace prefix listed in the $namespaces section of the document.
    • Execution Extensions: Any implementation extension that modifies execution semantics must be listed in the requirements field.
    • Input Metadata: Can be represented by input parameters that are explicitly propagated to outputs.