Cromwell Workflow Management System

Centaur is Cromwell's integration testing suite. It is designed to ingest .test files and transform them into runnable test suites using ScalaTest. The suite includes functionality to manage the lifecycle of the Cromwell server (start, stop, restart) and provides abstractions for asserting against workflow metadata and outputs.

For detailed developer documentation, refer to the official Cromwell documentation on Centaur.

Cromwell is a workflow engine that supports several versions of the Workflow Description Language (WDL). Users can describe their workflows using the following versions:

• WDL Draft 2: The original language Cromwell was built for. It supports the majority of the Draft-2 specification.
• WDL 1.0: A more recent version that includes support for nested scatter operations and the localization_optional optimization.
• WDL 'development': Cromwell provides support for the ongoing development version of the WDL specification to ensure compatibility with upcoming features as the spec evolves.

Cromwell includes several built-in backends:

• Local: Runs jobs on the local machine.
• HPC: High-Performance Computing backends including Sun Grid Engine (SGE), LSF, HTCondor, and SLURM. These can run jobs as subprocesses or via a dispatcher (using bash, qsub, or bsub) and support launching in Docker containers.
• Google Cloud (GCPBatch): Launches jobs on Google Compute Engine via Google Batch.
• GA4GH TES: Launches jobs on servers supporting the GA4GH Task Execution Schema.
• AWS Batch (beta): Uses Job Queues on AWS Batch.

The Workflow Object Model (WOM), located in the wom package, is a directed acyclic graph (DAG) that represents a parsed workflow. It captures the relationships between workflow inputs, outputs, calls, and their dependencies.

In the WOM graph:

• Nodes represent individual pieces of the workflow (like tasks or calls).
• Edges represent the flow of data, where an output from one node (represented as a hexagon) becomes an input for a subsequent node (represented as an oval).
• External Inputs: If a call requires an input that is not provided by a previous call within the graph, Cromwell generates an wom.graph.ExternalGraphInputNode. These must be provided by the user in the workflow's inputs file.
• Outputs: When a call's output is piped out of the workflow bounds, it becomes an wom.graph.ExpressionBasedGraphOutputNode.

A backend is a software layer that acts as an interface between the Cromwell engine and an underlying computing platform (e.g., Google Cloud Platform, AWS, TES, or Local). While Cromwell manages the workflow logic and dependencies, the backend is responsible for the actual execution of individual tasks on the platform.

When Cromwell requests a job, the backend receives a collection of information including:

• A Unix command line to run.
• Input file mappings (where files live vs. where the command expects them).
• Output file mappings (where the command writes vs. where they should end up).
• An optional Docker image for the execution environment.
• Arbitrary key/value pairs for platform tuning (e.g., memory or CPU requirements).

Sub-workflow execution files (scripts, logs, and outputs) are stored within the parent workflow's call directory. The directory structure follows this pattern:

{parent_execution_dir}/call-{sub_workflow_call_name}/{sub_workflow_name}/{sub_workflow_id}/

For example, if a main workflow calls a sub-workflow named hello_and_goodbye, the path would look like: cromwell-executions/main_workflow/[main_id]/call-hello_and_goodbye/hello_and_goodbye/[sub_id]/

It is important to distinguish between a workflow-level failure and a retryable failure:

• Retryable failures: These do not trigger workflow failure modes. An example is a preemptible VM being preempted. If a job fails due to a retryable reason, Cromwell will retry the job, and the workflow continues normally regardless of whether the failure mode is NoNewCalls or ContinueWhilePossible.
• Non-retryable failures: These trigger the configured failure mode (NoNewCalls or ContinueWhilePossible).

Interaction Example: If job B fails with a non-retryable failure and job A fails with a retryable failure:

• Under NoNewCalls: Job A will not be retried because the workflow has already entered the failure state.
• Under ContinueWhilePossible: Job A will be retried because Cromwell is still attempting to run all possible independent branches.

The IO subsystem in Cromwell uses a hierarchy of actors to manage input/output operations. All messages are initially sent to IoActorProxy. The proxy then routes messages based on whether they contain a Promise:

1. Messages without a Promise: Routed directly to IoActor.
2. Messages with a Promise: Routed to IoPromiseProxyActor. This actor forwards the message to IoActor and completes the provided Promise once the response from IoActor is received.

IoActor includes built-in back-pressure (always enabled) and configurable throttling to prevent the incoming message queue from overflowing.

When running tasks in AWS Batch, Cromwell handles file movement and environment variables as follows:

• Files: Input files are read from S3 into the container, and output files are copied back to S3 upon completion.
• Automatic S3 Files: Three specific files are written to the S3 bucket using these environment variable names:
  • AWS_CROMWELL_RC_FILE: The return code of the task.
  • AWS_CROMWELL_STDOUT_FILE: The STDOUT of the task.
  • AWS_CROMWELL_STDERR_FILE: The STDERR of the task.
• Logging: STDOUT and STDERR are also streamed to the task's CloudWatch logs.