DOCSEARCHbyAlgolia
Submit Docs

Pegasus Workflow Management System

repository·main·Indexed 42 days ago

https://github.com/pegasus-isi/pegasus

Pegasus WMS is a workflow management system that automates the mapping and execution of scientific workflows across heterogeneous distributed computing environments, including HPC, Cloud, and Grids. It provides a comprehensive Python API featuring DAX (Versions 2 and 3) and PDAX (Version 2) for workflow definition, as well as specialized APIs for monitoring, statistics, and database interaction via Stampede. The system supports automatic data management, reliability through error recovery, and execution via tools like pegasus-mpi-cluster and pegasus-aws-batch.

Tokens
218.9K
Snippets
631
Records
933
Agent score
52%

What's inside Pegasus

  1. Overview of HTCondor/Pegasus SFAPI Integration

    main

    The SFAPI integration allows Pegasus and HTCondor to submit jobs to the Perlmutter supercomputer at NERSC in environments where there is no shared filesystem between the submit host and the remote cluster.

    It functions as a BLAHP (Batch Local ASCII Helper Protocol) plugin, using a set of shell scripts to bridge HTCondor's Grid Universe and the remote batch system. Communication with Perlmutter (including file transfers, job submission, status queries, and cancellations) is managed via the NERSC Superfacility API (SFAPI) Python client.

  2. Overview of Pegasus WMS

    main

    Pegasus WMS is a configurable system designed to map and execute scientific workflows across diverse environments, including local laptops, campus clusters, Grids (like Open Science Grid), and clouds (like Amazon EC2).

    Key capabilities include:

    • Portability: Workflows can run on different environments (Condor, Grid, Cloud) without alteration.
    • Scalability: Supports workflows ranging from a few tasks to 1 million tasks.
    • Data Management: Automatically handles replica selection, data transfers, and output registration in data catalogs.
    • Reliability & Error Recovery: Provides automatic retries for jobs and data transfers, workflow-level checkpointing, and re-mapping. If a workflow fails completely, it can provide a 'rescue workflow' containing only the remaining work.
    • Provenance: Captures runtime provenance via the kickstart process, which can be analyzed using tools like pegasus-statistics, pegasus-plots, or SQL queries.

  3. Overview of Pegasus Workflow Management System

    main

    Pegasus WMS is a configurable system designed to map and execute scientific workflows across diverse computational infrastructures, including laptops, clusters, supercomputers, grids, and clouds (e.g., Amazon EC2, Google Cloud).

    Key capabilities include:

    • Automatic Mapping: Bridges high-level workflow descriptions to distributed resources by locating input data and planning data transfers and job submissions.
    • Portability: Workflows can run on Condor pools, Grid infrastructures (Open Science Grid, XSEDE), or HPC clusters without alteration.
    • Performance & Scalability: The mapper optimizes task ordering and grouping to handle workflows ranging from a few tasks to 1 million tasks.
    • Data Management: Automatically handles replica selection, data transfers, and output registration in data catalogs.
    • Reliability & Error Recovery: Provides automatic retries for jobs and transfers, workflow-level checkpointing, and 'rescue workflows' that describe only the remaining work after a failure.

  5. Overview of the Pegasus Python API package

    main

    The pegasus-wms.common package provides a comprehensive suite of Python APIs for interacting with the Pegasus Workflow Management System (WMS). It includes interfaces for workflow definition, monitoring, database interaction, and statistics visualization.

    Key API components available in this package include:

    • DAX API (Versions 2 and 3): For workflow definition and management.
    • PDAX API (Version 2): For workflow definition and management.
    • Monitoring API: For tracking workflow execution and status.
    • Stampede Database API: For interacting with the Stampede database.
    • Pegasus Statistics API: For retrieving workflow performance and execution statistics.
    • Pegasus Plots API: For generating visual representations of workflow data.
    • Pegasus Service: Includes the ensemble manager and the dashboard.
    • Miscellaneous Utilities: Various helper functions for Pegasus operations.

  6. Use the Monitoring REST API to query STAMPEDE databases

    main

    The Monitoring REST API allows developers to query a Pegasus workflow's STAMPEDE database. All URIs are prefixed by /api/v1/user/<string:user>.

    Endpoints return application/json. You can use the pretty-print query argument to receive formatted JSON.

    Error Codes:

    • 401: Authentication failure.
    • 403: Authorization failure.

  7. Explore example workflows on PegasusHub

    main

    PegasusHub is a community-driven gallery of Pegasus workflows located at https://pegasushub.github.io/. It contains a range of examples from beginner-friendly workflows (like the Diamond Workflow) to advanced ones (like the Montage Workflow).

    All workflows on PegasusHub are hosted in individual GitHub repositories and can be cloned using Git. These workflows serve as starting points for creating your own, though you may need to modify site and transformation catalogs (e.g., paths to scratch directories or software) to match your specific environment.

  8. Use pegasus-aws-batch to run tasks on Amazon AWS Batch

    main

    The pegasus-aws-batch client is used to execute jobs on Amazon AWS Batch. It manages the lifecycle of AWS Batch entities (job definitions, compute environments, and job queues), uploads required data files to an S3 bucket, and automatically fetches job stdout from CloudWatch Logs to your local filesystem.

    To run jobs, you must provide a JSON-formatted job submit file which contains the job descriptions. You can also specify files to be uploaded to S3 via the --files flag or the TRANSFER_INPUT_FILES environment variable within the job description.

    pegasus-aws-batch [-h] [-C propsfile] [-L <error, warn, info, debug, trace>] [-l logfile] [--merge-logs prefix] [-a AWS account id] [-r AWS region] [--create] [--delete] [-p prefix] [-j jsonfile or arn or job-definition name] [--ce jsonfile or arn or compute-environment name] [-q jsonfile or arn or queue name] [-s s3 bucket URL] [-f file1[,file2 ...]] job submit file

  9. Use the pegasus-em CLI to manage workflow ensembles

    main

    The pegasus-em command-line tool is used to submit and monitor ensembles of workflows. It operates via a server-client model where an ensemble manager server coordinates the execution, planning, and triggering of multiple workflows.

    Basic Usage Pattern:

    1. Start the server using pegasus-em server.
    2. Create an ensemble using pegasus-em create.
    3. Submit workflows to the ensemble using pegasus-em submit.
    4. Monitor or manage workflows using status, replan, or rerun commands.
    pegasus-em COMMAND [options] [ARGUMENT...]

  10. What is Kickstart and how does it work?

    main

    Kickstart is a lightweight C executable included with the pegasus worker package. It is used to launch jobs on remote nodes to capture runtime provenance information.

    Key behaviors:

    • Provenance: It captures provenance data and writes it as a YAML record to its own stdout.
    • Output Files: The stdout is captured in the workflow submit directory as <job>.out.0NN, where 0NN is a rotation index for each job retry.
    • Automatic Activation: Jobs are launched via Kickstart by default unless explicitly disabled during the pegasus-plan phase.

    Limitations (Kickstart is NOT compatible with):

    1. HTCondor Standard Universe Jobs: Pegasus automatically disables Kickstart for these; use dagman.out and logs for monitoring instead.
    2. MPI Jobs: Pegasus automatically disables Kickstart for these; use dagman.out and logs for monitoring instead.

  11. What is a Transformation Catalog?

    main
    A Transformation Catalog (TC) maps logical transformations (e.g., example::keg:1.0) to physical executables on a system. It provides metadata such as the required system architecture, operating system, environment variables (profiles), and whether the executable is INSTALLED on a site or needs to be STAGEABLE (transferred to the site).

  12. Overview of pegasus-monitord

    main

    pegasus-monitord is a monitoring daemon launched per workflow (or once for a hierarchical workflow) that tracks workflow execution in real-time. It parses DAGMan's dagman.out files and job logs to populate a runtime database (the Stampede schema) and generate a jobstate.log file.

    Key capabilities:

    • Automatically invoked by pegasus-run.
    • Tracks workflows and sub-workflows automatically.
    • Can mine information from submit/output files to populate databases or write NetLogger events.
    • Sends real-time notifications to users.
    • Supports automatic resumption when a workflow is re-submitted with a rescue DAG.