DOCSEARCHbyAlgolia
Submit Docs

Prefect Workflow Orchestration

repository·main·Indexed 12 days ago

https://github.com/PrefectHQ/prefect

A workflow orchestration framework for building resilient, dynamic data pipelines in Python. Prefect allows developers to turn scripts into production-ready workflows with scheduling, retries, and observability. It includes a lightweight `prefect-client` for remote server or Prefect Cloud interaction in ephemeral environments like AWS Lambda, and provides integrations for AWS services including S3, Lambda, and RDS IAM authentication.

Tokens
919.8K
Snippets
3.2K
Records
4.7K
Agent score
97%

What's inside Prefect

  1. Introduction to Prefect

    main

    Prefect is an open-source orchestration engine designed to turn Python functions into production-grade data pipelines. It allows you to build and schedule workflows using native Python without requiring specialized DSLs or complex configuration files.

    Core Capabilities:

    • Pythonic Workflows: Full support for type hints, async/await, and standard Python patterns (loops, conditionals).
    • State & Recovery: Automatic tracking of success, failure, and retry states, allowing you to resume interrupted runs.
    • Flexible Execution: Run flows locally for development or deploy them to containers, Kubernetes, or cloud services.
    • Event-Driven Orchestration: Trigger flows via schedules, external events, or API calls, and use automations to react to state changes.
    • Dynamic Runtime: Create tasks dynamically at runtime based on data or conditions.

  3. Overview of Prefect Integrations

    main

    Prefect integrations are PyPI packages that allow you to build and integrate your workflows with third-party services. These integrations act as connectors or plugins to extend Prefect's capabilities to various cloud providers, data tools, and infrastructure platforms.

    Maintained by Prefect

    Many core integrations are maintained directly by the Prefect team, including:

    • Cloud Providers: prefect-aws, prefect-azure, prefect-gcp
    • Infrastructure & Orchestration: prefect-docker, prefect-kubernetes, prefect-dask, prefect-ray, prefect-shell
    • Data & Storage: prefect-databricks, prefect-dbt, prefect-snowflake, prefect-sqlalchemy
    • Communication & VCS: prefect-github, prefect-gitlab, prefect-bitbucket, prefect-slack, prefect-email

    Third-Party Maintained Integrations

    Some integrations are maintained by external organizations or community members:

    • Coiled: Managed by Coiled
    • Fivetran: Managed by Fivetran
    • Slurm: Managed by EBI Metagenomics

  5. Manage concurrency limits in `prefect.server.models.concurrency_limits`

    main

    The prefect.server.models.concurrency_limits module provides functions for interacting with concurrency limit ORM objects.

    Note: These functions are intended for internal use by the Prefect REST API. Direct use by end-users is generally not recommended unless building custom server-side extensions or integrations.

  6. Resources for prefect-aws integration

    main

    When working with the prefect-aws integration, use the following resources for technical reference and support:

    Technical Documentation

    • SDK Reference: For complete API documentation regarding all prefect-aws blocks and tasks, visit the prefect-aws SDK Reference.
    • ECS Deployment: For step-by-step instructions on deploying workflows on Amazon ECS, refer to the ECS Deployment Guide.
    • Secrets Management: To learn how to use AWS credentials with third-party services, see the Prefect Secrets Management guide.

    AWS & SDK Documentation

    Support

  7. Manage flow runs with `prefect.server.models.flow_runs`

    main

    The prefect.server.models.flow_runs module provides functions for interacting with flow run ORM objects. Note: These functions are intended for internal use by the Prefect REST API.

    Key capabilities include:

    • Creating and updating flow runs.
    • Reading single or multiple flow runs with various filters (flow, task run, deployment, work pool, etc.).
    • Deleting flow runs (single or bulk).
    • Setting flow run states via orchestration logic.
    • Managing flow run labels and concurrency slots.

  8. Key capabilities of Per-Node dbt Orchestration

    main

    The Per-Node dbt Orchestration mode (using PrefectDbtOrchestrator) provides granular control over dbt model execution. Key features include:

    • Per-node retries: Failed models retry independently.
    • Cross-run caching: Cache persists across flow runs (uses a key without RUN_ID) with a 1-day default expiration.
    • Freshness-based expiration: Optionally override expiration using source freshness thresholds.
    • Result storage: Configure result_storage for remote cache persistence (e.g., S3, GCS).
    • Two-level concurrency: Prefect manages parallel nodes, while --threads controls parallelism within a single node.
    • Downstream skipping: If a node fails, all downstream dependents are automatically skipped.
    • State-based execution: Use --state, --defer, or --favor-state for efficient CI/CD workflows.
    • Execution modes: Choose between per-node (default, for retries/caching) or per-wave (for lower overhead).
    • Swappable executors: Supports dbt Core CLI (default) or dbt Cloud ephemeral jobs.
    • Observability & Lineage: Each node is a distinct Prefect task with timing/status, and dependencies are tracked in the asset graph via MaterializingTask.
    • Summary artifact: Generates a Markdown artifact containing success/error counts and failure details.

  11. Explore Prefect How-to Guides

    main

    Prefect provides several categories of how-to guides to help you build, deploy, and manage your data workflows. You can find specific instructions for the following areas:

    • Workflows: Writing and customizing Prefect workflows.
    • Deployments: Deploying and managing workflows as Prefect deployments.
    • Configuration: Configuring your Prefect environment.
    • Automations: Working with events, triggers, and automations.
    • Workflow Infrastructure: Deploying workflows to specific infrastructure types.
    • Prefect Cloud: Setting up and using a Prefect Cloud account.
    • Prefect Self-hosted: Hosting your own Prefect server.
    • AI: Integrating AI assistants with Prefect.
    • Migration: Migrating from other platforms or upgrading to the latest versions of Prefect.

  12. Use `prefect.utilities.engine` for engine-level utilities

    main
    The prefect.utilities.engine module provides low-level engine utilities used by Prefect to manage task execution, signal handling, and state transitions. While many of these functions are used internally by the Prefect orchestration engine, they are available in the prefect.utilities.engine namespace for specialized use cases involving task input collection, signal interception (like SIGTERM), and state proposal logic.