DOCSEARCHbyAlgolia
Submit Docs

pytask

repository·main·Indexed 40 days ago

https://github.com/pytask-dev/pytask

A workflow management system designed to facilitate reproducible data analyses by managing task dependencies and execution. It features automatic task discovery, a CLI for building and profiling pipelines, and a persistence mechanism to skip unchanged tasks. Key capabilities include DAG visualization, task locking for state management, and extensive debugging tools such as --pdb, --trace, and --show-locals.

Tokens
43.9K
Snippets
167
Records
228
Agent score
50%

What's inside pytask

  1. Overview of pytask features

    main

    pytask is a tool designed to facilitate reproducible data analyses. Its core capabilities include:

    • Automatic discovery of tasks: Automatically finds tasks within your project.
    • Lazy execution: Optimizes workflows by skipping tasks if their inputs and outputs remain unchanged.
    • Debug mode: Allows you to jump into a debugger if a task fails.
    • Task repetition: Enables looping over task functions with different inputs.
    • Task selection: Allows running specific subsets of tasks using expressions and markers.
    • Extensibility via Plugins: Supports various extensions including:
      • pytask-parallel for parallelization
      • pytask-latex for LaTeX support
      • pytask-r for R integration
      • pytask-stata for Stata integration

  2. Compare pytask to other workflow management systems

    main

    When choosing a workflow management system (WFM), consider the following alternatives to pytask based on your specific use case:

    Scientific Computing & Bioinformatics

    • Snakemake: Widely adopted; scales from local to cloud/clusters; supports containers and Conda; uses a Python/Make-style DSL; can export to CWL.
    • Nextflow: Popular in bioinformatics; runs on AWS, GCP, and Azure; uses Groovy for scripting; strong container support (Docker, Singularity, Podman).
    • SciPipe: Written in Go; uses Flow-Based Programming; compiles workflows to fast binaries; designed for command-line tool pipelines.
    • sciluigi: A lightweight, simplified wrapper around Luigi for scientific research (development stalled).

    Machine Learning & MLOps

    • Ploomber: Focuses on ML pipelines; integrates with MLflow, Docker, and AWS Batch; supports tasks in YAML, Python, Jupyter notebooks, or SQL.
    • Kedro: Mature framework for production ML; provides project templates, data catalogs, and deployment tooling; emphasizes software engineering best practices.
    • ZenML: MLOps framework for portable ML pipelines; supports orchestrators like Kubernetes, AWS SageMaker, GCP Vertex AI, Kubeflow, and Airflow; features automatic containerization and artifact tracking.
    • Flyte: Kubernetes-native orchestration; provides automatic retries, checkpointing, and failure recovery; scales across AWS, GCP, and Azure.
    • Hamilton: Focusly on modular transformations, lineage, and observability for data and ML pipelines using function graphs.

    General Purpose Task Runners & Build Systems

    • pydoit: A Python replacement for Make; focuses on executing command-line tools.
    • Waf: Mature build system for compiling software; handles complex dependencies and is extensible with Python.
    • SCons: Cross-platform software construction tool; uses Python scripts for configuration; supports C, C++, Java, Fortran, and automatic dependency analysis.
    • pypyr: Task-runner for automation pipelines defined in YAML; includes built-in steps for loops, conditionals, and retries; common in CI/CD.
    • Luigi: Built for long-running batch processes; integrates with Hadoop, Spark, and databases (development has slowed).

    Python-Native Dataflow Libraries

    • pipefunc: Lightweight library for creating function pipelines as Directed Acyclic Graphs (DAGs) in pure Python; supports map-reduce and parallel execution.

  3. What is the `DataCatalog` and when to use it

    main

    The pytask.DataCatalog is an optional feature used to manage task dependencies and products without manually defining and repeating file paths. It is particularly useful in large projects to avoid path redundancy and to hide intermediate representations (files that are not of primary interest) by storing them automatically in the .pytask folder.

    Key benefits:

    • Path Abstraction: Instead of BLD / "data.pkl", you use data_catalog["data"].
    • Automatic Node Creation: If an entry does not exist, the catalog automatically creates a pytask.PickleNode, which allows you to save any Python object to a pickle file.
    • Simplified Dependencies: Consuming tasks can access the loaded data directly via the catalog without manual loading logic.

  4. How pytask extension works

    main

    pytask is built on pluggy, a plugin system for Python. Extension is achieved by implementing hook functions (hook implementations) that respond to entry-points (hook specifications) called throughout the pytask execution lifecycle. When pytask calls a hook, it iterates through all registered implementations, allowing each one to potentially alter the result of that entry-point.

    There are two primary ways to add implementations:

    1. Temporary/Local registration: Using the --hook-module CLI option or the hook_module configuration value.
    2. Permanent/Distributed registration: Packaging your plugin as a Python package using setuptools entry-points.

  5. Generate execution reports with CollectionReport, ExecutionReport, and DagReport

    main

    Reports in pytask provide structured summaries of task executions. Depending on your needs, you can use different report types:

    • CollectionReport: Summarizes the outcomes of a specific collection of tasks.
    • ExecutionReport: Provides a detailed report of a full execution run.
    • DagReport: Provides a report specifically structured around the Directed Acyclic Graph (DAG) of task dependencies.

  6. Define task products using `pytask.Product` or `produces`

    main

    A product is a file or resource that a task is expected to create. pytask verifies the existence of these products after a task completes. You can define products using two different interfaces:

    1. Annotated Interface (Recommended for type safety): Use the pytask.Product type annotation on function arguments. This explicitly marks the argument as a product.
    2. produces Interface: Use a magic argument named produces. Any value passed to this argument is automatically treated as a task product.

    Note: This tutorial focuses on local files. For remote files (S3, GCP, etc.), refer to the remote files guide.

    # Using pytask.Product (Annotated)
from pytask import task, Product
from pathlib import Path

@task
def task_data_preparation(output_path: Product) -> None:
    output_path.write_text("data")

# Using 'produces' magic argument
@task
def task_data_preparation(produces: Path) -> None:
    produces.write_text("data")

  7. Distinguish between public and private functions

    main

    When interacting with the pytask API, follow these naming conventions:

    • Public functions: Names that do not start with an underscore (_). These are intended to be imported and used in other modules.
    • Private functions: Names that start with an underscore (_). These are intended for internal use only within the module where they are defined and should not be relied upon by consumers.

  8. Handle Tracebacks and Warnings in pytask

    main

    When tasks fail or encounter non-fatal issues, pytask provides mechanisms to capture and process this information:

    Tracebacks

    Use pytask.Traceback to handle and represent error tracebacks encountered during task execution.

    Warnings

    pytask allows for capturing and filtering warnings during execution:

    • WarningReport: A structured report containing warnings encountered during a run.
    • pytask.parse_warning_filter: Used to parse configuration strings that define which warnings should be captured or ignored.
    • pytask.warning_record_to_str: Converts a warning record into a human-readable string format.

  9. Use pytask Protocols to implement custom Nodes and Tasks

    main

    If you are extending pytask by creating custom node or task types, you should implement the following protocols to ensure compatibility with the engine:

    • pytask.PNode: The base protocol for all node types.
    • pytask.PPathNode: For nodes that are associated with a filesystem path.
    • pytask.PTask: The protocol for task objects.
    • pytask.PTaskWithPath: For tasks that are explicitly tied to a specific filesystem path.
    • pytask.PProvisionalNode: For nodes that are temporary or intermediate during execution.

  10. Understand the pytask lockfile

    main

    The pytask.lock file is the default state backend for pytask. It stores task state in a portable, git-friendly TOML format, allowing task runs to be resumed or shared across different machines.

    Key behaviors during a run:

    1. pytask reads pytask.lock.
    2. It compares the current state() of dependencies, products, and tasks against the stored state.
    3. Tasks with matching states are skipped; others are executed.
    4. pytask.lock is updated atomically after each completed task.
    5. pytask.lock is updated after skipping unchanged tasks (unless --dry-run or --explain flags are used).

    Note on Legacy Formats: SQLite is the legacy format. If no lockfile exists, pytask reads the legacy database state and writes a new pytask.lock. pytask build maintains the legacy database for downgrade compatibility.

    # Example pytask.lock content
lock-version = "1"

[[task]]
id = "src/tasks/data.py::task_clean_data"
state = "f9e8d7c6..."

[task.depends_on]
"data/raw/input.csv" = "e5f6g7h8..."

[task.produces]
"data/processed/clean.parquet" = "m3n4o5p6..."

  11. How pytask detects changes in remote files

    main

    pytask uses the Entity Tag (ETag) header to detect if a remote file has changed. If a remote server (like AWS S3) provides an ETag (often an MD5 digest), pytask will automatically detect changes when the ETag updates.

    Limitations: Many web servers do not provide an ETag. If the remote file lacks an ETag, pytask cannot detect changes. In such cases, to force a task to rerun, you must manually delete the task's product (the output file/result).

  12. How the pytask plugin architecture works with pluggy

    main

    pytask uses pluggy to implement an extensible plugin system based on the concept of hooking. This allows multiple plugins to coexist and modify or extend the behavior of the host (pytask) without the need for method overriding or monkey patching.

    Core Concepts

    • Host: The main application (pytask) that defines where extensions can occur.
    • Hook Specification: Defined by the host at specific execution points. These act as entry-points that signal to plugins that an event is occurring.
    • Hook Implementation: Code provided by a plugin that targets a specific hook specification. It receives a message from the host and can optionally return a response.
    • Hooking: The mechanism where the host sends messages to all registered plugin implementations for a given specification and then processes the returned responses (either all of them or just the first valid one, depending on the host's design).

    This architecture ensures that plugins can be highly specific in their targets and can work alongside many other plugins simultaneously.