DOCSEARCHbyAlgolia
Submit Docs

Metaflow Documentation

website·Indexed 35 days ago

https://docs.metaflow.org/

A human-friendly library for scientists and engineers to build and manage real-life data science projects. Metaflow provides tools for scaling compute via AWS, Kubernetes, and GPUs, as well as features for distributed computing, task scheduling with Apache Airflow, Kubeflow, and AWS Step Functions, and a client API for inspecting flow results.

Tokens
84.9K
Snippets
393
Records
578
Agent score
71%

What's inside Metaflow

  1. Overview of Argo Workflows integration in Metaflow

    Argo Workflows is a Kubernetes-native orchestrator that allows Metaflow flows to be mapped automatically to Argo Workflow Templates. Key benefits include support for event-triggering via Argo Events and seamless integration with Kubernetes monitoring and alerting. Metaflow code runs on Argo Workflows exactly as it does locally; all tasks respect the @resources decorator as if @kubernetes were applied to all steps.

  2. Overview of Metaflow event triggering

    Event triggering allows Metaflow flows to start automatically based on external events rather than a fixed time-based schedule. This pattern enables flows to react to changes in real-time from external systems (e.g., data warehouses, ETL pipelines, microservices) or to chain multiple flows together where one flow triggers another upon completion. This creates a decoupled interface between workflows, allowing them to be developed and operated independently. Currently, event triggering is only supported on Argo Workflows.

  3. Overview of the Metaflow Core Integration Test Harness

    The Metaflow core integration test harness (located at test/core) is designed to surface bugs caused by unintended interactions across different layers of the Metaflow stack (Python interpreter, core, plugins, user-defined graphs, step functions, and user interfaces). Instead of simple unit tests, it automatically generates and executes synthetic Metaflow flows by combining four specifications: Contexts, Tests, Graphs, and Checkers.

  4. Overview of Metaflow AWS Integrations

    Metaflow allows developers to move between local execution and remote AWS execution without changing code or libraries. It integrates with specific AWS services to replace local resources for scaling and production deployment.
    ServiceLocal EquivalentAWS Service
    DatastoreLocal DirectoryAmazon S3
    ComputeLocal ProcessAWS Batch
    MetadataLocal DirectoryAWS Fargate + Amazon RDS
    NotebooksLocal NotebookAmazon Sagemaker Notebooks
    SchedulingN/AAWS Step Functions + Amazon EventBridge

  5. Overview of Metaflow production deployment orchestrators

    Metaflow allows you to deploy flows to production-grade orchestrators to ensure they run without human intervention, are highly available, and scale effectively. You can deploy flows with a single command without changing your code, allowing you to switch between schedulers easily.

    Supported orchestrators include:

    • Argo Workflows: Runs on Kubernetes. This is the recommended default as it is the only orchestrator that supports Metaflow's event-triggering functionality.
    • AWS Step Functions: A managed AWS service providing minimal operational overhead.
    • Apache Airflow: An open-source orchestrator; useful for teams that already have many DAGs deployed and want to maintain compatibility with existing pipelines.
    • Kubeflow: Used to stay compatible with existing Kubeflow deployments.

  6. Overview of Metaflow capabilities

    Metaflow is an open-source Python library (Apache License 2.0) that provides a unified API for the infrastructure stack required to move data science projects from prototype to production. It addresses several key layers of the ML lifecycle:

    • Modeling: Compatible with any Python library and ensures dependency availability across environments.
    • Deployment: Supports production-grade workflow orchestration.
    • Versioning: Automatically tracks flows, experiments, and artifacts.
    • Orchestration: Facilitates construction and local testing of workflows.
    • Compute: Integrates with cloud accounts and Kubernetes clusters for scalability.
    • Data: Manages data flow within workflows and provides patterns for accessing data lakes and warehouses.

  7. Overview of Metaflow dependency management

    Metaflow ensures stable and reproducible execution environments by managing dependencies across five layers:

    1. The flow file (e.g., flow.py) defining the DAG.
    2. Local Python modules and packages, which Metaflow packages automatically.
    3. The Metaflow framework itself, which is packaged for remote execution to ensure version consistency.
    4. External libraries managed via @pypi and @conda decorators.
    5. The operating system and low-level drivers, configurable via custom Docker images.

    Dependency management is primarily required when scaling to the cloud or deploying to production; local development uses the libraries installed on the user's machine.

  8. Overview of Metaflow Card Templates

    Card Templates are flexible, shareable Python packages used to create customized visual reports in Metaflow. They are more powerful than default cards or basic Card Components, allowing for the use of off-the-shelf JavaScript libraries, project-specific layouts, and public sharing via PyPi (typically using the metaflow-card- prefix). Card Templates are automatically packaged by Metaflow when executing tasks remotely (e.g., via AWS Batch) or deploying to production via Step Functions.

  9. Overview of Metaflow scaling patterns

    Metaflow provides several patterns for scaling compute, which can be mixed and matched within a single flow. These range from local execution for rapid development to horizontal and vertical scaling in the cloud. To use cloud capabilities like @batch and @kubernetes, a Metaflow stack must be deployed first.

  10. Overview of Composing Flows with Custom Decorators in Metaflow

    Introduced in Metaflow 2.16, custom decorators and mutators allow developers to encapsulate shared, project-specific logic (such as data access, ETL, model lineage, or LLM endpoint routing) outside of individual step code. This promotes separation of concerns, consistency across flows, and the ability to distribute common patterns as installable packages. These tools are intended for sharing components across flows; for deep infrastructure integrations (like new production orchestrators), the Metaflow Extension mechanism should be used instead.

  11. Overview of Metaflow Mutators

    Mutators allow developers to programmatically control decorators and parameters applied to a flow. Unlike standard decorators, mutators are applied at deploy time (before a run or deployment begins) and cannot modify the flow during execution. They are useful for applying stacks of decorators automatically, creating template flows based on configurations, or implementing the BaseFlow pattern for shared project tooling.

  12. Overview of Metaflow Cards for result visualization

    Cards are a built-in Metaflow mechanism used to create automatic, human-readable reports for flow executions. They allow developers to observe model performance, data quality, or task progress without needing to manually run Jupyter notebooks for every execution.

    Key characteristics include:

    • Content: Can contain images, text, and tables.
    • Live Updates: As of Metaflow 2.11, cards can update in real-time while tasks are executing to monitor intermediate results.
    • Safety: Cards are stored independently of the flow. If card creation fails, the main workflow execution is not affected, making them safe for production.
    • Scope: Ideal for experiment tracking, sharing results with non-technical stakeholders, and debugging. They are not intended for complex interactive dashboards.