DOCSEARCHbyAlgolia
Submit Docs

Metaflow Documentation

repository·master·Indexed 27 days ago

https://github.com/netflix/metaflow

A human-centric framework for building, managing, and scaling AI and ML systems from local prototyping to production deployment. Documentation covers the Devstack local Kubernetes environment using Minikube and Tilt, development shell configuration, UX testing across backends (Argo Workflows, SFN + Batch, Airflow), @cards UI development, and a series of tutorials including Hello World, Movie Playlist, Statistics, and Conda-based dependency management.

Tokens
16.9K
Snippets
48
Records
119
Agent score
94%

What's inside Metaflow

  1. Overview of Metaflow capabilities

    master

    Metaflow is a human-centric framework for building and managing AI and ML systems. It provides a Pythonic API designed to bridge the gap between prototyping and production through three main stages:

    1. Rapid Local Prototyping: Includes support for notebooks, experiment tracking, versioning, and visualization.
    2. Cloud Scaling: Enables horizontal and vertical scaling in the cloud using CPUs and GPUs, supporting massive embarrassingly parallel workloads (via foreach) and gang-scheduled distributed computing.
    3. Production Deployment: Facilitates dependency management and one-click deployment to highly available production orchestrators with support for reactive orchestration (event triggering).

  2. Overview of Metaflow Cards

    master

    Metaflow Cards allow you to automatically generate human-readable report cards from Metaflow tasks. They are used to observe run results, visualize models, and share outcomes with stakeholders.

    Key features include:

    • Default Cards: Built-in functionality that displays all task outputs without code changes.
    • Custom Cards: Created using Python to highlight specific data or structure reports.
    • Custom Templates: Advanced reports that generate arbitrary HTML.
    • Portability: Cards can be shared as standard Python packages and accessed via the Metaflow CLI in offline or security-conscious environments.
    • GUI Integration: Cards can enrich the Metaflow GUI task view with application-specific information.

  3. Understand the Environment Escape design

    master
    Metaflow typically uses Conda to provide a pinned, reproducible environment for flows, ensuring dependencies do not change between runs. However, the Environment escape plugin allows for a hybrid model where most code executes in a pinned environment (like Conda), but specific parts of the code can 'escape' to execute in a different Python environment. This is useful for using packages not available in Conda or for using packages that require dynamic updates (e.g., data access clients).

  4. Understand the Metaflow Datastore Architecture

    master

    The Metaflow datastore is responsible for storing and retrieving artifacts (data produced/consumed in steps), logs, metadata, and code packages. It is organized hierarchically:

    1. DataStoreStorage: The lowest level. It abstracts the physical storage (e.g., S3 or local filesystem) and provides methods to read/write bytes and manage file metadata.
    2. ContentAddressedStore: A layer on top of DataStoreStorage that provides de-duplication (identical objects are stored once) and data transformation (e.g., compression). It is shared per flow.
    3. TaskDataStore: The primary interface for executing tasks. It handles artifact persistence (via ContentAddressedStore), logs, and task-specific metadata (via DataStoreStorage).
    4. FlowDataStore: The top-level orchestrator. It manages the ContentAddressedStore and provides access to individual TaskDataStore instances for specific tasks.

  6. Schedule Metaflow flows on AWS Step Functions

    master

    You can schedule a Metaflow workflow to run on AWS Step Functions without modifying your source code by using the step-functions command line argument.

    To deploy a flow to AWS Step Functions, use the step-functions create argument. You can specify the number of workers using the --max-workers flag.

    To execute the scheduled flow, use the step-functions trigger argument.

  7. Install the Metaflow R package

    master

    You can install Metaflow from GitHub using devtools. After installing the package, run metaflow::install_metaflow() to complete the setup.

    devtools::install_github("Netflix/metaflow", subdir="R")
metaflow::install_metaflow()

  8. Define an emulated module for Environment Escape

    master

    To emulate a library that is unavailable in a specific environment (e.g., a Conda environment) but available in a base environment, create a configuration directory following this structure:

    1. Create a subdirectory in plugins/env_escape/configurations/ named emulate_<name>, where <name> is the library name (or a list of names separated by __).
    2. Inside that directory, implement __init__.py and the following two files:

    server_mappings.py

    This file defines what the plugin can proxy between the client and server. It must contain:

    • EXPORTED_CLASSES: A dictionary mapping module prefixes/names and class suffixes to the actual server-side classes.
    • EXPORTED_FUNCTIONS: A dictionary mapping module prefixes/names and function names to the actual server-side functions.
    • EXPORTED_VALUES: A dictionary mapping module prefixes/names and attribute names to the actual server-side values.
    • PROXIED_CLASSES: A tuple of other objects the server is allowed to return.
    • EXPORTED_EXCEPTIONS: A dictionary mapping module prefixes/names and exception names to the actual server-side exception classes. You must specify all exceptions up to a basic Exception type to maintain the hierarchy.

    overrides.py

    This file allows you to intercept method calls or attribute access. It uses decorators from override_decorators.py:

    • local_override / remote_override: Intercept method calls.
    • local_getattr_override / local_setattr_override: Intercept attribute access.
    • local_exception / remote_exception_serialize: Customize how exceptions are recreated on the client side.

  9. Run the Statistics tutorial

    master

    To execute the Statistics tutorial, navigate to the metaflow-tutorials directory and use the stats.py script to either show or run the flow. You can then open the associated Jupyter Notebook to visualize the results.

    1. Show the flow: python 02-statistics/stats.py show
    2. Run the flow: python 02-statistics/stats.py run
    3. Open the notebook: jupyter-notebook 02-statistics/stats.ipynb
    cd metaflow-tutorials
python 02-statistics/stats.py show
python 02-statistics/stats.py run
jupyter-notebook 02-statistics/stats.ipynb

  10. Implement a custom storage backend using `DataStoreStorage`

    master

    To integrate a new storage system (like GCS) into Metaflow, you must implement the DataStoreStorage class. This class provides a file-and-directory-like abstraction for byte-level operations.

    Key requirements:

    • Implement save_bytes and load_bytes for byte-level storage.
    • Implement path manipulation routines like path_join, path_split, basename, and dirname.
    • Support is_file checks.
    • load_bytes returns a CloseAfterUse object, which must be used within a with statement to ensure data is accessible before the scope terminates.

  11. Use Sidecars for opportunistic background tasks

    master

    Sidecars are child processes used to run internal tasks in parallel with scheduling or user code execution. They are designed for best-effort, opportunistic tasks that are not critical to the success of the main task (e.g., sending heartbeats to a metadata service).

    Key Characteristics:

    • Lifetime: Bound to the parent process.
    • Communication: One-way, lossy, and non-blocking (similar to UDP). The parent does not wait for acknowledgments.
    • Observability: Set METAFLOW_DEBUG_SIDECAR=1 to see the commands used to launch sidecars. You can interact with them by sending messages via stdin during testing.

    Constraint: A sidecar cannot perform any operation that must succeed for the task or run to be considered valid.