DOCSEARCHbyAlgolia
Submit Docs

Flytekit Python SDK

repository·master·Indexed 43 days ago

https://github.com/flyteorg/flytekit

The primary SDK for authoring, testing, deploying, and interacting with Flyte tasks, workflows, and launch plans. It serves as the core extensible library for building data processing pipelines on the Flyte platform, featuring a variety of plugins for AWS Athena, AWS Batch, AWS EMR Serverless, AWS SageMaker, Airflow, and Xarray Zarr persistence.

Tokens
57.1K
Snippets
183
Records
246
Agent score
53%

What's inside Flytekit

  1. Overview of Flytekit logical components

    master

    Flytekit is organized into several logical components that facilitate different stages of the Flyte lifecycle:

    • Models Files: Protobuf-generated files that define the data structures used across the system.
    • Authoring: The core experience for developers to define and write Flyte code, including tasks, workflows, and launch plans.
    • Control Plane: Python objects that allow users to interact with the Flyte control plane (e.g., via FlyteRemote).
    • Execution: A shim layer that manages interactions with the Flyte ecosystem during the actual execution of tasks and workflows.
    • CLIs and Clients: Command line tools for user interaction and the underlying control plane client used by those tools.

  2. Overview of the Flytekit Slurm Plugin

    master
    The Flytekit Slurm Plugin provides a Slurm connector that integrates Flyte workflows with Slurm-managed high-performance computing (HPC) clusters. It allows Flyte users to utilize Slurm's native capabilities for compute resource allocation, scheduling, and monitoring within their workflows.

  3. Explore Flytekit Plugins API reference

    master

    Flytekit provides a wide range of plugins to integrate with various data platforms, compute engines, and machine learning tools. You can find detailed API references for each of the following categories:

    Cloud & Data Warehouses

    • AWS Athena: AWS Athena integration
    • AWS Batch: AWS Batch integration
    • Google Bigquery: Google Bigquery integration
    • Hive: Hive integration
    • Snowflake: Snowflake integration
    • DuckDB: DuckDB integration
    • DBT: DBT integration
    • SQLAlchemy: SQLAlchemy integration

    Compute & Distributed Engines

    • Dask: Dask standard API
    • Ray: Ray integration
    • Spark: Spark integration
    • Modin: Modin integration
    • Vaex: Vaex integration
    • Pod: Pod-based execution

    Machine Learning & Model Serving

    • Inference: General Inference API
    • SageMaker Inference: AWS SageMaker Inference integration
    • OpenAI: OpenAI integration
    • ONNX: Support for ONNX with PyTorch, TensorFlow, and ScikitLearn
    • KF (Kubeflow) specialized engines: KF MPI, KF Pytorch, and KF Tensorflow
    • MLflow: MLflow integration
    • Weights & Biases: Weights & Biases integration
    • Neptune: Neptune integration

    Data Validation & Notebooks

    • Great expectations: Data quality validation
    • Pandera: Data validation
    • Papermill: Jupyter Notebook execution
    • Deck / Dolt: Standard API references for Deck and Dolt

    For specific implementation details, usage patterns, and configuration for any of these plugins, refer to their individual documentation pages.

  4. Use the AWS Batch Plugin for Flytekit

    master
    The flytekitplugins.awsbatch module provides integration capabilities for running Flyte tasks on AWS Batch. This plugin allows you to leverage AWS Batch as a compute backend for your Flyte workflows, enabling scalable execution of containerized tasks on AWS managed infrastructure.

  5. Use the Flytekit Perian Job Platform Plugin

    master

    The Flytekit Perian Job Platform Plugin is a Flyte Connector plugin designed to execute Flyte tasks on the Perian Job Platform (perian.io). This allows you to leverage Perian's infrastructure for running your Flyte workflows and tasks.

    For detailed setup instructions and getting started guides, refer to the official Perian documentation.

  7. How Flytekit models interact with IDL (Protobuf)

    master

    Flytekit models act as a Pythonic wrapper around the underlying Protobuf-generated classes. To bridge the gap between the developer-friendly Python models and the wire-format Protobuf objects, every model class implements two key conversion methods:

    • to_flyte_idl(): Converts the Python model instance into the Protobuf-generated Python class.
    • from_flyte_idl(): Creates a Python model instance from a Protobuf-generated Python class.

    If you are extending or working with these models, ensure that conversion logic is maintained to allow seamless interaction with the Flyte engine.

  8. Handle recoverable and non-recoverable exceptions

    master

    Flytekit distinguishes between system-level exceptions and user-level exceptions. For user-level errors, only flytekit.exceptions.user.FlyteRecoverableException is treated as recoverable (meaning it will be retried according to the task's retry policy). All other exceptions, including standard Python exceptions like ValueError, are considered non-recoverable and will cause the task to fail immediately.

    User Exception Hierarchy:

    • Exception
      • FlyteValidationException
      • FlyteEntityAlreadyExistsException
      • FlyteValueException
      • FlyteTimeout
      • FlyteAuthenticationException
      • FlyteRecoverableException (The only one that triggers retries)

  9. Configure OmegaConf Transformer modes

    master

    The plugin uses a transformer to handle how DictConfig objects are deserialized. You can choose between three modes via OmegaConfTransformerMode:

    • Dataclass: Reconstructs the config from a matching dataclass during deserialization. This preserves typing information and validation. Requirement: The dataclass definition must be available via Python import in the Flyte execution environment.
    • DictConfig: Deserializes the config into a DictConfig object. Dataclasses are translated into DictConfig objects, and only primitive types are checked. The dataclass definition is only required during initial serialization.
    • Auto: The default mode. It attempts Dataclass mode first and falls back to DictConfig mode if the dataclass definition is unavailable.

    Important Constraint: Because DictConfig is flattened and keys are transformed into dot notation, the keys of the DictConfig must not contain dots.

  10. Understand the difference between Flytectl and Pyflyte

    master

    Flyte uses two primary Command Line Interfaces (CLIs) that serve different purposes in the workflow lifecycle:

    • Flytectl: A network-aware CLI used to communicate with the Flyte control plane (FlyteAdmin). It is similar to kubectl for Kubernetes. It is responsible for shipping compiled Protobuf files to FlyteAdmin but does not understand your Python code.
    • Pyflyte: A code-aware CLI responsible for the serialization (compilation) step. It parses your Python code to find tasks, workflows, and launch plans, and compiles them into Protobuf files.

  11. Types of Flytekit Plugins

    master

    Flytekit supports various plugin types to extend functionality:

    • Python-side functionality: Adds logic purely on the Python side (e.g., flytekit-greatexpectations).
    • Type Plugins / TypeTransformers: Adds new types to Flyte and defines how to marshal them between Flytekit and the backend or other languages (e.g., flytekit-pandera).
    • TaskTemplate Plugins: Supplies prebuilt containers for runtime execution (e.g., flytekit-sqlalchemy).
    • Backend Plugins: Handles query invocation or backend-specific logic (e.g., flytekit-snowflake).
    • Meta Plugins: Wraps other tasks to add functionality (e.g., flytekit-papermill).
    • Execution Command Plugins: Modifies the execution command (e.g., flytekit-spark).
    • Context Modification Plugins: Executes user containers with specific context modifications (e.g., flytekit-kf-tensorflow).
    • Persistence Plugins: Allows data to be stored in different persistence layers (e.g., flytekit-data-fsspec).

  12. How Flytekit bridges Python and Flyte type systems

    master
    Flyte uses its own type system defined in the Flyte IDL. To bridge the gap between Python's type system (PEP 484) and Flyte's IDL, Flytekit uses the flytekit.extend.TypeEngine. The engine operates by invoking a series of TypeTransformer objects, where each transformer handles the conversion logic for a specific native Python type.