DOCSEARCHbyAlgolia
Submit Docs

Kubeflow Pipelines Documentation

repository·master·Indexed 26 days ago

https://github.com/kubeflow/pipelines

An end-to-end machine learning (ML) workflow orchestration service designed for Kubernetes. This documentation covers the use of the dedicated SDK, the kfp-server-api Python HTTP client for interacting with the API server, and instructions for generating Golang and Python protocol buffer code for the Pipeline Spec.

Tokens
127.7K
Snippets
250
Records
606
Agent score
86%

What's inside Kubeflow Pipelines

  2. Overview of MLflow Integration for Kubeflow Pipelines

    master
    The MLflow integration for Kubeflow Pipelines (KFP) provides automatic experiment tracking by linking KFP pipeline runs with MLflow runs. This integration eliminates the need for manual MLflow client configuration within individual pipeline components. When configured, the KFP API server, driver, and launcher handle MLflow interactions transparently, allowing for automatic logging of scalar metrics and input/output parameters without modifying existing pipeline code.

  3. Overview of Kubeflow Pipelines (KFP)

    master

    Kubeflow Pipelines (KFP) is a platform for building and deploying portable, scalable machine learning (ML) workflows using containers on Kubernetes-based systems.

    Users can author workflows using the KFP Python SDK, compile them into an intermediate representation (IR) YAML, and submit them to various backends including:

    • The open source KFP backend (part of Kubeflow or standalone)
    • Google Cloud Vertex AI Pipelines
    • KFP local

  4. Use Pipeline Run Workspaces for efficient data sharing

    master

    Kubeflow Pipelines (KFP) introduces a Pipeline Run Workspace abstraction to provide shared, ephemeral storage between components during a single pipeline run. This avoids the overhead of uploading/downloading large intermediate datasets to S3/object storage and simplifies the use of Persistent Volume Claims (PVCs).

    Key Features

    • Shared Storage: Components can share large datasets via a workspace without cluttering MLMD or requiring manual S3 uploads.
    • Simplified Configuration: Users can typically specify only a workspace size, while administrators can set environment-specific defaults (e.g., storageClassName, accessModes).
    • Automated Lifecycle: Workspaces are automatically cleaned up when a pipeline run succeeds or is deleted.
    • On-demand Mounting: The workspace is only mounted to components that explicitly use it, preventing unnecessary node affinity constraints in ReadWriteOnce modes.
    • Artifact Importer Support: Supports downloading artifacts directly into the workspace using dsl.importer.

    Important Constraints

    • Ephemeral Nature: Workspaces are intended for intermediate data sharing. Final output artifacts must still be uploaded to object storage to be treated as formal KFP artifacts.
    • Single Workspace: The initial implementation supports only one workspace per pipeline run.

  5. Identify beginner-friendly issues via /good-first-issue label

    master

    The Kubeflow Pipelines repository uses the /good-first-issue label to identify issues that are suitable for first-time contributors. This label is applied based on an analysis of the issue's scope, context, guidance, and complexity.

    New contributors should look for this label when searching for tasks to work on to ensure their subsequent Pull Requests can pass the CI gatekeeper workflow.

  6. Understand the ML-Metadata (MLMD) Removal Proposal

    master

    Kubeflow Pipelines (KFP) v2 is undergoing a transition to remove ML-Metadata (MLMD) as a standalone dependency. The functionality previously handled by MLMD (tracking pipeline executions, storing artifacts/metadata, managing parameters/metrics, and lineage tracking/caching) is being replaced by native KFP API server capabilities.

    Key benefits of this change include:

    • Reduced Operational Complexity: Fewer services to deploy and maintain.
    • Improved Technical Alignment: Metadata storage will use KFP's native database schema, enabling better multi-tenancy and support for newer database versions (MySQL/PostgreSQL).
    • Simplified Architecture: Fewer integration points for easier debugging and troubleshooting.

  7. Understand the Kubernetes Native API for Pipelines

    master

    The Kubernetes Native API (KEP-11551) introduces Pipeline and PipelineVersion as Custom Resource Definitions (CRDs) within Kubernetes. This allows users to manage pipelines using standard Kubernetes tools and GitOps workflows.

    Key Considerations for Users:

    • Performance: Queries via the REST API may be slower because they must query the Kubernetes API instead of a direct SQL database.
    • Filtering/Sorting: Server-side sorting and field selectors are limited by Kubernetes CRD capabilities (until v1.31+). The REST API performs client-side filtering, which may impact performance in very large environments.
    • Backup/Restore: When performing backups, you must include both the database and the Kubernetes objects stored in etcd to ensure a complete state.

  8. Understand the Kubeflow Pipelines Driver and Executor model

    master

    In Kubeflow Pipelines (KFP), each node in a pipeline graph is non-atomic and consists of two distinct components running in separate Kubernetes pods:

    1. Driver: A lightweight component that performs API calls, checks for cached results, and creates MLMD (Machine Learning Metadata) executions. It generates a pod-spec-patch for the executor.
    2. Executor: The component that handles the heavy workload, running the actual user-defined container/code.

    Additionally, every pipeline run spawns a root DAG driver pod to initialize the entire pipeline execution.

  10. Orchestrate GenAI workflows with Kubeflow Pipelines

    master
    Kubeflow Pipelines serves as an orchestration engine for end-to-end Generative AI (GenAI) workflows. It provides a modular and reproducible way to manage the lifecycle of GenAI models across several stages: data preparation, fine-tuning, prompt engineering, evaluation, and deployment.

  11. Install and set up the UI Smoke Test Tool

    master

    The UI Smoke Test Tool performs visual regression testing for the Kubeflow Pipelines frontend by capturing screenshots and comparing them against a base reference using a Kind backend.

    Prerequisites

    Ensure the following are installed:

    • Node.js >= 18
    • git >= 2.5
    • Docker (required for --compare)
    • kind (required for --compare)
    • kubectl (required for --compare)
    • gh CLI (required for --pr)

    First-Time Setup

    Run these commands to install dependencies and the Chromium browser used for screenshot capture:

    # 1. Navigate to the tool directory
cd frontend/scripts/ui-smoke-test

# 2. Install tool dependencies
npm install

# 3. Install Chromium for screenshot capture
npx playwright install chromium

# 4. (For --compare) Ensure Docker is running
open -a Docker  # macOS

# 5. Run your first comparison
node smoke-test-runner.js --compare master
    # 1. Navigate to the tool directory
cd frontend/scripts/ui-smoke-test

# 2. Install tool dependencies
npm install

# 3. Install Chromium for screenshot capture
npx playwright install chromium

# 4. (For --compare) Ensure Docker is running
open -a Docker  # macOS

# 5. Run your first comparison
node smoke-test-runner.js --compare master

  12. Migrate MLMD metrics to the metrics table

    master

    If you are migrating from a version where metrics were stored as artifacts, use a migration script to move them to the dedicated metrics table.

    Migration Logic:

    1. Identify system.Metrics artifacts in the MLMD database.
    2. Move these entries to the metrics table.
    3. Convert custom properties from the artifacts into JSON format within the metrics table.
    4. Ensure that non-metric artifacts (e.g., Model or Dataset) remain untouched in the artifacts table.