DOCSEARCHbyAlgolia
Submit Docs

Kubeflow Documentation

repository·master·Indexed 41 days ago

https://github.com/kubeflow/website

Documentation for Kubeflow, including community guidelines, contribution processes, and the Kubeflow MCP Server for managing machine learning training jobs and cluster inspection via AI agents.

Tokens
153.1K
Snippets
326
Records
642
Agent score
52%

What's inside Kubeflow

  1. What is Kubeflow Kale?

    master
    Kubeflow Kale (Kubeflow Automated pipeLines Engine) is a tool that converts annotated Jupyter notebooks into production-ready Kubeflow Pipelines (KFP v2). Instead of writing KFP SDK code manually, you tag cells in your notebook to define steps and data dependencies. Kale then compiles these annotations into a KFP v2 pipeline DSL that can be executed on any compliant Kubeflow cluster.

  2. What is Kubeflow Trainer

    master

    Kubeflow Trainer is a Kubernetes-native distributed AI platform designed for scalable large language model (LLM) fine-tuning and training. It supports a wide range of frameworks including PyTorch, MLX, HuggingFace, DeepSpeed, JAX, and XGBoost.

    Key technical capabilities include:

    • MPI on Kubernetes: Orchestrates multi-node, multi-GPU distributed jobs for high-throughput communication and ultra-fast synchronization between GPU nodes.
    • Distributed Data Caching: Uses Apache Arrow and Apache DataFusion to stream large-scale data with zero-copy transfer directly to GPU nodes, minimizing I/O overhead and maximizing GPU utilization.
    • Cloud Native Integration: Integrates with Kueue (topology-aware scheduling), JobSet, and LeaderWorkerSet for workload orchestration.
    • Python SDK Support: AI practitioners can use the Kubeflow Python SDK to interact with TrainJob and Runtimes APIs.

  3. Overview of Kubeflow Notebooks

    master

    Kubeflow Notebooks provides interactive development environments for AI, ML, and Data workloads directly on Kubernetes. Instead of running environments locally, users can create notebook containers within the cluster.

    Key capabilities include:

    • Supported IDEs: Native support for JupyterLab, RStudio, and Visual Studio Code (code-server).
    • Custom Environments: Admins can provide standardized notebook images with pre-installed packages for organizational use.
    • Access Control: Integration with Kubeflow's RBAC (Role-Based Access Control) for managing notebook sharing and permissions.

  4. Overview of the Kubeflow SDK

    master

    The Kubeflow SDK provides a unified set of Pythonic APIs designed to run AI workloads at any scale without requiring direct Kubernetes knowledge. It abstracts infrastructure complexity, allowing users to interact with various Kubeflow subprojects through a consistent interface.

    Key capabilities include:

    • Unified Experience: A single SDK for multiple Kubeflow subprojects.
    • Scalability: Ability to scale workloads from a local laptop to large-scale production clusters with thousands of GPUs using the same APIs.
    • Local Development: Supports local development via pip installation without needing a Kubernetes cluster.

  5. What is KServe?

    master

    KServe is an open-source project that enables serverless inferencing on Kubernetes. It provides high-abstraction interfaces for common machine learning (ML) frameworks such as TensorFlow, XGBoost, scikit-learn, PyTorch, and ONNX.

    Key functionalities include:

    • Kubernetes CRDs: Provides Custom Resource Definitions for serving ML models on arbitrary frameworks.
    • Automated Infrastructure: Encapsulates complexity for autoscaling (including GPU autoscaling), networking, health checking, and server configuration. It supports features like 'scale to zero' and canary rollouts.
    • Inference Pipeline: Provides out-of-the-box support for prediction, pre-processing, post-processing, and explainability through a pluggable architecture.

  6. What is Kubeflow

    master
    Kubeflow is a cloud-native AI platform composed of modular, open-source projects that form a Kubernetes-native stack for data and AI workloads. It is designed to support AI practitioners, platform administrators, and decision-makers by providing scalable and extensible tools for data processing, model training, and AI/ML/HPC use-cases.

  8. Overview of Kubeflow Pipelines (KFP)

    master

    Kubeflow Pipelines (KFP) is a platform for building and deploying portable and scalable machine learning (ML) workflows using containers on Kubernetes-based systems.

    Key capabilities include:

    • Python-native authoring: Create end-to-end ML workflows using the KFP Python SDK.
    • Component-based architecture: Compose workflows using custom or existing ML components.
    • Artifact & Parameter management: Pass parameters and ML artifacts between components easily.
    • Execution efficiency: Utilize parallel task execution and caching to eliminate redundant computations.
    • Portability: Compile pipelines into an intermediate representation (IR) YAML to maintain cross-platform compatibility.
    • Observability: Manage, track, and visualize pipeline definitions, runs, experiments, and artifacts.

  9. What is OGX?

    master

    OGX (formerly Llama Stack) is an open-source API layer designed for building agentic AI applications on user-controlled infrastructure. It provides an abstraction layer that supports multiple model and infrastructure backends while exposing standardized APIs.

    Key capabilities include:

    • OpenAI-compatible APIs: Standard endpoints for chat completions, completions, and embeddings.
    • Agentic Orchestration: A Responses API that handles tool calling, MCP server integration, and built-in RAG (file search) in a single call.
    • Managed Storage: Endpoints for vector stores and file management.
    • Extensibility: A Skills API for managing versioned skill bundles.

  10. Overview of Kubeflow Pipelines V1 SDK packages

    master

    The Kubeflow Pipelines V1 SDK is a set of Python packages used to specify and run machine learning (ML) workflows. A pipeline is a description of an ML workflow consisting of components (the individual steps) and their interactions.

    Warning: This documentation refers to Kubeflow Pipelines V1. For the latest information, use the V2 documentation. While V2 backends can run V1 pipelines, migration to the V2 SDK is strongly recommended.

    Core packages include:

    • kfp.compiler: Compiles Python DSL into YAML workflow specifications.
    • kfp.components: Tools for interacting with components (e.g., converting functions to containers).
    • kfp.dsl: The Domain-Specific Language (DSL) for defining pipelines, components, and control flow.
    • kfp.Client: Python client libraries for interacting with the Kubeflow Pipelines API.
    • kfp.extensions: Platform-specific modules (GCP, AWS, Azure, etc.).
    • kfp.cli.diagnose_me: Modules for environment diagnostics.

  12. What is the Kubeflow Central Dashboard?

    master

    The Kubeflow Central Dashboard is an authenticated web interface that serves as a central hub for your machine learning platform. It exposes the user interfaces (UIs) of various Kubeflow components running in your cluster and provides a unified access point for managing them.

    Key capabilities include:

    • Authentication and Authorization: Access control is managed via Profiles and Namespaces.
    • Component Access: Direct navigation to the UIs of installed Kubeflow components.
    • Customization: Ability to customize and include links to third-party applications.