DOCSEARCHbyAlgolia
Submit Docs

Seldon Core 2 Documentation

repository·v2·Indexed 26 days ago

https://github.com/seldonio/seldon-core

An MLOps and LLMOps framework for deploying, managing, and scaling AI systems on Kubernetes. Seldon Core 2 features a microservice architecture split into a Control Plane (Scheduler, Agent, Controller) and a Data Plane (Envoy, Pipeline Gateway, Model Gateway, Dataflow Engine). It supports individual models and complex data-centric applications using Kafka-based dataflows and the Open Inference Protocol (OIP).

Tokens
210.2K
Snippets
496
Records
1.1K
Agent score
86%

What's inside Seldon Core 2

  1. Overview of SCv2 Operational Metrics

    v2

    Seldon Core v2 collects operational metrics via Prometheus to monitor throughput, latency, memory, and CPU. These metrics are categorized by the component providing them:

    • Agent Metrics: Collected by the agent sitting next to the inference servers.
    • Gateway Metrics: Collected by the pipeline gateway that handles requests to pipelines.

    Note: Many metrics are counters and gauges at the model and pipeline levels. While per-pipeline histogram metrics are currently stored, per-model histogram metrics are not currently stored for performance reasons. These metrics are experimental and subject to change.

  2. Overview of Seldon Core 2 Kubernetes Resources

    v2

    Seldon Core 2 provides a set of Custom Resources (CRDs) to manage the lifecycle of machine learning deployments on Kubernetes. The primary resources include:

    • SeldonRuntime: Used to install Seldon within a specific namespace.
    • Servers: Used to deploy sets of replicas for core inference servers, such as MLServer or Triton.
    • Models: Used to deploy single machine learning models, custom transformation logic, drift detectors, outlier detectors, and explainers.
    • Experiments: Used for testing new versions of models.
    • Pipelines: Used to connect data flows between different models.

  3. Overview of Kafka in Seldon Core 2

    v2

    Kafka is an external component used by Seldon Core 2 to provide scalable, reliable, and flexible communication for machine learning deployments. It acts as the backbone for building complex inference pipelines, managing high-throughput asynchronous predictions, and integrating with event-driven systems.

    Important Note: Kafka is managed outside of the main Seldon stack. Cluster administrators are responsible for the administration and management of the Kafka instance. For production environments, using a managed Kafka instance is highly recommended.

  4. Overview of the Agent Service

    v2

    The Agent Service facilitates communication between the scheduler and agents regarding model lifecycle management. It enables two primary workflows:

    1. Model Loading/Unloading: The scheduler can request an agent to load or unload a specific model.
    2. Lifecycle Notifications: The agent can notify the scheduler when a model has successfully been loaded or unloaded.

  6. Overview of Seldon V2 Capabilities

    v2

    Seldon V2 provides a machine learning inference platform designed for both local development (Docker) and production scale (Kubernetes). It supports a wide range of standard and custom artifacts, allowing for the deployment of single models or complex multi-step pipelines. Key capabilities include:

    • Multi-model serving: Efficiently pack multiple models onto fewer inference servers to reduce infrastructure costs.
    • Resource Overcommitment: Deploy more models than available memory by allowing Seldon to transparently unload inactive models.
    • Extensible Pipelines: Dynamically extend models with pipelines using a data-centric approach backed by Kafka.
    • Observability: Deploy drift detectors, outlier detectors, and state-of-the-art explanation techniques alongside models.
    • Service Mesh Agnostic: Integrates with any Kubernetes service mesh (e.g., Istio, Ambassador, Traefik).

  7. Overview of the Open Inference Protocol

    v2

    The Open Inference Protocol (formerly known as the 'V2 Inference Protocol') is a framework-agnostic API designed for machine learning/deep learning inference. It provides standardized interfaces for both easy-to-use and high-performance use cases, ensuring portability between different inference clients and servers.

    Key characteristics:

    • Interoperability: Endorsed by NVIDIA Triton Inference Server, TensorFlow Serving, and ONNX Runtime Server.
    • Transport Options: A compliant server may implement either or both the HTTP/REST API and the GRPC API.
    • Extensibility: The protocol includes a required extension mechanism to allow for future specialized features.
    • Compliance: To be considered compliant, a server must implement all described APIs unless a feature is explicitly marked as optional.

  8. Overview of Seldon Core 2 Documentation

    v2

    Seldon Core 2 is a production-ready ML serving framework designed for deploying modular, data-centric AI applications at scale. The documentation is organized into several key areas:

    • Installation: Covers learning environments (self-hosted Kafka), production environments (managed Kafka, Istio ingress), and advanced configurations (Helm, ServerConfig, SeldonConfig, PipelineConfig).
    • User Guide: Provides instructions for Quickstart, managing Kubernetes resources, configuring Servers (including GPU allocation), and managing Models (Multi-Model Serving, Parameterized Models, Scheduling).
    • Inference: Details how to use Inference Servers, run inference, and perform batch inference.
    • Open Inference Protocol: Documentation for the v2 protocol, including REST implementation.
    • Pipelines & Scaling: Guides on building pipelines and implementing autoscaling via Seldon Core or Kubernetes HPA.
    • Monitoring: Covers Data Science monitoring (Drift, Outlier detection, Explainability) and Operational monitoring (Observability, Tracing, Metrics).
    • Integrations: Support for Service Meshes (Istio, Ambassador, Traefik) and Confluent (Schema Registry).
    • CLI & APIs: Reference for the Seldon CLI (managing Config, Experiments, Models, Pipelines, and Servers) and various API surfaces.

  9. Overview of Seldon Kubernetes Custom Resources

    v2

    Seldon provides a set of Kubernetes Custom Resources (CRDs) to manage the lifecycle of AI applications. The primary resources are:

    • SeldonRuntime: Used for installing Seldon within a specific namespace.
    • Servers: Used to deploy sets of replicas for core inference servers (such as MLServer or Triton).
    • Models: Used to deploy individual machine learning models, custom transformation logic, drift detectors, outlier detectors, and explainers.
    • Experiments: Used for testing new versions of models.
    • Pipelines: Used to connect data flows between multiple models.

  10. Overview of Seldon Core 2

    v2
    Seldon Core 2 is a Kubernetes-native MLOps framework designed for deploying and managing machine learning (ML) and Large Language Model (LLM) systems at scale. It utilizes a data-centric approach and a modular architecture to support deployments across on-premise, hybrid, and multi-cloud environments. The framework is designed to handle everything from simple models to complex ML applications with a focus on flexibility, standardization, observability, and cost efficiency.

  11. Use Multi-model Serving and Overcommit

    v2

    Seldon Core 2 is designed for multi-model serving (MMS) by default. This architecture allows a single ML inference server (like Nvidia Triton or Seldon MLServer) to host multiple models simultaneously, reducing hardware costs (e.g., GPU usage).

    Key Features:

    • Automatic Server Selection: Seldon finds an appropriate server based on the requirements defined in your Model deployment.
    • Resource Overcommit: Seldon can register more models than the provisioned memory infrastructure can support. It manages this by dynamically swapping models based on usage (least used) to minimize latency overhead while maximizing model density.

  12. Key features of Seldon Core 2

    v2

    Seldon Core 2 provides several advanced capabilities for AI deployment:

    • Pipelines: Deploy composable AI applications using Kafka for real-time data streaming between components.
    • Autoscaling: Scale models and application components based on native or custom logic.
    • Multi-Model Serving: Consolidate multiple models on shared inference servers to reduce infrastructure costs.
    • Overcommit: Deploy more models than available memory allows to optimize costs for infrequently used models.
    • Experiments: Support for A/B testing and shadow deployments by routing data between candidate models or pipelines.
    • Custom Components: Plug-and-play integration for custom logic, drift & outlier detection, and LLMs.