Open Model Engine (OME)

Multi Node Prober is a Golang-based HTTP server designed to manage Kubernetes pod health probes for applications running vLLM within a Ray Cluster. It acts as a proxy that exposes standard Kubernetes probe endpoints by checking the health of the underlying vLLM service.

It provides three primary endpoints:

• /healthz (Liveness): Checks if vLLM and its services are running and responsive.
• /readyz (Readiness): Ensures vLLM is ready to accept traffic (e.g., after model initialization).
• /startupz (Startup): Verifies the vLLM has started correctly, specifically useful for containers with long startup durations.

A FineTunedWeight is a cluster-scoped Kubernetes resource that represents weights fine-tuned from an existing Base Model. Instead of a standalone model, it contains only the artifacts and metadata specific to the fine-tuning process (e.g., a LoRA adapter or a distilled variant) and points back to a base model.

Key characteristics:

• Independence: You can manage multiple customizations of the same base model independently, each with its own storage, metadata, and lifecycle.
• Composition: When an InferenceService references one or more FineTunedWeight resources, OME serves them on top of the referenced base model.
• Scope: Being cluster-scoped, it is available to workloads in any namespace.

Accelerator-Aware Runtime Selection is a feature in OME that allows for efficient model deployment in heterogeneous GPU environments. It introduces two primary abstractions:

1. AcceleratorClass (Cluster-scoped): Defines specific accelerator capabilities (e.g., memory, compute capability) and discovery patterns (e.g., nodeSelector).
2. ServingRuntime and InferenceService extensions: Allow users to specify accelerator requirements, preferences, and GPU-specific optimizations.

This system enables the automatic matching of model requirements with the optimal available hardware, allowing platform engineers to define universal runtimes that automatically apply specific optimizations (like FP8 or speculative decoding) based on the detected GPU type.

A BenchmarkJob is an OME resource used to automate performance benchmarking of inference services or OCI Generative AI Service endpoints. It leverages genai-bench to evaluate model serving performance under different traffic patterns and load conditions.

A BenchmarkJob consists of four core components:

1. Endpoint Configuration: The target service being tested.
2. Traffic Patterns: The load testing scenarios (e.g., constant load, bursty load).
3. Resource Configuration: The environment where the benchmark runs (pod resources, images).
4. Output Management: Where the resulting benchmark data is stored.

A ClusterServingRuntime is a cluster-scoped resource in OME that defines the templates for Pods used to serve one or more specific models. It acts as a reusable configuration that specifies the runtime environment, including the container image, supported model formats, and hardware requirements.

By using ClusterServingRuntime (cluster-scoped) or its namespace-scoped counterpart, users can customize or define new runtimes without modifying the OME controller code. This allows for high flexibility in managing different serving engines like SGLang or VLLM across the cluster.