llmaz Inference Platform

The llmaz repository is organized into several key directories that separate binaries, deployment logic, and core application code:

• bin/: Contains binaries used by the project (e.g., kustomize, ginkgo).
• chart/: Contains Helm charts for deployment.
• cmd/: Contains the main entry points for the application.
• docs/: Contains documentation, including examples and installation guidance.
• llmaz/: Contains the model loader logic.
• pkg/: Contains the primary logic for Kubernetes controllers.

llmaz # root
├── bin # where the binaries locates, like the kustomize, ginkgo, etc.
├── chart # where the helm chart locates
├── cmd # where the main entry locates
├── docs # where all the documents locate, like examples, installation guidance, etc.
├── llmaz # where the model loader logic locates
├── pkg # where the main logic for Kubernetes controllers locates

llmaz supports a wide variety of inference backends, allowing you to integrate different serving frameworks depending on your hardware and performance requirements. Supported backends include:

• llama.cpp: For minimal setup and high performance across diverse hardware (local and cloud).
• ollama: A local deployment tool based on llama.cpp, supporting models like Llama 3.2, Mistral, and Gemma 2.
• SGLang: A fast serving framework for Large Language Models (LLMs) and Vision Language Models (VLMs).
• TensorRT-LLM: NVIDIA's optimized framework for efficient inference on NVIDIA GPUs using Python and C++ runtimes.
• Text-Generation-Inference (TGI): A high-performance Rust/Python/gRPC server used by Hugging Face for production-scale inference.
• vLLM: A high-throughput, memory-efficient inference and serving engine.

Heterogeneous Cluster Support allows the InftyAI scheduler to manage workloads across different GPU types (e.g., mixing A100, A10, and 4090 GPUs). This concept is known as Resource Fungibility.

By using the ResourceFungibility plugin in the InftyAI scheduler, you can schedule workloads on up to 8 alternative GPU types when high-end GPUs are unavailable. This helps maintain Service Level Objectives (SLOs) and optimizes costs by utilizing available, diverse hardware.

The Metric Aggregator is a proposed component designed to handle the unique scheduling challenges of GenAI workloads. Unlike traditional services that use simple round-robin or least-connection algorithms, GenAI inference requires advanced scheduling based on heavy computational metrics.

The aggregator collects indicators from inference engines to enable latency-aware and resource-aware dispatching. This allows for more intelligent routing decisions in clusters where inference time is significant and variable.

When implementing or using metric-based routing with the aggregator, the following constraints and behaviors apply:

• Fallback Mechanism: If the metrics are unavailable, outdated, or if a selected Pod becomes suddenly unavailable after the gateway has picked it, the system must support a fallback to default service routing.
• Resilience: The system must remain functional even if metrics are suboptimal. For example, if metrics are outdated, the system should still work, even if it results in slower response times.
• Error Handling: Routing decisions should avoid scenarios that lead to hard failures (like 500 server errors). While latency-based routing might result in a slow response if metrics are wrong, scheduling that relies on specific model availability (like LoRA) must be handled carefully to avoid unacceptable errors.

llmaz separates concerns between two primary roles to hide infrastructure complexity:

1. The User: Focuses on deploying LLMs by providing a model name and replica configurations. Users interact primarily with the Playground abstraction and do not need deep Kubernetes knowledge.
2. The Platform Runner: Manages the underlying complexity via the BackendRuntime. This role handles the specific configurations of inference engines (like vLLM), resource requirements, boot commands, and health probes.

This separation allows users to use high-level templates while still allowing the platform to optimize configurations or allowing advanced users to override them.

The Metrics Aggregator (MA) currently functions as both a control plane (to sync metrics) and a data plane. It consists of two primary components:

• Pod Controller: Manages the lifecycle of Pods. When a Pod becomes ready, the controller adds it to the internal store and spawns a background goroutine to continuously sync metrics (defaulting to a 100ms interval). When a Pod is deleted, the associated goroutine is stopped and the Pod is removed from the store.
• Internal Store: Responsible for parsing metric results and persisting them to backend storage. While the current implementation only supports disk memory, the system uses an interface that allows for future storage extensions.

Goals

• Provision AWS Spot instances for inference workloads based on the model's flavor requirements.
• Support flexible and preferred GPU scheduling using Spot instances.
• Ensure service availability by managing spot instance disruption events (Beta requirement).
• Maintain latency sensitivity by accelerating model loading during scaling (Beta requirement).

Non-Goals

• Integration with the standard Kubernetes Cluster Autoscaler.
• Adding custom scheduler support to upstream Karpenter (this is tracked separately in the Karpenter project).
• Support for multi-host inference (due to stability risks during reclamation).