Azure Kubernetes Service (AKS) Community Repository

AKS provides several built-in enhancements designed to improve the scalability, performance, and stability of large clusters. These features are applied automatically and do not require manual configuration. Key enhancements include:

• Streaming encoder for LIST responses: Reduces memory consumption during large LIST requests by processing items individually rather than serializing the entire response into a single memory block. Available in AKS versions 1.31.9+ and 1.32.6+.
• Higher control plane resource limits: Allows the control plane to receive up to 4x higher CPU and memory limits during autoscaling, which is particularly useful for AI inference and training workloads.
• AKS managed API server guard: A safeguard that throttles non-system API server requests when the API server remains unstable or experiences OOM incidents despite scaling.
• etcd defragmentation optimizations: Reduces etcd defragmentation time by up to 50%, minimizing response time spikes and transient client timeouts.

Note that cluster scalability still depends on the 'Kubernetes scale envelope,' which includes the number and size of cluster objects (pods, nodes, CRDs, Secrets, etc.) and client interaction patterns.

KubeFleet is an open-source, multi-cluster Kubernetes resource management solution designed to orchestrate and distribute Kubernetes resources (such as configs like roles/quotas/policies and applications like deployments/services) across a fleet of clusters.

Architecture: Hub-and-Spoke Model

KubeFleet operates using a hub-and-spoke model:

• Hub Cluster: Acts as the central control plane for the entire fleet. It hosts the orchestration and coordination logic.
• Member Clusters: Part of the fleet that receive resources from the hub.

Connectivity and Security

A key architectural feature is its support for one-way connections. KubeFleet does not require connections from the hub cluster to member clusters; instead, it only requires connections from member clusters to the hub. This allows member clusters to remain private and stay behind network isolation/private egress.

NVIDIA Dynamo is an open-source distributed inference serving framework designed to optimize large-scale LLM workloads. When deployed on Azure Kubernetes Service (AKS) using ND GB200-v6 VMs, it provides a holistic solution for high-performance inference by disaggregating the prefill and decode phases.

Core Components of NVIDIA Dynamo

• Disaggregated Execution: Separates the compute-bound prefill phase from the memory-bound decode phase across different GPUs. This allows for independent scaling and the use of specialized hardware configurations for each phase (e.g., high-compute GPUs for prefill and high-memory GPUs for decode).
• SLA-based Planner: Proactively manages GPU scaling. It uses pre-deployment profiling to recommend configurations that meet latency targets like Time to First Token (TTFT) and Inter-Token Latency (ITL). At runtime, it uses time-series models to forecast traffic and dynamically adjust worker counts.
• LLM-aware Router: Manages the Key-Value (KV) cache across GPU clusters. It hashes requests and tracks cache locations to route incoming requests to GPUs that maximize cache reuse, reducing redundant computation.
• KV Block Manager (KVBM): Reduces GPU memory overhead by offloading infrequently accessed KV blocks to CPU RAM, SSDs, or object storage, enabling petabyte-scale cache storage.

Supported Engine Backends

Dynamo supports multiple engine backends, including:

• vLLM
• TensorRT-LLM
• SGLang

mirrord is an open-source "remocal" development tool that connects a local process to a remote Kubernetes cluster (such as AKS). Instead of using a VPN, mirrord hooks system calls in your local process to proxy environment variables, files, and network traffic to and from the cluster.

Key benefits include:

• Non-intrusive mirroring: In mirror mode, traffic is copied to your local process, but the cluster's original pod still handles the request and the response from your local process is discarded. This allows you to observe traffic without affecting the cluster.
• Intrusive stealing: In steal mode, mirrord intercepts traffic intended for a cluster pod and routes it to your local process. The response from your local process is then sent back to the cluster, allowing you to test code changes in real-time as if they were deployed.
• Low overhead: It requires no root privileges on your machine and features fast startup speeds.
• Highly configurable: You can granularly specify which files, environment variables, and network traffic are shared between the local and remote environments.

Azure Container Storage supports several backing storage options depending on your performance and resiliency requirements:

Azure Disks (GA)

Best for tier 1 and general-purpose databases (e.g., MySQL, MongoDB, PostgreSQL) and workloads requiring rapid scaling of thousands of PVs (e.g., JupyterHub, ElasticSearch). Supports storage-level replication using Premium SSD ZRS or Standard SSD ZRS.

Ephemeral Disks (GA)

Best for latency-sensitive and IOPS-intensive workloads (e.g., Cassandra). Requires storage-optimized VM SKUs like L-series, Ev3-series, or Esv3-series which provide local NVMe or temporary storage. Supports synchronous replication across NVMe disks in a storage pool.

Azure Elastic SAN (Preview)

Best for on-demand, fully managed block storage where you want to avoid granular control and focus on rapid provisioning. Ideal for general-purpose databases, streaming/messaging services (e.g., Kafka), and CI/CD environments. Uses the iSCSI protocol for fast attach/detach.