Cluster API Provider Azure (CAPZ)

Azure Service Operator (ASO) is a collection of Kubernetes Custom Resource Definitions (CRDs) and controllers that extend the Kubernetes API to manage Azure resources (such as virtual machines and resource groups) declaratively.

In the context of Cluster API Provider for Azure (CAPZ), ASO is used to handle the reconciliation of Azure infrastructure. Instead of CAPZ making direct Azure API calls and managing the state of long-running operations manually, CAPZ translates Cluster API resource definitions into ASO custom resources. ASO then takes responsibility for ensuring the desired state is achieved in Azure. This allows CAPZ to focus on the transformation of Cluster API resources into infrastructure definitions rather than the low-level mechanics of Azure API interaction.

ClusterClass is an experimental feature (requires the ClusterTopology=true feature gate) that uses a collection of templates to define a cluster topology, including control plane and machine deployments. It allows for continuous reconciliation of one or more clusters using shared parameters to streamline lifecycle management while using the standard Cluster API underlying API.

CAPZ implements ClusterClass using four custom resources that act as templates for their corresponding CAPZ resources:

1. AzureClusterTemplate: Template for AzureCluster resources.
2. AzureManagedClusterTemplate: Template for managed (AKS) clusters.
3. AzureManagedControlPlaneTemplate: Template for the managed control plane.
4. AzureManagedMachinePoolTemplate: Template for managed machine pools (agent pools).

This proposal outlines a strategy to bridge the gap between the full Azure Kubernetes Service (AKS) feature set and what is natively exposed via Cluster API Provider for Azure (CAPZ).

Currently, CAPZ exposes an opinionated subset of AKS features through AzureManagedControlPlane and AzureManagedMachinePool resources. When new features are released in AKS, they are not immediately available in CAPZ, requiring manual implementation, review, and testing. This creates a gap for users who want to use the latest AKS capabilities or adopt existing AKS clusters into Cluster API management.

The goal of this automation is to make all AKS features available in CAPZ on an ongoing basis with minimal maintenance, reducing the manual effort required to keep CAPZ in sync with the rapidly evolving AKS API.

Azure Workload Identity (AZWI) is the successor to the deprecated AAD Pod Identity. It allows Kubernetes workloads to authenticate to Azure resources (like Key Vault or Virtual Machines) using OpenID Connect (OIDC) federation, removing the need for managing static secrets or managing complex AAD Pod Identity components that intercept IMDS traffic.

In the context of Cluster API Provider Azure (CAPZ), the primary goal of this integration is to enable CAPZ pods running in a management cluster to use AZWI to authenticate to Azure. This allows the provider to create, update, or delete Azure resources as part of the workload cluster lifecycle management without relying on static Azure credentials.

A MachinePool is a Cluster API (CAPI) resource used to manage a group of machines as a single entity by delegating orchestration to the underlying infrastructure provider (e.g., Azure Virtual Machine Scale Sets).

Key differences:

• MachineDeployment: Uses MachineSets to orchestrate updates and maintain the desired number of Machine resources. It manages individual machines directly.
• MachinePool: Delegates the responsibility of availability, health, and configuration updates to provider-specific resources like Azure VMSS. It is an optional abstraction and does not replace MachineSet/Machine for providers that lack such a grouping mechanism.

Feature Status: Experimental (Beta). Enable via the MachinePool=true feature gate.

If you want to avoid the overhead of rotating Service Principal secrets, you can use an Azure User Assigned Identity to provision workload cluster infrastructure.

Workflow:

1. Create a User Assigned Identity in Azure with the necessary RBAC rights (e.g., Subscription Owner) for the target workload subscription.
2. In the management cluster running CAPZ controllers, create Cluster API resources that reference this User Assigned Identity.
3. The CAPZ controller uses the User Assigned Identity credentials to reconcile the AzureCluster and manage the infrastructure in the target subscription.

The Node OS Image caching feature follows a specific graduation path:

Experimental Phase

• The feature is toggleable via an environment variable.
• Implementation includes end-to-end (e2e) tests to verify that new nodes created after a patch match the OS image of the original nodes.

Stable Phase

• The feature is enabled by default.
• It utilizes a more sophisticated method for selecting healthy nodes, specifically by annotating AzureMachinePoolMachine instances after every patch. This optimization allows the controller to determine if updates are actually needed within a specific interval, reducing unnecessary operations.

Node OS Image caching is an optional feature for the AzureMachinePool controller designed to accelerate horizontal scaling and improve security compliance.

Instead of relying on standard images that may require immediate patching upon boot, this feature uses the Node Prototype Pattern. In this pattern, a known good/working OS snapshot is taken from a running node to create a "prototype" image. Future scale-out events then use this cached image, ensuring that new nodes are already patched and compliant, which reduces wait times for security updates and container image pre-caching.

The automation of AKS features in CAPZ is planned to be implemented via new API versions for AzureManagedControlPlane and AzureManagedMachinePool.

• Backward Compatibility: Existing v1beta1 types will continue to be served. Users with existing clusters do not need to take immediate action; their clusters will continue to function as they currently do.
• New API Versions: The proposed features will reside in a new v2alpha1 API version.
• Conversion Mechanism: For certain implementation options (Option 3 and Option 4), a backwards-compatible version like v1beta2 may be introduced, using conversion webhooks to bridge the gap between v1beta1 and v1beta2.