repository·main·Indexed 40 days ago
https://github.com/siderolabs/talos-cloud-controller-managerA Kubernetes Cloud Controller Manager (CCM) implementation for Talos that enables multi-cloud management. It integrates Kubernetes with cloud infrastructure to manage resources like persistent volumes and load balancers, featuring specialized controllers including cloud-node, cloud-node-lifecycle, node-ipam-controller, and node-csr-approval.
The Talos CCM manages the lifecycle of nodes in the following sequence:
--cloud-provider=external joins the cluster. The Node object is created with the node.cloudprovider.kubernetes.io/uninitialized taint, the alpha.kubernetes.io/provided-node-ip annotation, and nodeInfo.Node object with necessary labels, taints, and the providerID field.node.cloudprovider.kubernetes.io/uninitialized taint, making the node ready for use.Warning: If the Kubelet is not configured with --cloud-provider=external, it will attempt to manage its own lifecycle, causing the CCM to skip the node and fail to initialize it.
The cloud-node-controller is enabled by default. It detects new cloud instances and registers them as Kubernetes nodes, assigning labels and taints based on cloud metadata.
Labels applied to nodes:
topology.kubernetes.io/regiontopology.kubernetes.io/zonenode.kubernetes.io/instance-typenode.cloudprovider.kubernetes.io/clustername (Talos specific)node.cloudprovider.kubernetes.io/platform (Talos specific)node.cloudprovider.kubernetes.io/lifecycle (Talos specific - e.g., spot instance type)Node Specs:
providerID magic stringInternalIP and ExternalIP addresses# Enable specifically via CLI flag
--controllers=cloud-nodeWithin transformation rules (annotations, labels, or platformMetadata), you can use Go templates to access platform-specific metadata. These variables are case-sensitive.
Available variables include:
Platform: The Talos platform (e.g., aws, gcp, azure, openstack, metal).Hostname: The node's hostname.Region: The cloud region (e.g., us-east-1).Zone: The cloud zone (e.g., us-west-1f).InstanceType: The instance type (e.g., t2.micro).InstanceID: The cloud instance ID.ProviderID: The cloud provider ID.Spot: Boolean indicating if it is a spot instance.InternalDNS: Internal DNS name.ExternalDNS: External DNS name.You can inspect these values using the talosctl command:
talosctl get PlatformMetadatas -oyamlTransformation rules can also access hardware-level system information via Go templates. These variables are case-sensitive.
Available variables include:
Manufacturer: Hardware manufacturer.ProductName: Product name.Version: Hardware version.SerialNumber: Hardware serial number.UUID: Hardware UUID.WakeUpType: Wake-up type.SKUNumber: System SKU number.You can inspect these values using the talosctl command:
talosctl get SystemInformation -oyamlTalos CCM provides several specialized controllers to manage different aspects of cloud-integrated Kubernetes nodes:
Node resource with cloud metadata and assigns labels and taints based on cloud metadata and specific configuration.Node resource when a cloud instance is deleted.The cloud-node-lifecycle-controller is currently optimized for GCP. It handles GCP spot instance evictions by catching the event and removing the node resource from the cluster. This allows the node to re-initialize and join the cluster cleanly after the instance is recreated.
# Enable via CLI flag
--controllers=node-lifecycle-controllerThe node-ipam-controller manages the allocation of CIDR addresses to pods across nodes. It is particularly useful in hybrid environments to solve IPv6 CIDR allocation problems by allowing each node to have a unique IPv6 CIDR range based on cloud metadata.
Key Concepts:
--node-cidr-mask-size-ipv4 and --node-cidr-mask-size-ipv6.RangeAllocator: The default Kubernetes CIDR allocator.CloudAllocator: A custom allocator that uses Talos/Cloud metadata to allocate ranges.Recommended CLI Arguments:
To enable the CloudAllocator and set specific mask sizes, use:
--controllers=node-ipam-controller \
--allocate-node-cidrs \
--node-cidr-mask-size-ipv4=24 \
--node-cidr-mask-size-ipv6=80 \
--cidr-allocator-type=CloudAllocator# Example CLI invocation for Node IPAM
--controllers=node-ipam-controller \
--allocate-node-cidrs \
--node-cidr-mask-size-ipv4=24 \
--node-cidr-mask-size-ipv6=80 \
--cidr-allocator-type=CloudAllocatorYou can install the Talos Cloud Controller Manager using one of three methods:
Integrate the installation directly into your Talos cluster configuration by enabling the externalCloudProvider feature and providing the manifest URL.
Apply the deployment manifests directly using kubectl. You can choose between the latest release or the latest stable (edge) version.
Deploy using the official Helm chart via OCI registry. This is the recommended method for production environments.
# Method 1: Talos machine config
cluster:
externalCloudProvider:
enabled: true
manifests:
- https://raw.githubusercontent.com/siderolabs/talos-cloud-controller-manager/main/docs/deploy/cloud-controller-manager.yml# Method 2: kubectl (Latest)
kubectl apply -f https://raw.githubusercontent.com/siderolabs/talos-cloud-controller-manager/main/docs/deploy/cloud-controller-manager.yml
# Method 2: kubectl (Edge)
kubectl apply -f https://raw.githubusercontent.com/siderolabs/talos-cloud-controller-manager/main/docs/deploy/cloud-controller-manager-edge.yml
# Method 3: Helm
helm upgrade -i -n kube-system talos-cloud-controller-manager oci://ghcr.io/siderolabs/charts/talos-cloud-controller-managerTo use the Talos Cloud Controller Manager (CCM), you must configure your Talos machine configuration to allow an external cloud provider and enable specific API access features.
Set the following in your control plane configuration to enable external cloud provider support, server certificate rotation, and Talos API access for the kube-system namespace:
machine:
kubelet:
extraArgs:
cloud-provider: external
rotate-server-certificates: true
features:
kubernetesTalosAPIAccess:
enabled: true
allowedRoles:
- os:reader
allowedKubernetesNamespaces:
- kube-systemWorker nodes must be configured to use the external cloud provider and enable certificate rotation:
machine:
kubelet:
extraArgs:
cloud-provider: external
rotate-server-certificates: trueYou can deploy the Talos CCM using the provided Helm chart. It is recommended to use a custom values.yaml file to configure the number of replicas and the specific controllers you wish to enable.
talos-ccm.yaml):replicaCount: 2
enabledControllers:
- cloud-node
- node-csr-approvalhelm upgrade -i --namespace=kube-system -f talos-ccm.yaml \
talos-cloud-controller-manager charts/talos-cloud-controller-managerhelm upgrade -i --namespace=kube-system -f talos-ccm.yaml \
talos-cloud-controller-manager charts/talos-cloud-controller-managerUse Cosign keyless verification to validate container images. Depending on the image type, use the corresponding GitHub Action identity and workflow reference.
To verify the edge image, use the build-edge.yaml workflow identity.
To verify a specific release tag, use the workflow identity associated with that specific tag (e.g., release.yaml@refs/tags/v1.4.1).
# Verify Edge version
cosign verify ghcr.io/siderolabs/talos-cloud-controller-manager:edge --certificate-identity https://github.com/siderolabs/talos-cloud-controller-manager/.github/workflows/build-edge.yaml@refs/heads/main --certificate-oidc-issuer https://token.actions.githubusercontent.com
# Verify a specific Release
cosign verify ghcr.io/siderolabs/talos-cloud-controller-manager:v1.4.1 --certificate-identity https://github.com/siderolabs/talos-cloud-controller-manager/.github/workflows/release.yaml@refs/tags/v1.4.1 --certificate-oidc-issuer https://token.actions.githubusercontent.com