DOCSEARCHbyAlgolia
Submit Docs

Proxmox Cloud Controller Manager (CCM)

repository·main·Indexed 43 days ago

https://github.com/sergelogvinov/proxmox-cloud-controller-manager

A Kubernetes Cloud Controller Manager for Proxmox that automates node initialization, labeling, and cleanup for virtual machines. It manages the lifecycle of Kubernetes nodes via cloud-node and cloud-node-lifecycle controllers, mapping Proxmox environment data to Kubernetes labels and providerIDs.

Tokens
10.7K
Snippets
31
Records
38
Agent score
55%

What's inside Proxmox Cloud Controller Manager

  1. Understand Node Labels and providerID mapping

    main

    The CCM automatically applies several labels and a providerID to Kubernetes nodes based on the Proxmox environment. This enables pod affinity/anti-affinity and cluster management.

    Kubernetes Labels

    • node.kubernetes.io/instance-type: Generated based on the VM's CPU and RAM (e.g., 2VCPU-2GB).
    • topology.kubernetes.io/region: Maps to the Proxmox cluster region defined in the config.
    • topology.kubernetes.io/zone: Maps to the Proxmox hypervisor host machine name.
    • topology.proxmox.sinextra.dev/region: Proxmox-specific region label.
    • topology.proxmox.sinextra.dev/zone: Proxmox-specific zone label.
    • group.topology.proxmox.sinextra.dev/${HAGroup}: HA group labels (similar to node-role).

    providerID Format

    Nodes are identified by a magic string following this pattern: proxmox://<cluster-name>/<vm-id>

    Example: proxmox://cluster-1/123

  2. How the Proxmox CCM controllers work

    main

    The CCM operates using two primary controllers to manage the lifecycle of Kubernetes nodes in relation to Proxmox VMs:

    1. cloud-node: Responsible for updating the Kubernetes Node resource. It assigns labels and taints based on the specific Proxmox VM configuration.
    2. cloud-node-lifecycle: Responsible for cleanup. It removes the Kubernetes Node resource when the corresponding Proxmox VM is deleted.

  3. Configure Node Addressing modes

    main

    Proxmox CCM supports three modes for determining node IP addresses. All modes will respect IPs provided via the alpha.kubernetes.io/provided-node-ip annotation unless they are explicitly excluded via the ignored cidrs list in non-default modes.

    1. Default Mode: The node must be provided with its private IP address via the kubelet flag --node-ip. This mode sets the node's internal IP but does not set an External IP. There is no IP validation in this mode.
    2. Auto Mode: Uses both host-networking access and the QEMU guest agent API to detect IP addresses. It sets all detected addresses as internal IPs and can be configured to identify specific addresses as external based on provided CIDRs.
    3. QEMU-only Mode: Relies exclusively on the QEMU guest agent API to retrieve IP addresses and set them as Node Addresses. It also respects the alpha.kubernetes.io/provided-node-ip annotation.

  4. How Proxmox CCM Node Lifecycle Works

    main

    When kubelet is configured with --cloud-provider=external, the following lifecycle occurs:

    1. Node Join: kubelet joins the cluster and sends a Node object to the API server. This object contains a node.cloudprovider.kubernetes.io/uninitialized taint.
    2. Detection: CCM detects the new node and queries the Proxmox API for VM configuration (VMID, hostname, etc.).
    3. Update: CCM updates the Node object with labels, taints, and a providerID.
      • Note: The providerID is immutable and follows the format proxmox://$REGION/$VMID.
    4. Ready: CCM removes the node.cloudprovider.kubernetes.io/uninitialized taint, allowing the node to be used.

    Warning: If kubelet is NOT configured with --cloud-provider=external, it will attempt to manage the node itself, causing a conflict. CCM will skip the node, and it will remain uninitialized.

  5. Gather metrics from Proxmox CCM

    main

    The Proxmox CCM exposes metrics on the https://localhost:10258/metrics endpoint by default. To ensure the metrics endpoint is accessible, you must allow the path through the authorization layer and specify the secure port.

    If running the binary directly, use the following flags:

    proxmox-cloud-controller-manager --authorization-always-allow-paths="/metrics" --secure-port=10258

  6. Configure the Proxmox Cloud Controller Manager

    main

    The Proxmox CCM is configured using a YAML file that defines global features and a list of clusters. The configuration allows you to manage multiple Proxmox clusters, specify authentication via API tokens, and control how network addresses and provider IDs are handled.

    Configuration Structure

    features section

    Controls global behavior for the CCM:

    • provider: Determines the providerID format. Use default for proxmox://<region>/<vm-id> or capmox for Cluster API for Proxmox (CAPMox) which uses proxmox://<SystemUUID>.
    • network: Defines how node IPs are retrieved. Options are default (uses kubelet --node-ips), qemu (uses QEMU agent API), or auto (detects best mode).
    • ipv6_support_disabled: If true, the CCM ignores IPv6 addresses.
    • external_ip_cidrs: A comma-separated list of CIDRs to identify external IPs. Use ! to exclude a specific CIDR.
    • ip_sort_order: A comma-separated list of CIDRs to define the priority order for IP sorting. IPs not matching these CIDRs are sorted by detection order.
    • ha_group: If true, Proxmox HA groups are used as Kubernetes zone labels.

    clusters section

    A list of Proxmox clusters to manage. Each cluster entry requires:

    • url: The Proxmox cluster API URL.
    • insecure: Set to true to skip TLS certificate verification.
    • token_id: The Proxmox API token ID.
    • token_secret: The name of the Kubernetes Secret containing the Proxmox API token.
    • region: The region name (used for topology.kubernetes.io/region labels).

    Note: You can also use token_id_file and token_secret_file to load tokens from specific file paths instead of Kubernetes Secrets.

    features:
  provider: default|capmox
  network: default|qemu|auto
  ipv6_support_disabled: true|false
  external_ip_cidrs: '192.168.0.0/16,2001:db8:85a3::8a2e:370:7334/112,!fd00:1234:5678::/64'
  ip_sort_order: '192.168.0.0/16,2001:db8:85a3::8a2e:370:7334/112'
  ha_group: true|false

clusters:
  - url: https://cluster-api-1.exmple.com:8006/api2/json
    insecure: false
    token_id: "kubernetes-csi@pve!csi"
    token_secret: "secret"
    region: Region-1

  - url: https://cluster-api-2.exmple.com:8006/api2/json
    insecure: false
    token_id: "kubernetes-csi@pve!csi"
    token_secret: "secret"
    region: Region-2

  7. Configure Kubelet for Proxmox CCM

    main

    To use Proxmox CCM, you must configure the kubelet on all nodes to offload cloud-specific responsibilities to the external controller. This is done by setting the --cloud-provider=external flag.

    Required Flag

    kubelet --cloud-provider=external

    Optional Kubelet Flags

    If your environment requires specific node identification, use the following flags:

    • Multiple IPs: Use --node-ip=${IP} where ${IP} is a single IP or a comma-separated list (for dual stack).
    • Manual Provider ID: If CCM cannot automatically define the VMID, use --provider-id=${ID}. The format must be proxmox://$REGION/$VMID.
    • Hostname Override: If the node hostname differs from the cluster registration, use --hostname-override=${NODENAME}.
    # Required
kubelet --cloud-provider=external

# Optional examples
kubelet --node-ip=192.168.1.10,192.168.1.11
kubelet --provider-id=proxmox://cluster-1/100
kubelet --hostname-override=my-custom-node

  8. Configure Proxmox Permissions for CCM

    main

    The CCM requires a specific role and user in Proxmox to perform audits and manage resources. Run the following commands on your Proxmox host to set up the CCM role, a dedicated user kubernetes@pve, and an API token:

    1. Create the CCM role with VM.Audit and Sys.Audit privileges.
    2. Create the kubernetes@pve user.
    3. Grant the CCM role to the user at the root (/) level.
    4. Generate an API token for the user.
    # Create role CCM
pveum role add CCM -privs "VM.Audit Sys.Audit"
# Create user and grant permissions
pveum user add kubernetes@pve
pveum aclmod / -user kubernetes@pve -role CCM
pveum user token add kubernetes@pve ccm -privsep 0

  10. Create Proxmox Authentication Token

    main

    Proxmox CCM requires a dedicated user and token with specific permissions to audit VMs and system settings. You can create these via the Proxmox CLI (pveum) or using Terraform with the bpg/proxmox provider.

    Required Permissions

    The role (named CCM) must include these privileges:

    • VM.Audit
    • VM.GuestAgent.Audit
    • Sys.Audit
    # Create role CCM
pveum role add CCM -privs "VM.Audit VM.GuestAgent.Audit Sys.Audit"

# Create user and grant permissions
pveum user add kubernetes@pve
pveum aclmod / -user kubernetes@pve -role CCM

# Create token
pveum user token add kubernetes@pve ccm -privsep 0

  11. Install Proxmox Cloud Controller Manager via Helm

    main

    To install the Proxmox CCM, use the Helm chart from the official OCI registry. You should provide a configuration file (e.g., proxmox-ccm.yaml) containing your Proxmox cluster details.

    Prerequisite: You must set --cloud-provider=external in the kubelet arguments for all nodes in your Kubernetes cluster.

    helm upgrade -i --namespace=kube-system -f proxmox-ccm.yaml \
    proxmox-cloud-controller-manager oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager

  12. Verify the Helm chart signature

    main

    Use Cosign keyless verification to ensure the Helm chart was built via the official GitHub Actions workflow. This confirms the integrity and origin of the chart.

    cosign verify ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager:0.1.5 --certificate-identity https://github.com/sergelogvinov/proxmox-cloud-controller-manager/.github/workflows/release-charts.yaml@refs/heads/main --certificate-oidc-issuer https://token.actions.githubusercontent.com