Cluster API Provider for Proxmox VE (CAPMOX)

In v1alpha2, IP address assignment uses network.networkDevices and ipPoolRef.

Important: Pool references are additive.

• defaultIPv4 or defaultIPv6 (set to true) attaches addresses from the selected cluster zone (or the default zone if network.zone is not specified).
• networkDevices[].ipPoolRef adds additional addresses from the listed pools.

If both are configured on the same NIC, that NIC will receive multiple addresses (one for each configured pool reference).

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: ProxmoxMachineTemplate
metadata:
  name: "test-control-plane"
spec:
  template:
    spec:
      network:
        networkDevices:
          - name: net0
            bridge: ${BRIDGE}
            defaultIPv4: true
            ipPoolRef:
              - apiGroup: ipam.cluster.x-k8s.io
                kind: GlobalInClusterIPPool
                name: shared-inclusterippool

Upgrading to v0.8 introduces several breaking changes that require manual updates to your YAML manifests, Helm charts, or GitOps templates. While existing resources in the cluster are converted automatically via a webhook, your source files must be updated.

Key Breaking Changes

CAPMOX provides several templates (flavors) for different use cases. Some flavors, like cilium and calico, require creating ConfigMaps containing ClusterResourceSet (CRS) files before deployment to enable CNI.

| Flavor              | Template File                                | CRS File                                |
|---------------------|----------------------------------------------|-----------------------------------------|
| cilium              | templates/cluster-template-cilium.yaml       | templates/crs/cni/cilium.yaml           |
| calico              | templates/cluster-template-calico.yaml     | templates/crs/cni/calico.yaml           |
| multiple-vlans      | templates/cluster-template-multiple-vlans.yaml | -                                       |
| default             | templates/cluster-template.yaml             | -                                       |
| cilium loadbalancer | templates/cluster-template-cilium-load-balancer.yaml | templates/crs/cni/cilium.yaml, templates/crs/metallb.yaml |
| external-creds      | templates/cluster-template-external-creds.yaml | -                                       |
| flatcar             | templates/cluster-template-flatcar.yaml     | -                                       |

To develop CAPMOX using Tilt, you must set up a specific directory structure containing the Cluster API ecosystem repositories. Tilt manages the orchestration of these components.

Prerequisites

• Required Tools: git, make, Go, Docker, Tilt, kubectl, kind, clusterctl.
• Proxmox Access: A running Proxmox VE instance and an API token.

Setup Steps

1. Clone Repositories: Create a parent directory and clone the following into it:

   • cluster-api (This will be your working directory for Tilt)
   • cluster-api-ipam-provider-in-cluster
   • cluster-api-provider-proxmox

2. Configure Tilt: Inside the cluster-api directory, create a tilt-settings.json file:

   {
     "default_registry": "ghcr.io/ionos-cloud",
     "provider_repos": ["../cluster-api-provider-proxmox/", "../cluster-api-ipam-provider-in-cluster/"],
     "enable_providers": ["ipam-in-cluster", "proxmox", "kubeadm-bootstrap", "kubeadm-control-plane"],
     "allowed_contexts": ["minikube"],
     "kustomize_substitutions": {},
     "extra_args": {
       "proxmox": ["--v=4"]
     },
     "debug": {
       "proxmox": {
         "port": 30000
       }
     }
   }

   Note: The debug section enables a Delve remote debugger on port 30000. To use it, run: kubectl port-forward deployments/capmox-controller-manager 30000:30000 -n capmox-system.

3. Configure Proxmox Environment Variables:

   • Copy ../cluster-api-provider-proxmox/envfile.example to ../cluster-api-provider-proxmox/envfile.
   • Populate it with your Proxmox credentials (refer to Usage.md#environment-variables).
   • If using a specific kind cluster name, add CAPI_KIND_CLUSTER_NAME=<yourclustername> to this file.

4. Launch: From the cluster-api directory, run:

   . ../cluster-api-provider-proxmox/envfile && tilt up

The cilium-load-balancer flavor provides specialized nodes for load balancing. These nodes feature an extra network card connected to BGP receiving switches. All services of type LoadBalancer will use IPs from METALLB_IPV4_RANGE or METALLB_IPV6_RANGE, which are announced via BGP.

Technical Requirements:

• Routing: Uses source IP-based routing. This requires all nodes to be created with ipvs in kube-proxy and strictARP enabled to prevent reverse path filter issues.
• Immutability: If you need to change METALLB_IPV{4,6}_RANGE after deployment, you must redeploy the load balancer nodes because these variables are used during bootstrap for routing.
• Node Behavior: LoadBalancer nodes are tainted to ensure only load-balancing pods run on them.

# Example environment variables for BGP LoadBalancer setup
export LOAD_BALANCER_MACHINE_COUNT=2
export EXT_SERVICE_BRIDGE="vmbr2"
export LB_BGP_IPV4_RANGES="[172.16.4.10-172.16.4.20]"
export LB_BGP_IPV4_PREFIX="24"
export METALLB_IPV4_ASN="65400"
export METALLB_IPV4_BGP_PEER="172.16.4.1"
export METALLB_IPV4_BGP_PEER2="172.16.4.2"
export METALLB_IPV4_BGP_PEER_ASN="65500"
export METALLB_IPV4_RANGE="7.6.5.0/24"

# Generate the cluster
clusterctl generate cluster test-bgp-lb  \\
  --infrastructure proxmox \\
  --kubernetes-version v1.34.8 \\
  --control-plane-machine-count=1 \\
  --worker-machine-count=2 \\
  --flavor=cilium-load-balancer > cluster.yaml

When upgrading to Cluster API (CAPI) v1beta2, two structural changes are required in your KubeadmControlPlane or KubeadmConfigTemplate resources:

1. kubeletExtraArgs: This field has changed from a flat map to a list of name/value pairs. This applies to initConfiguration, joinConfiguration, and other locations where it appears.
2. infrastructureRef: This field has moved one level deeper, under machineTemplate.spec. Additionally, the apiVersion field is replaced by apiGroup in all resource references.

Example Changes

kubeletExtraArgs:

# v1beta1
kubeletExtraArgs:
  provider-id: "proxmox://..."

# v1beta2
kubeletExtraArgs:
- name: provider-id
  value: "proxmox://..."

infrastructureRef:

# v1beta1
spec:
  machineTemplate:
    infrastructureRef:
      kind: ProxmoxMachineTemplate
      apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
      name: my-cluster-control-plane

# v1beta2
spec:
  machineTemplate:
    spec:
      infrastructureRef:
        kind: ProxmoxMachineTemplate
        apiGroup: infrastructure.cluster.x-k8s.io
        name: my-cluster-control-plane

To deploy a Kubernetes cluster using the Cluster API Provider for Proxmox VE (CAPMOX), you must satisfy the following requirements:

• Proxmox VE Template: Required for cluster creation. You can build these using:
  • Proxmox VE builder (recommended).
  • image-builder.
  • Building Raw Images.
• clusterctl: The Cluster API CLI, available via CAPI releases.
• Management Cluster: A running Kubernetes cluster to host the CAPMOX controller.
• Proxmox VE Bridge: A configured bridge (e.g., vmbr0) with an available IP range for the VMs.
• IPAM Provider: The cluster-api provider IPAM in-cluster is required to manage IPv4 and/or IPv6 addresses for machines without DHCP and to support dual-stack setups.