Kamaji Cluster API Control Plane Provider

When using Kamaji with Equinix (Packet), you must annotate your PacketCluster resource to indicate it is managed by Kamaji.

Why this is required: The CAPI Packet provider normally creates a 'VIP Manager' service for Control Plane VMs. However, since Kamaji provides the Control Plane as a service and manages the Load Balancer from the management cluster, the annotation prevents conflicts.

Effect: By adding the cluster.x-k8s.io/managed-by: kamaji annotation, the Kamaji Control Plane provider will wait for the Control Plane address to become reachable, then patch the PacketCluster with the correct address and mark the infrastructure as ready.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: PacketCluster
metadata:
  annotations:
    cluster.x-k8s.io/managed-by: kamaji
  name: capi-quickstart
  namespace: default
spec:
  metro: REDACTED
  projectID: REDACTED

To create a Kubernetes cluster backed by Proxmox using the Kamaji Control Plane provider, you must define a set of interconnected Cluster API resources: a Cluster, a KamajiControlPlane, a ProxmoxCluster (infrastructure), a MachineDeployment, a ProxmoxMachineTemplate, and a KubeadmConfigTemplate.

Key Requirements:

• The ProxmoxCluster.spec.externalManagedControlPlane field must be set to true.
• The KamajiControlPlane manages the control plane components (like CoreDNS and kube-proxy) via Kamaji.
• The ProxmoxMachineTemplate defines the VM specifications (disks, memory, CPU, network) for the worker nodes.
• The KubeadmConfigTemplate must include a provider-id in kubeletExtraArgs using the pattern proxmox://'{{ ds.meta_data.instance_id }}' to ensure correct node identification.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: proxmox-quickstart
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - REDACTED/REDACTED
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha2
    kind: KamajiControlPlane
    name: proxmox-quickstart
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: ProxmoxCluster
    name: proxmox-quickstart
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha2
kind: KamajiControlPlane
metadata:
  name: proxmox-quickstart
  namespace: default
spec:
  dataStoreName: default
  addons:
    coreDNS: { }
    kubeProxy: { }
  kubelet:
    cgroupfs: systemd
    preferredAddressTypes:
    - InternalIP
  network:
    serviceType: LoadBalancer
  deployment:
  replicas: 2
  version: 1.29.7
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxCluster
metadata:
  name: proxmox-quickstart
  namespace: default
spec:
  allowedNodes:
  - pve
  dnsServers:
  - REDACTED
  - REDACTED
  externalManagedControlPlane: true
  ipv4Config:
    addresses:
    - REDACTED-REDACTED
    gateway: REDACTED
    prefix: REDACTED
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: proxmox-quickstart-workers
  namespace: default
spec:
  clusterName: proxmox-quickstart
  replicas: 2
  selector:
    matchLabels: null
  template:
    metadata:
      labels:
        node-role.kubernetes.io/node: ""
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: proxmox-quickstart-worker
      clusterName: proxmox-quickstart
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
        kind: ProxmoxMachineTemplate
        name: proxmox-quickstart-worker
      version: v1.29.7
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxMachineTemplate
metadata:
  name: proxmox-quickstart-worker
  namespace: default
spec:
  template:
    spec:
      disks:
        bootVolume:
          disk: scsi0
          sizeGb: REDACTED
      format: qcow2
      full: true
      memoryMiB: REDACTED
      network:
        default:
          bridge: REDACTED
          model: virtio
      numCores: REDACTED
      numSockets: REDACTED
      sourceNode: pve
      templateID: REDACTED
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: proxmox-quickstart-worker
  namespace: default
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            provider-id: proxmox://'{{ ds.meta_data.instance_id }}'
      users:
      - name: root
        sshAuthorizedKeys:
        - REDACTED

Because Kamaji architecture does not include physical Control Plane nodes, you cannot run pods directly on them. To ensure the Nutanix Cloud Controller Manager (CCM) runs correctly, you must customize its installation to remove the nodeSelector, allowing it to run on worker nodes instead.

Using HelmChartProxy

Set nodeSelector: '' in the valuesTemplate of your HelmChartProxy resource:

apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: nutanix-cloud-provider
spec:
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: capi-quickstart
  releaseName: nutanix-cloud-provider
  repoURL: https://nutanix.github.io/helm/
  chartName: nutanix-cloud-provider
  namespace: kube-system
  valuesTemplate: |
    prismCentralEndPoint: XXX
    username: XXX
    password: 'XXX'
    nodeSelector: ''

Using Helm directly

If installing manually via Helm, use the --set flag to clear the nodeSelector:

helm install nutanix-ccm nutanix/nutanix-cloud-provider -n kube-system --set prismCentralEndPoint=xxx,username=xxx,password='xxx',nodeSelector=''

To create a Kubernetes cluster using Kamaji as the control plane provider and metal-stack as the infrastructure provider, you must define several interconnected resources: a Cluster, a KamajiControlPlane, a MetalStackCluster, and associated infrastructure components (Firewalls, Machine Templates, and Machine Deployments).

Prerequisites

• A Service Load Balancer (e.g., MetalLB, Kube-VIP) must be installed in your management cluster to provide a Virtual IP (CONTROL_PLANE_IP) for the Kamaji Tenant API server.
• For a fully operational cluster (including pod scheduling), you must deploy a CNI-Plugin (like Calico) and metal-ccm into the tenant cluster after creation.

Configuration Requirements

To ensure connectivity, the following fields must be synchronized:

1. API Server Address: KamajiControlPlane.spec.network.serviceAddress must match MetalStackCluster.spec.controlPlaneEndpoint.host.
2. API Server Port: Cluster.spec.clusterNetwork.apiServerPort must match MetalStackCluster.spec.controlPlaneEndpoint.port.
3. Firewall Egress: The MetalStackFirewallTemplate must allow egress traffic to:
   • The Kamaji Tenant API Server on port 6443 (for kubeadm join and kubelet).
   • The Konnectivity Server on port 8132 (for control-plane-to-worker tunnels).

# Example snippet of the required synchronization
# KamajiControlPlane
spec:
  network:
    serviceAddress: ${CONTROL_PLANE_IP}

# MetalStackCluster
spec:
  controlPlaneEndpoint:
    host: ${CONTROL_PLANE_IP}
    port: 6443

To create a Kubernetes cluster using Kamaji as the control plane provider and Nutanix as the infrastructure provider, you must define a set of interconnected Cluster API resources.

Prerequisites:

• A Service Load Balancer (e.g., MetalLB, Kube-VIP) installed in your management cluster.
• CAAPH (Cluster API Addon Provider Helm) installed in your management cluster.

Required Resources:

1. Cluster: The root resource linking the KamajiControlPlane and NutanixCluster.
2. NutanixCluster: Defines the infrastructure for the control plane endpoint.
3. KamajiControlPlane: Configures the Kamaji-managed control plane (e.g., version, addons, network settings).
4. NutanixMachineTemplate: Defines the worker node specifications (image, memory, CPU, etc.).
5. KubeadmConfigTemplate: Provides bootstrap configuration for workers.
6. MachineDeployment: Manages the lifecycle of worker machines.
7. HelmChartProxy: (Optional) Used to deploy addons like Cilium or the Nutanix Cloud Provider via Helm.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: capi-quickstart
  name: capi-quickstart
spec:
  clusterNetwork:
    apiServerPort: 6443
    pods:
      cidrBlocks:
        - 192.168.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KamajiControlPlane
    name: kamaji-nutanix-127
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: NutanixCluster
    name: capi-quickstart
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: NutanixCluster
metadata:
  name: capi-quickstart
spec:
  controlPlaneEndpoint:
    host: 0.0.0.0 # will be automatically patch by Kamaji controller
    port: 0 # will be automatically patch by Kamaji controller
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha2
kind: KamajiControlPlane
metadata:
  name: kamaji-nutanix-127
  namespace: default
spec:
  controllerManager:
    extraArgs:
      - --cloud-provider=external
  dataStoreName: default
  addons:
    coreDNS: { }
    kubeProxy: { }
  kubelet:
    cgroupfs: systemd
    preferredAddressTypes:
      - ExternalIP
      - InternalIP
      - Hostname
  network:
    # serviceAddress: 10.83.1.2 # can be statically assigned
    serviceType: LoadBalancer
  deployment:
  replicas: 2
  version: 1.27.8

When deploying the Kamaji Control Plane on a management cluster that is itself running on Hetzner Cloud, you must ensure the resulting Service object is correctly annotated for the Hetzner Load Balancer.

In your KamajiControlPlane resource, configure the network.serviceAnnotations to include the Hetzner location annotation. The specific region (e.g., fsn1) should match your deployed region.

Key Configuration Steps:

1. Set network.serviceType to LoadBalancer.
2. Add the annotation load-balancer.hetzner.cloud/location=<your-region> to network.serviceAnnotations.
3. If you are managing the Load Balancer manually (offloading Kamaji from creating it), set controlPlaneLoadBalancer.enabled: false in your HetznerCluster resource.

apiVersion: controlplane.cluster.x-k8s.io/v1alpha2
kind: KamajiControlPlane
metadata:
  name: workload-control-plane
  namespace: default
spec:
  # ... other spec fields ...
  network:
    serviceType: LoadBalancer
    serviceAnnotations:
      load-balancer.hetzner.cloud/location: fsn1

You can use the Kamaji Control Plane provider to create Kubernetes clusters where the control plane is managed by Kamaji and the worker nodes are KubeVirt virtual machines.

To achieve this, you must coordinate three main components:

1. A Cluster resource that references both a KamajiControlPlane and a KubevirtCluster.
2. A KamajiControlPlane resource to manage the control plane components.
3. A KubevirtCluster resource for the infrastructure.

Note that when using Kamaji with KubeVirt, you must mark the KubevirtCluster as externally managed to prevent the KubeVirt provider from attempting to manage Load Balancer services that Kamaji will instead handle.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: capi-quickstart
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 10.243.0.0/16
    services:
      cidrBlocks:
      - 10.95.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha2
    kind: KamajiControlPlane
    name: capi-quickstart-kubevirt
    namespace: default
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: KubevirtCluster
    name: capi-quickstart
    namespace: default