Cluster API Provider k3s

Cluster API Provider k3s provides two specialized Cluster API (CAPI) providers designed to bring up k3s clusters instead of full Kubernetes clusters:

• Cluster API Bootstrap Provider k3s (CABP3): Responsible for generating instructions (encoded as cloud-init) to transform a Machine into a k3s Kubernetes Node.
• Cluster API ControlPlane Provider k3s (CACP3): Responsible for managing the lifecycle of control plane machines specifically for k3s.

To delete a cluster created via Cluster API, you must delete the top-level Cluster resource. Do not attempt to delete the cluster by deleting the original YAML file used for creation.

kubectl delete cluster test1

To use this provider, you first need a management cluster. This guide uses kind with an extra mount to expose the local Docker socket, which is required for the Docker infrastructure provider (CAPD).

1. Create a configuration file kind-cluster-with-extramounts.yaml to expose /var/run/docker.sock.
2. Create the cluster using kind create cluster.
3. Verify connectivity with kubectl get pods -A.

cat > kind-cluster-with-extramounts.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: capi-quickstart
nodes:
- role: control-plane
  extraMounts:
    - hostPath: /var/run/docker.sock
      containerPath: /var/run/docker.sock
EOF

kind create cluster --config kind-cluster-with-extramounts.yaml

kubectl get pods -A

Once your custom kind node image is built, you can generate the cluster configuration and deploy it to your management cluster.

Important: Ensure that the airGapped field is set to true within the agentConfig section of your template to enable air-gapped mode.

Follow these steps:

1. Generate the YAML manifest using clusterctl from your template.
2. Apply the manifest using kubectl to the management cluster.

# 1. Generate the cluster yaml
clusterctl generate yaml --from ./k3s-template.yaml > k3s-airgapped.yaml

# 2. Create the cluster in the management cluster
kubectl apply -f k3s-airgapped.yaml

Once the cluster is created, you can retrieve its kubeconfig using clusterctl get kubeconfig.

Note for Docker Desktop users: The generated kubeconfig might point to an inaccessible container IP. You must update the server address to point to the host's loopback address via the exposed load balancer port.

# Get kubeconfig
clusterctl get kubeconfig test1 > workload-kubeconfig.yaml

# Fix for Docker Desktop (re-pointing to localhost via load balancer port)
sed -i -e "s/server:.*/server: https:\/\/$(docker port test1-lb 6443/tcp | sed "s/0.0.0.0/127.0.0.1/")/g" ./workload-kubeconfig.yaml

# Connect
kubectl --kubeconfig workload-kubeconfig.yaml get pods -A

You can generate a cluster definition from existing templates in the samples/ directory.

1. Use clusterctl generate cluster with the --from flag pointing to a template (e.g., samples/docker/cluster-template-quickstart.yaml).
2. Specify the k3s-compatible Kubernetes version using the +k3sX suffix (e.g., v1.30.2+k3s2).
3. Validate the generated cluster.yaml using kubectl apply --dry-run=server.
4. Apply the definition to create the cluster.

export KIND_IMAGE_VERSION=v1.30.0
clusterctl generate cluster --from samples/docker/cluster-template-quickstart.yaml test1 --kubernetes-version v1.30.2+k3s2 --worker-machine-count 2 --control-plane-machine-count 1 > cluster.yaml

# Validate
kubectl apply -f cluster.yaml --dry-run=server

# Deploy
kubectl apply -f cluster.yaml

To support air-gapped installations, you must build a custom kind node Docker image that contains the K3s binary, required images, and installation scripts. This is done by passing the K3s version, the installation script path, and the alpine/socat tarball as build arguments.

Use the following command to build the image:

export AIRGAPPED_KIND_IMAGE=kindnode:airgapped
export AIRGAPPED_INSTALL_SCRIPT_PATH=/k3s-airgapped-install.sh
export KUBERNETES_VERSION=v1.28.6+k3s2

docker build -t $AIRGAPPED_KIND_IMAGE . \
--build-arg="K3S_VERSION=$KUBERNETES_VERSION" \
--build-arg="INSTALL_SCRIPT_PATH=$AIRGAPPED_INSTALL_SCRIPT_PATH" \
--build-arg="ALPINE_SOCAT_TARBALL_PATH=./alpine-socat.tar"

You can debug CAPI K3s controllers by running them with Delve via Tilt.

1. In tilt-settings.yaml, configure the debug section for the target service (e.g., k3s-bootstrap or k3s-control-plane) with continue: true and a specific port.
2. In VS Code, use a remote attach configuration. Ensure the port in your .vscode/launch.json matches the port defined in tilt-settings.yaml.

{
    "name": "K3s Bootstrap Controller",
    "type": "go",
    "request": "attach",
    "mode": "remote",
    "remotePath": "",
    "port": 32000,
    "host": "127.0.0.1",
    "showLog": true,
    "trace": "log",
    "logOutput": "rpc"
}

To install the k3s providers, you must first configure clusterctl to know where to find the provider manifests.

1. Create a clusterctl.yaml file in $HOME/.config/cluster-api/ containing the provider names and their component URLs.
2. Run clusterctl init specifying the k3s bootstrap and control-plane providers, along with your chosen infrastructure provider (e.g., docker).
3. Wait for the provider pods to reach a running state.

# $HOME/.config/cluster-api/clusterctl.yaml
providers:
  - name: "k3s"
    url: "https://github.com/k3s-io/cluster-api-k3s/releases/v0.2.1/bootstrap-components.yaml"
    type: "BootstrapProvider"
  - name: "k3s"
    url: "https://github.com/k3s-io/cluster-api-k3s/releases/v0.2.1/control-plane-components.yaml"
    type: "ControlPlaneProvider"

clusterctl init --bootstrap k3s --control-plane k3s --infrastructure docker