kubeadm Documentation

If your environment does not provide a managed load balancer (e.g., from a cloud provider), you must implement user-managed software load balancing.

Common strategies include:

• Using dedicated load balancing hardware/software provided by your data center.
• Setting up a cluster of hosts to manage a Virtual IP (vIP). In this setup, each host runs a load balancer instance, and only the host currently holding the vIP is active, while others remain on standby.
• Using tools like keepalived and haproxy to manage the vIP and traffic distribution.

kubeadm provides a user experience for creating a best-practice, bare Kubernetes cluster from scratch using kubeadm init and kubeadm join.

Key characteristics of a kubeadm-bootstrapped cluster:

• Secure: Enforces RBAC, uses the Node Authorizer, and ensures secure communication between control plane components, the API Server, and kubelets. It also locks down access to system components (kube-proxy, kube-dns) and Bootstrap Tokens.
• Easy to use: Requires minimal commands to get a cluster running.
• Extendable: Does not favor a specific network provider (network configuration is out-of-scope) and provides a configuration file for customizing parameters.

Important Design Principle: Every phase of the kubeadm execution must be idempotent.

After cluster discovery, kubeadm join performs TLS Bootstrapping to allow the kubelet to authenticate securely:

1. kubeadm writes a bootstrap-kubelet.conf file.
2. The kubelet uses the shared token to temporarily authenticate with the Kubernetes Master.
3. The kubelet submits a Certificate Signing Request (CSR) for a locally created key pair.
4. The CSR is automatically approved (managed by the csrapprover controller).
5. Upon approval, ca.crt and kubelet.conf are saved for the kubelet to use, and bootstrap-kubelet.conf is deleted.

Authentication Details:

• Temporary authentication is validated against the token saved during kubeadm init or tokens created via kubeadm token.
• The temporary authentication resolves to a user in the system:bootstrappers:kubeadm:default-node-token group, which is granted CSR API access during kubeadm init.

Decide which tool to use based on your requirements for Kubernetes cluster management:

• Use kind when you need a working Kubernetes cluster for standard development, testing, or CI/CD workflows where you simply want to interact with a cluster.
• Use kinder when you need granular control over the creation process of a Kubernetes cluster from "bare machines" (running as containers). It is specifically designed for kubeadm contributors and kubeadm E2E testing, allowing for more complex configurations like external etcd, external load balancers, and custom image manipulation.

When running kubeadm join, the tool follows these phases:

1. Phase 1: Fetch the cluster-info ConfigMap: The node attempts to fetch the cluster-info ConfigMap from the kube-public namespace via GET /api/v1/namespaces/kube-public/configmaps/cluster-info. This phase is skipped if the ConfigMap is not public or if the user provides the cluster information via a local file.
2. Phase 2: TLS Bootstrap flow: The node posts a Certificate Signing Request (CSR) to the API server. Once approved and signed by the controller-manager, kubeadm writes the signed credentials to /etc/kubernetes/kubelet.conf for the kubelet to use.

When running kubeadm init, the tool executes a sequence of phases to bootstrap the control plane:

1. Phase 1: Generate certificates: Creates CA and component certificates (API Server, Kubelet client, ServiceAccount signing keys, etc.). By default, these are stored in /etc/kubernetes/pki.
2. Phase 2: Generate KubeConfig files: Creates configuration files for master components (/etc/kubernetes/admin.conf, /etc/kubernetes/kubelet.conf, /etc/kubernetes/controller-manager.conf, and /etc/kubernetes/scheduler.conf).
3. Phase 3: Bootstrap control plane via Static Pods: Spins up etcd, the API Server, Controller Manager, and Scheduler as Static Pods.
4. Phase 4+: Post-bootstrap: Performs tasks like marking the node as a master, creating the cluster-info ConfigMap, self-hosting manifests as DaemonSets, and deploying kube-proxy and kube-dns.

To build control plane docker images from the Kubernetes source tree, use the make commands within the ${GOPATH}/src/k8s.io/kubernetes directory.

cd ${GOPATH}/src/k8s.io/kubernetes

# Build docker images only for linux/amd64
make quick-release-images

# Build all docker images
make release-images

Use kinder test e2e-kubeadm to run a suite of tests that checks if kubeadm has correctly created and configured the necessary ConfigMaps, Secrets, RBAC Roles, and RoleBindings required for kubeadm join or kubeadm upgrade operations. Tests are executed against the cluster defined in your current kubeconfig file.

Supported flags:

• --kube-root: Set the folder where the Kubernetes sources are stored.
• --single-node: Instruct the Ginkgo test suite to skip tests labeled with [multi-node].
• --automatic-copy-certs: Instruct the Ginkgo test suite to skip tests labeled with [copy-certs].

As an alternative to keepalived and haproxy, kube-vip provides both Virtual IP (VIP) management and load balancing in a single service. It can be deployed as a static pod on control plane nodes using either Layer 2 (ARP with leaderElection) or Layer 3 (BGP peering).

Requirements & Constraints:

• Hosts negotiating a VIP must be in the same IP subnet.
• kube-vip requires access to the API server. During kubeadm init, it uses the admin.conf kubeconfig. Post-setup, it is recommended to use a custom client kubeconfig and rotate it manually upon expiration.
• For Kubernetes v1.29+, since admin.conf is not immediately usable during bootstrap, consider using super-admin.conf for initialization and switching to admin.conf once complete.

After initializing your cluster, use the following commands to verify the state and run tests:

• Cluster Overview: Run kinder do cluster-info to check for nodes (status: Ready), verify Kubernetes version and component dependencies, and check etcd membership.
• Smoke Test: Run kinder do smoke-test to execute a basic functional test of the cluster.

Note: You can use the --only-node flag with kinder do to execute these validation actions on a specific node. Alternatively, you can use docker exec and docker cp to run kubeadm, kubectl, or other shell commands directly on the nodes.

# get an overview of the resulting cluster
kinder do cluster-info

# run a smoke test
kinder do smoke-test