repository·master·Indexed 13 days ago
https://github.com/kelseyhightower/kubernetes-the-hard-wayA manual, step-by-step tutorial for bootstrapping a Kubernetes cluster from scratch to understand its core components and underlying mechanics. This educational guide covers the installation of Kubernetes v1.32.x, containerd v2.1.x, cni v1.6.x, and etcd v3.6.x on Debian 12 (bookworm) machines.
Kubernetes The Hard Way is a learning-oriented tutorial designed to teach the fundamentals of Kubernetes by manually bootstrapping a cluster. Unlike automated tools, this guide requires performing each task individually to ensure a deep understanding of how core components fit together.
Note: The resulting cluster is intended for educational purposes and should not be considered production-ready.
The tutorial guides you through building a basic cluster consisting of one node running all control plane components and two worker nodes.
Core Component Versions:
Once files are transferred, log in to each worker node (e.g., ssh root@node-0) and perform the following provisioning steps:
socat (required for kubectl port-forward), conntrack, ipset, and kmod.swapon --show. If active, run swapoff -a./usr/local/bin or /bin and move CNI plugins to /opt/cni/bin.# Install dependencies
apt-get update && apt-get -y install socat conntrack ipset kmod
# Disable swap
swapoff -a
# Create directories
mkdir -p /etc/cni/net.d /opt/cni/bin /var/lib/kubelet /var/lib/kube-proxy /var/lib/kubernetes /var/run/kubernetes
# Install binaries
mv crictl kube-proxy kubelet runc /usr/local/bin/
mv containerd containerd-shim-runc-v2 containerd-stress /bin/
mv cni-plugins/* /opt/cni/bin/Install kubectl to the /usr/local/bin/ directory so it can be used to interact with the Kubernetes control plane once the cluster is provisioned.
After installation, verify the client version using kubectl version --client.
cp downloads/client/kubectl /usr/local/bin/
kubectl version --clientIn a manual Kubernetes setup, pods on different nodes cannot communicate because the underlying VPC network does not know how to reach the Pod CIDR ranges assigned to each node. To enable cross-node pod communication, you must manually create routes in the VPC routing table that map each node's Pod CIDR range to that node's internal IP address.
To implement this, you must:
# 1. Gather node information from machines.txt
{
SERVER_IP=$(grep server machines.txt | cut -d " " -f 1)
NODE_0_IP=$(grep node-0 machines.txt | cut -d " " -f 1)
NODE_0_SUBNET=$(grep node-0 machines.txt | cut -d " " -f 4)
NODE_1_IP=$(grep node-1 machines.txt | cut -d " " -f 1)
NODE_1_SUBNET=$(grep node-1 machines.txt | cut -d " " -f 4)
}
# 2. Configure routes on the server (control plane)
ssh root@server <<EOF
ip route add ${NODE_0_SUBNET} via ${NODE_0_IP}
ip route add ${NODE_1_SUBNET} via ${NODE_1_IP}
EOF
# 3. Configure routes on worker nodes to reach each other
ssh root@node-0 <<EOF
ip route add ${NODE_1_SUBNET} via ${NODE_1_IP}
EOF
ssh root@node-1 <<EOF
ip route add ${NODE_0_SUBNET} via ${NODE_0_IP}
EOFMove the transferred configuration files and systemd service units into their respective system locations on the worker node.
# Configure containerd
mkdir -p /etc/containerd/
mv containerd-config.toml /etc/containerd/config.toml
mv containerd.service /etc/systemd/system/
# Configure kubelet
mv kubelet-config.yaml /var/lib/kubelet/
mv kubelet.service /etc/systemd/system/
# Configure kube-proxy
mv kube-proxy-config.yaml /var/lib/kube-proxy/
mv kube-proxy.service /etc/systemd/system/After configuring all components, reload the systemd daemon, enable the services to start on boot, and start them immediately. You can verify the status of the kubelet service using systemctl.
To verify the entire cluster from the jumpbox, use kubectl with the admin kubeconfig to list the nodes.
# On the worker node: Start services
systemctl daemon-reload
systemctl enable containerd kubelet kube-proxy
systemctl start containerd kubelet kube-proxy
# Verify kubelet is active
systemctl is-active kubelet
# From the jumpbox: Verify nodes are Ready
ssh root@node-0 "kubectl get nodes --kubeconfig admin.kubeconfig"To enable encryption at rest for Kubernetes Secrets, you must first generate a high-entropy encryption key. This key is used by the encryption configuration to protect sensitive data stored in etcd.
You can generate a 32-byte key and encode it in base64 using the following command:
export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)Verify cluster observability and container interaction:
kubectl logs <pod-name> to view the stdout/stderr of a container.kubectl exec -ti <pod-name> -- <command> to run commands directly inside a running container (e.g., checking a version).# Retrieve logs
kubectl logs $POD_NAME
# Execute a command (e.g., check nginx version)
kubectl exec -ti $POD_NAME -- nginx -vThe tutorial uses a machines.txt file as a central database to store machine attributes. This file is used by automation scripts to iterate through the cluster nodes. Each line in the file must follow this schema:
IPV4_ADDRESS FQDN HOSTNAME POD_SUBNET
IPV4_ADDRESS: The machine's IP address.FQDN: The Fully Qualified Domain Name.HOSTNAME: The short hostname.POD_SUBNET: The unique IP address range (subnet) assigned to that machine for Kubernetes pods.Example machines.txt content:
XXX.XXX.XXX.XXX server.kubernetes.local server
XXX.XXX.XXX.XXX node-0.kubernetes.local node-0 10.200.0.0/24
XXX.XXX.XXX.XXX node-1.kubernetes.local node-1 10.200.1.0/24cat machines.txtAfter provisioning your machines, verify that they are running Debian 12 (bookworm) by checking the /etc/os-release file. The VERSION_ID should be 12 and the VERSION_CODENAME should be bookworm.
cat /etc/os-releaseCreate a deployment to verify that the cluster can manage workloads and schedule pods. Use kubectl create deployment to launch a container and kubectl get pods with a label selector to verify its status.
# Create an nginx deployment
kubectl create deployment nginx \
--image=nginx:latest
# List the resulting pod using a label selector
kubectl get pods -l app=nginx