DOCSEARCHbyAlgolia
Submit Docs

Kubernetes Certificate Mechanism

website·Indexed 31 days ago

https://edgarding.cn

Technical documentation covering Kubernetes certificate types, validity periods, and rotation mechanisms. Includes PEM certificate commands, inspection methods, the structure of certificates within /etc/kubernetes/, CA certificates, etcd configurations, proxy identity, and OIDC authentication.

Tokens
9.2K
Snippets
33
Records
50
Agent score
0%

What's inside Kubernetes

  1. Overview of Kubernetes and etcd certificates

    A standard Kubernetes cluster utilizes several sets of certificates to secure communication between components:

    etcd Certificates:

    • etcd server: For etcd to provide services.
    • etcd peer: For communication between etcd nodes.
    • etcd client: For kube-apiserver to access etcd.

    Kubernetes Certificates:

    • kube-apiserver server: For the API server to provide services.
    • kube-apiserver client: For components (scheduler, controller-manager, proxy, kubelet) to access the API server.
    • Service Account CA: Used by the controller-manager to sign service account certificates.
    • kubelet server: For the kubelet to provide services.
    • kubelet client: For the API server to access the kubelet.

  2. Overview of Kubernetes component certificate roles

    Kubernetes employs a mutual TLS (mTLS) architecture where service providers use server certificates and service consumers provide client certificates, both validated by a shared CA certificate. Key certificate roles include:

    1. etcd Peer: Used for communication between etcd nodes (acts as both server and client certificate).
    2. etcd Server: Used by etcd to provide services to external clients.
    3. API Server to etcd: Client certificate used by kube-apiserver to access etcd.
    4. API Server Server: Server certificate used by kube-apiserver to provide services.
    5. Controller Manager to API Server: Client certificate for kube-controller-manager.
    6. Scheduler to API Server: Client certificate for kube-scheduler.
    7. Kube-proxy to API Server: Client certificate for kube-proxy.
    8. Kubelet to API Server: Client certificate for kubelet.
    9. Admin to API Server: Client certificate for kubectl administrators.
    10. Kubelet Server: Server certificate used by kubelet to provide services.
    11. API Server to Kubelet: Client certificate used by kube-apiserver to access kubelets.
    12. Service Account Token Signing: Used by kube-controller-manager to sign and verify service account tokens (private key signs, API server public key verifies).

  3. Understand Kubernetes certificate terminology

    Key concepts used in the Kubernetes PKI infrastructure:

    • CA (Certificate Authority): The root entity that signs certificates to prove their legitimacy. The ca.crt is distributed to clients and automatically mounted into Pods at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.
    • Key: Refers to either the Public Key or Private Key.
    • CSR (Certificate Signing Request): A request sent to a CA to obtain a signed certificate, containing the public key.
    • CRT/CER (Certificate): The signed certificate, typically in PEM format.

  4. Understand Kubernetes TLS Authentication

    Kubernetes uses mutual TLS (mTLS) for communication between components. Unlike single-sided TLS where only the client verifies the server, mTLS requires both the client and server to provide and verify certificates to ensure identity and secure communication. Certificates are categorized into server certificates, client certificates, and peer certificates.

  5. Understand Pod authentication via Service Accounts

    Pods authenticate to the API server using Service Accounts (SA). Each namespace has a default SA. The necessary credentials—namespace, token, and CA certificate—are stored in a Secret and mounted into the pod's filesystem at /var/run/secrets/kubernetes.io/serviceaccount/. The CA is used by the pod to verify the API server's certificate, and the token is used for identity authentication.

  6. Renew Kubernetes certificates using kubeadm

    The kubeadm certs renew command allows for the manual renewal of certificates using the CA (or front-proxy-CA) and keys stored in /etc/kubernetes/pki.

    Critical Requirement: After renewing certificates, you must restart the control plane components (kube-apiserver, kube-controller-manager, kube-scheduler, and etcd) because these components do not all support dynamic certificate reloading.

    In High Availability (HA) clusters, this command must be executed on all control-plane nodes.

    # Renew all available certificates
kubeadm certs renew all

# Renew a specific certificate (e.g., apiserver)
kubeadm certs renew apiserver

  7. Manage Kubernetes certificates with kubeadm

    In kubeadm-managed clusters, control plane certificates are automatically renewed during version upgrades using kubeadm upgrade apply or kubeadm upgrade node.

    Key management options include:

    • Disable automatic renewal: Pass the -certificate-renewal=false flag during the upgrade command.
    • Manual renewal: Use kubeadm certs renew to renew certificates at any time. By default, certificates are renewed for one year.
    • Certificate Storage: CA and keys used for renewal are stored in /etc/kubernetes/pki.
    • Post-renewal Action: Control plane components must be restarted after renewal because not all components support dynamic reloading. Since Static Pods are managed by the local kubelet, they cannot be restarted via kubectl. To restart a static pod, temporarily remove the manifest from /etc/kubernetes/manifests/, wait approximately 20 seconds, and then move it back.
    # Manual renewal of certificates
kubeadm certs renew

# Upgrade without automatic certificate renewal
kubeadm upgrade apply -certificate-renewal=false

  8. Understand how kube-proxy authenticates with the API server

    kube-proxy does not use a traditional kubeconfig file. Instead, it uses a ServiceAccount token for authentication. This is a one-way TLS authentication where the kube-proxy uses a CA certificate to verify the API server, and the API server uses the token to verify the kube-proxy. The CA certificate is provided via a ConfigMap (typically kube-root-ca.crt) and the token is mounted as a secret into the pod via a projected volume.
    # Inspect the ConfigMap containing the CA certificate used by kube-proxy
kubectl get configmap -n kube-system kube-root-ca.crt -o yaml

# Inspect the ServiceAccount token secret used by kube-proxy
kubectl get secrets -n kube-system <kube-proxy-token-name> -o yaml

  9. Understand ServiceAccount Token Authentication

    Kubernetes uses JWT tokens for ServiceAccount authentication. This is a one-way TLS process where the Pod verifies the API server using a CA certificate, but the API server verifies the Pod using a JWT token signed by a private key.

    • Signing: The kube-controller-manager uses a private key (sa.key) to sign the token.
    • Verification: The kube-apiserver uses the corresponding public key (sa.pub) to verify the token.
    • Pod Access: Tokens are mounted into Pods at /var/run/secrets/kubernetes.io/serviceaccount/token and the CA is at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.
    # kube-apiserver: Public key for verification
--service-account-key-file=/etc/kubernetes/pki/sa.pub

# kube-controller-manager: Private key for signing
--service-account-private-key-file=/etc/kubernetes/pki/sa.key

  10. Manually rotate Kubernetes certificates and kubeconfigs

    When performing a full manual rotation of certificates and their associated kubeconfig files, follow these steps to ensure backup and consistency across the control plane.
    # 1. Backup old certs and kubeconfigs
mkdir /etc/kubernetes.bak
cp -r /etc/kubernetes/pki/ /etc/kubernetes.bak
cp /etc/kubernetes/*.conf /etc/kubernetes.bak

# 2. Renew all certs
kubeadm certs renew all --config kubeadm.yaml

# 3. Renew all kubeconfigs
# Option A: Individual user renewal
kubeadm kubeconfig user --client-name=admin
kubeadm kubeconfig user --org system:masters --client-name kubernetes-admin > /etc/kubernetes/admin.conf
kubeadm kubeconfig user --client-name system:kube-controller-manager > /etc/kubernetes/controller-manager.conf
kubeadm kubeconfig user --org system:nodes --client-name system:node:$(hostname) > /etc/kubernetes/kubelet.conf
kubeadm kubeconfig user --client-name system:kube-scheduler > /etc/kubernetes/scheduler.conf

# Option B: Bulk renewal of all run configurations
kubeadm init phase kubeconfig all --config kubeadm.yaml

# 4. Update local kubeconfig for the admin user
mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# 5. Restart kube-apiserver, kube-controller-manager, kube-scheduler, and etcd

  11. Inspect Kubernetes X509 Certificates

    Kubernetes certificates follow the X509 digital certificate standard and are typically stored with .crt (certificate) and .key (private key) extensions. They are often in PEM (Privacy Enhanced Mail) format, which is a Base64 encoded text format.
    # View certificate details using OpenSSL
openssl x509 -in xxx.crt -text -noout

# View certificate info using cfssl
csfssl certinfo -cert xxx.crt

  12. Automate Kubelet client certificate generation using TLS Bootstrapping

    TLS Bootstrapping simplifies the process of issuing client certificates to Kubelets in large clusters. Instead of manually generating certificates for every node, the Kubelet uses a low-privilege bootstrap token to request a signed certificate from the kube-apiserver.

    Prerequisites:

    • The kube-apiserver must be started with the --enable-bootstrap-token-auth flag.
    • A bootstrap token secret must be created (e.g., using kubeadm token create).

    Workflow:

    1. A bootstrap token is generated and placed in a kubeconfig file.
    2. The Kubelet is started with the -bootstrap-kubeconfig flag pointing to this file.
    3. Kubelet uses the token to call the kube-apiserver's Certificate Signing Request (CSR) API to obtain its own server and client certificates.
    4. Once certificates are generated, Kubelet switches to using them for communication and deletes the bootstrap kubeconfig file to prevent token leakage.