terraform-hcloud-kubernetes

repository·main·Indexed 20 days ago

https://github.com/hcloud-k8s/terraform-hcloud-kubernetes

A Terraform module for deploying production-grade, highly available Kubernetes clusters on Hetzner Cloud using Talos Linux. It supports AMD64 and ARM64 architectures, Hetzner Bare Metal Server integration, and includes preconfigured components such as Cilium CNI, Hcloud CCM/CSI, Longhorn, and Cluster Autoscaler. The module provides features for dual-stack networking, immutable infrastructure, and multi-layer security including Talos Disk Encryption and Hetzner Cloud Firewalls.

Tokens
10.9K
Snippets
30
Records
38
Agent score
22%

What's inside terraform-hcloud-kubernetes

  1. Overview of Hcloud Kubernetes

    main
    Hcloud Kubernetes is a Terraform module designed to deploy a fully declarative, managed, and highly available Kubernetes cluster on Hetzner Cloud. It uses Talos Linux, a secure, immutable, and minimal operating system specifically built for Kubernetes, which reduces the attack surface by removing SSH and shell access. The module is designed for production-grade environments, supporting features like autoscaling, multi-architecture (AMD64/ARM64), and integrated bare metal server support.
  2. Key features of Hcloud Kubernetes

    main

    The module provides several production-ready features:

    • Immutable Infrastructure: Uses Talos Linux for a declarative cluster state.
    • Multi-Architecture: Supports both AMD64 and ARM64 instances.
    • Bare Metal Integration: Managed integration of Hetzner Bare Metal Servers.
    • High Availability: HA configuration for control plane and worker components.
    • Autoscaling: Supports automatic scaling of both Nodes and Pods.
    • Dual-Stack Networking: Native IPv4 and IPv6 connectivity via load balancers with PROXY Protocol support.
    • Isolated Network: Internal traffic is confined to a private Hetzner Cloud Network.
    • Security: Perimeter firewalls and encryption for data in transit and at rest.
  3. Understand Network Segmentation and CIDR calculations

    main

    The module automatically segments the network based on the provided network_ipv4_cidr.

    Segmentation Logic:

    • 1st Quarter: Reserved for other uses (e.g., classic VMs).
    • 2nd Quarter:
      • 1st Half: Allocated for Node Subnets (network_node_ipv4_cidr).
      • 2nd Half: Allocated for Service IPs (network_service_ipv4_cidr).
    • 3rd and 4th Quarters:
      • Full Span: Allocated for Pod Subnets (network_pod_ipv4_cidr).

    Key Constraints & Limits:

    • Each Kubernetes node requires a /24 subnet within network_pod_ipv4_cidr.
    • Hetzner Cloud limits: Up to 100 servers per network, 100 routes per network, and 50 subnets per network.
    • A /16 Network CIDR is recommended to fully utilize scaling capabilities (supports up to 100 nodes and 50 nodepools).
  4. Included Kubernetes components

    main

    The module bundles and preconfigures several essential components:

    • Talos Cloud Controller Manager (CCM): Automatically approves kubelet server CSRs.
    • Talos Backup: Automates etcd snapshots to S3 storage.
    • Hcloud Cloud Controller Manager (CCM): Integrates Kubernetes with Hetzner Cloud services (nodes, private networks, load balancers).
    • Hcloud Container Storage Interface (CSI): Provides persistent storage via Hetzner Cloud Volumes.
    • Longhorn: Distributed block storage for high availability and snapshots.
    • Cilium CNI: High-performance networking using eBPF for security and observability.
    • Cilium Gateway API: Implements Kubernetes Gateway API using eBPF and Envoy for L7 routing.
    • Cert Manager: Automates certificate issuance and renewal.
    • Cert Manager Webhook Hetzner: Adds Hetzner DNS support for ACME DNS-01 challenges.
    • Cluster Autoscaler: Dynamically adjusts cluster size based on demand.
    • Metrics Server: Provides resource metrics for autoscaling (HPA/VPA).
  5. Understand the Hcloud K8s Lifecycle and Versioning

    main

    The module manages the versions of Talos Linux, Kubernetes, and bundled components as a single unit. Each major version of the Hcloud K8s module targets a specific Talos and Kubernetes version to ensure compatibility.

    Hcloud Kubernetes Versions

    Hcloud K8sTalos LinuxKubernetes
    (7)(1.15)1.36
    (6)(1.14)1.35
    51.131.34
    41.121.33

    Note: Parenthesized versions are planned targets.

    Upgrade Policy

    • Major Version Changes: Any minor or major upgrade to Talos or Kubernetes results in a major version change for this module.
    • Downgrades: Generally not supported or tested.
    • Manual Changes: Changing software versions manually is not recommended. Component versions are pre-selected and tested for compatibility with the specific Kubernetes release.

    Best Practices for Upgrading

    1. Do not combine major or minor module upgrades with infrastructure changes.
    2. Upgrade to the latest minor release before moving to the next major version.
    3. Verify cluster health both before and after any upgrade.
  6. Security architecture of Hcloud Kubernetes

    main

    Security is implemented at multiple layers:

    • OS Level: Talos Linux is minimal and immutable, managed via a secure mTLS API. It follows NIST and CIS hardening standards.
    • Perimeter Security: External access is restricted using Hetzner Cloud Firewall.
    • Network Policy: Internal traffic is governed by Kubernetes Network Policies via Cilium CNI.
    • Encryption in Transit: Pod traffic is transparently encrypted by Cilium using WireGuard (default) or IPsec.
    • Encryption at Rest: The STATE and EPHEMERAL partitions are encrypted by default using Talos Disk Encryption (LUKS2), with keys derived from the unique nodeID.
  7. Create a Cert Manager ClusterIssuer for Let's Encrypt DNS-01

    main

    For wildcard certificates or environments without public HTTP routing, use the Hetzner DNS webhook.

    1. Enable the webhook in Terraform:

      • cert_manager_enabled = true
      • cert_manager_webhook_hetzner_enabled = true
    2. Create a Secret in the cert-manager namespace containing your Hetzner API token:

    apiVersion: v1
    kind: Secret
    metadata:
      name: hetzner
      namespace: cert-manager
    stringData:
      token: <hetzner-dns-token>
    1. Create the ClusterIssuer:
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: letsencrypt-dns01
    spec:
      acme:
        email: <user@example.com>
        server: https://acme-v02.api.letsencrypt.org/directory
        privateKeySecretRef:
          name: letsencrypt-dns01-key
        solvers:
          - dns01:
              webhook:
                groupName: acme.hetzner.com
                solverName: hetzner
                config:
                  tokenSecretKeyRef:
                    name: hetzner
                    key: token

    Note: When using this with the Gateway shim, use cert-manager.io/cluster-issuer: letsencrypt-dns01 instead of cert-manager.io/issuer.

  8. Test OIDC Authentication with kubelogin

    main

    Before configuring kubectl, verify that your OIDC provider returns the expected JWT tokens. Use the kubectl oidc-login setup command. This will trigger a browser-based login flow.

    Ensure the resulting JWT contains:

    • A groups array with your expected groups.
    • An email field matching your user email.
    • email_verified: true (required by Kubernetes).
    kubectl oidc-login setup \
      --oidc-issuer-url=https://your-oidc-provider.com \
      --oidc-client-id=your-client-id \
      --oidc-client-secret=your-client-secret \
      --oidc-extra-scope=openid,email,profile
  9. Configure Bare Metal Worker Nodes via Hetzner Robot

    main

    You can add bare metal worker nodes from Hetzner Robot to your cluster. The module handles enabling rescue mode, installing Talos, attaching servers to a vSwitch, and applying the Talos worker configuration.

    Each server requires a unique private_ipv4 from the vSwitch subnet. For a default 10.0.0.0/16 network, you can calculate subnets using tofu console with cidrsubnet("10.0.0.0/16", 9, 176) to get 10.0.88.0/25.

    Important:

    • If enabling bare metal for the first time, you may need to restart Cilium components to pick up routing changes:
      kubectl -n kube-system rollout restart ds/cilium ds/cilium-envoy
      kubectl -n kube-system rollout restart deploy/cilium-operator
    • Install Disk Warning: If you specify install_disk, it must be a disk ID from /dev/disk/by-id. Installing Talos is destructive; the selected disk and all other eligible install disks will be wiped.
    hcloud_robot_user     = "<robot-user>"
    hcloud_robot_password = "<robot-password>"
    
    bare_metal_nodepools = [
      {
        name = "bare-metal"
        servers = [
          { number = 1111111, private_ipv4 = "10.0.88.2" },
          { number = 2222222, private_ipv4 = "10.0.88.3" }
        ]
      }
    ]
    
    # Example with explicit install disk
    hcloud_vswitch_id = 12345
    
    bare_metal_nodepools = [
      {
        name         = "bare-metal"
        architecture = "amd64"
        servers = [
          { number = 1234567, private_ipv4 = "10.0.88.2", install_disk = "ata-Samsung_SSD_870_ABC123" }
        ]
      }
    ]
  10. Install the hcloud-k8s module

    main

    To install the module, create a kubernetes.tf file with the module "kubernetes" configuration.

    Configuration Options

    • cluster_name: Name of the cluster.
    • hcloud_token: Your Hetzner Cloud API token.
    • cluster_kubeconfig_path (optional): Path where the kubeconfig will be exported.
    • cluster_talosconfig_path (optional): Path where the talosconfig will be exported.
    • cert_manager_enabled (optional): Boolean to enable Cert Manager.
    • cilium_gateway_api_enabled (optional): Boolean to enable Cilium Gateway API.
    • control_plane_nodepools: List of control plane node configurations.
    • worker_nodepools: List of worker node configurations.

    Node Requirements

    • Control Plane: Minimum 4GB memory per node. For High-Availability (HA), use at least 3 nodes.
    • Worker: Minimum 2GB memory per node. For HA, use at least 3 nodes.
    module "kubernetes" {
      source  = "hcloud-k8s/kubernetes/hcloud"
      version = "<version>"
    
      cluster_name = "k8s"
      hcloud_token = "<hcloud-token>"
    
      # Export configs for talosctl and kubectl (optional)
      cluster_kubeconfig_path  = "kubeconfig"
      cluster_talosconfig_path = "talosconfig"
    
      # Enable Cilium Gateway API and Cert Manager (optional)
      cert_manager_enabled       = true
      cilium_gateway_api_enabled = true
    
      control_plane_nodepools = [
        { name = "control", type = "cpx22", location = "nbg1", count = 3 }
      ]
      worker_nodepools = [
        { name = "worker", type = "cpx22", location = "nbg1", count = 3 }
      ]
    }
  11. Enable Cilium Gateway API and Cert Manager

    main

    The module installs Gateway API CRDs by default and can deploy Cert Manager with Gateway API support. To enable Cilium's Gateway API implementation, set cilium_gateway_api_enabled = true in your Terraform configuration.

    If you enable this after the initial deployment, you must restart the controllers to pick up the changes:

    kubectl -n cert-manager rollout restart deployment
    kubectl -n kube-system rollout restart deployment/cilium-operator
    cilium_gateway_api_enabled = true
  12. Install kubelogin for OIDC authentication

    main

    To use OIDC authentication with kubectl, you must install the kubelogin plugin. Choose the command corresponding to your operating system and package manager.

    # Homebrew (macOS and Linux)
    brew install kubelogin
    
    # Krew (macOS, Linux, Windows and ARM)
    kubectl krew install oidc-login
    
    # Chocolatey (Windows)
    choco install kubelogin