terraform-hcloud-kubernetes
repository·main·Indexed 20 days ago
https://github.com/hcloud-k8s/terraform-hcloud-kubernetesA Terraform module for deploying production-grade, highly available Kubernetes clusters on Hetzner Cloud using Talos Linux. It supports AMD64 and ARM64 architectures, Hetzner Bare Metal Server integration, and includes preconfigured components such as Cilium CNI, Hcloud CCM/CSI, Longhorn, and Cluster Autoscaler. The module provides features for dual-stack networking, immutable infrastructure, and multi-layer security including Talos Disk Encryption and Hetzner Cloud Firewalls.
What's inside terraform-hcloud-kubernetes
- Hcloud Kubernetes is a Terraform module designed to deploy a fully declarative, managed, and highly available Kubernetes cluster on Hetzner Cloud. It uses Talos Linux, a secure, immutable, and minimal operating system specifically built for Kubernetes, which reduces the attack surface by removing SSH and shell access. The module is designed for production-grade environments, supporting features like autoscaling, multi-architecture (AMD64/ARM64), and integrated bare metal server support.
Key features of Hcloud Kubernetes
mainThe module provides several production-ready features:
- Immutable Infrastructure: Uses Talos Linux for a declarative cluster state.
- Multi-Architecture: Supports both AMD64 and ARM64 instances.
- Bare Metal Integration: Managed integration of Hetzner Bare Metal Servers.
- High Availability: HA configuration for control plane and worker components.
- Autoscaling: Supports automatic scaling of both Nodes and Pods.
- Dual-Stack Networking: Native IPv4 and IPv6 connectivity via load balancers with PROXY Protocol support.
- Isolated Network: Internal traffic is confined to a private Hetzner Cloud Network.
- Security: Perimeter firewalls and encryption for data in transit and at rest.
Understand Network Segmentation and CIDR calculations
mainThe module automatically segments the network based on the provided
network_ipv4_cidr.Segmentation Logic:
- 1st Quarter: Reserved for other uses (e.g., classic VMs).
- 2nd Quarter:
- 1st Half: Allocated for Node Subnets (
network_node_ipv4_cidr). - 2nd Half: Allocated for Service IPs (
network_service_ipv4_cidr).
- 1st Half: Allocated for Node Subnets (
- 3rd and 4th Quarters:
- Full Span: Allocated for Pod Subnets (
network_pod_ipv4_cidr).
- Full Span: Allocated for Pod Subnets (
Key Constraints & Limits:
- Each Kubernetes node requires a
/24subnet withinnetwork_pod_ipv4_cidr. - Hetzner Cloud limits: Up to 100 servers per network, 100 routes per network, and 50 subnets per network.
- A
/16Network CIDR is recommended to fully utilize scaling capabilities (supports up to 100 nodes and 50 nodepools).
Included Kubernetes components
mainThe module bundles and preconfigures several essential components:
- Talos Cloud Controller Manager (CCM): Automatically approves kubelet server CSRs.
- Talos Backup: Automates etcd snapshots to S3 storage.
- Hcloud Cloud Controller Manager (CCM): Integrates Kubernetes with Hetzner Cloud services (nodes, private networks, load balancers).
- Hcloud Container Storage Interface (CSI): Provides persistent storage via Hetzner Cloud Volumes.
- Longhorn: Distributed block storage for high availability and snapshots.
- Cilium CNI: High-performance networking using eBPF for security and observability.
- Cilium Gateway API: Implements Kubernetes Gateway API using eBPF and Envoy for L7 routing.
- Cert Manager: Automates certificate issuance and renewal.
- Cert Manager Webhook Hetzner: Adds Hetzner DNS support for ACME DNS-01 challenges.
- Cluster Autoscaler: Dynamically adjusts cluster size based on demand.
- Metrics Server: Provides resource metrics for autoscaling (HPA/VPA).
Understand the Hcloud K8s Lifecycle and Versioning
mainThe module manages the versions of Talos Linux, Kubernetes, and bundled components as a single unit. Each major version of the Hcloud K8s module targets a specific Talos and Kubernetes version to ensure compatibility.
Hcloud Kubernetes Versions
Hcloud K8s Talos Linux Kubernetes (7) (1.15) 1.36 (6) (1.14) 1.35 5 1.13 1.34 4 1.12 1.33 Note: Parenthesized versions are planned targets.
Upgrade Policy
- Major Version Changes: Any minor or major upgrade to Talos or Kubernetes results in a major version change for this module.
- Downgrades: Generally not supported or tested.
- Manual Changes: Changing software versions manually is not recommended. Component versions are pre-selected and tested for compatibility with the specific Kubernetes release.
Best Practices for Upgrading
- Do not combine major or minor module upgrades with infrastructure changes.
- Upgrade to the latest minor release before moving to the next major version.
- Verify cluster health both before and after any upgrade.
Security architecture of Hcloud Kubernetes
mainSecurity is implemented at multiple layers:
- OS Level: Talos Linux is minimal and immutable, managed via a secure mTLS API. It follows NIST and CIS hardening standards.
- Perimeter Security: External access is restricted using Hetzner Cloud Firewall.
- Network Policy: Internal traffic is governed by Kubernetes Network Policies via Cilium CNI.
- Encryption in Transit: Pod traffic is transparently encrypted by Cilium using WireGuard (default) or IPsec.
- Encryption at Rest: The
STATEandEPHEMERALpartitions are encrypted by default using Talos Disk Encryption (LUKS2), with keys derived from the uniquenodeID.
Create a Cert Manager ClusterIssuer for Let's Encrypt DNS-01
mainFor wildcard certificates or environments without public HTTP routing, use the Hetzner DNS webhook.
Enable the webhook in Terraform:
cert_manager_enabled = truecert_manager_webhook_hetzner_enabled = true
Create a Secret in the
cert-managernamespace containing your Hetzner API token:
apiVersion: v1 kind: Secret metadata: name: hetzner namespace: cert-manager stringData: token: <hetzner-dns-token>- Create the ClusterIssuer:
apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-dns01 spec: acme: email: <user@example.com> server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-dns01-key solvers: - dns01: webhook: groupName: acme.hetzner.com solverName: hetzner config: tokenSecretKeyRef: name: hetzner key: tokenNote: When using this with the Gateway shim, use
cert-manager.io/cluster-issuer: letsencrypt-dns01instead ofcert-manager.io/issuer.Test OIDC Authentication with kubelogin
mainBefore configuring
kubectl, verify that your OIDC provider returns the expected JWT tokens. Use thekubectl oidc-login setupcommand. This will trigger a browser-based login flow.Ensure the resulting JWT contains:
- A
groupsarray with your expected groups. - An
emailfield matching your user email. email_verified: true(required by Kubernetes).
kubectl oidc-login setup \ --oidc-issuer-url=https://your-oidc-provider.com \ --oidc-client-id=your-client-id \ --oidc-client-secret=your-client-secret \ --oidc-extra-scope=openid,email,profile- A
Configure Bare Metal Worker Nodes via Hetzner Robot
mainYou can add bare metal worker nodes from Hetzner Robot to your cluster. The module handles enabling rescue mode, installing Talos, attaching servers to a vSwitch, and applying the Talos worker configuration.
Each server requires a unique
private_ipv4from the vSwitch subnet. For a default10.0.0.0/16network, you can calculate subnets usingtofu consolewithcidrsubnet("10.0.0.0/16", 9, 176)to get10.0.88.0/25.Important:
- If enabling bare metal for the first time, you may need to restart Cilium components to pick up routing changes:
kubectl -n kube-system rollout restart ds/cilium ds/cilium-envoy kubectl -n kube-system rollout restart deploy/cilium-operator - Install Disk Warning: If you specify
install_disk, it must be a disk ID from/dev/disk/by-id. Installing Talos is destructive; the selected disk and all other eligible install disks will be wiped.
hcloud_robot_user = "<robot-user>" hcloud_robot_password = "<robot-password>" bare_metal_nodepools = [ { name = "bare-metal" servers = [ { number = 1111111, private_ipv4 = "10.0.88.2" }, { number = 2222222, private_ipv4 = "10.0.88.3" } ] } ] # Example with explicit install disk hcloud_vswitch_id = 12345 bare_metal_nodepools = [ { name = "bare-metal" architecture = "amd64" servers = [ { number = 1234567, private_ipv4 = "10.0.88.2", install_disk = "ata-Samsung_SSD_870_ABC123" } ] } ]- If enabling bare metal for the first time, you may need to restart Cilium components to pick up routing changes:
Install the hcloud-k8s module
mainTo install the module, create a
kubernetes.tffile with themodule "kubernetes"configuration.Configuration Options
cluster_name: Name of the cluster.hcloud_token: Your Hetzner Cloud API token.cluster_kubeconfig_path(optional): Path where thekubeconfigwill be exported.cluster_talosconfig_path(optional): Path where thetalosconfigwill be exported.cert_manager_enabled(optional): Boolean to enable Cert Manager.cilium_gateway_api_enabled(optional): Boolean to enable Cilium Gateway API.control_plane_nodepools: List of control plane node configurations.worker_nodepools: List of worker node configurations.
Node Requirements
- Control Plane: Minimum 4GB memory per node. For High-Availability (HA), use at least 3 nodes.
- Worker: Minimum 2GB memory per node. For HA, use at least 3 nodes.
module "kubernetes" { source = "hcloud-k8s/kubernetes/hcloud" version = "<version>" cluster_name = "k8s" hcloud_token = "<hcloud-token>" # Export configs for talosctl and kubectl (optional) cluster_kubeconfig_path = "kubeconfig" cluster_talosconfig_path = "talosconfig" # Enable Cilium Gateway API and Cert Manager (optional) cert_manager_enabled = true cilium_gateway_api_enabled = true control_plane_nodepools = [ { name = "control", type = "cpx22", location = "nbg1", count = 3 } ] worker_nodepools = [ { name = "worker", type = "cpx22", location = "nbg1", count = 3 } ] }Enable Cilium Gateway API and Cert Manager
mainThe module installs Gateway API CRDs by default and can deploy Cert Manager with Gateway API support. To enable Cilium's Gateway API implementation, set
cilium_gateway_api_enabled = truein your Terraform configuration.If you enable this after the initial deployment, you must restart the controllers to pick up the changes:
kubectl -n cert-manager rollout restart deployment kubectl -n kube-system rollout restart deployment/cilium-operatorcilium_gateway_api_enabled = trueInstall kubelogin for OIDC authentication
mainTo use OIDC authentication with
kubectl, you must install thekubeloginplugin. Choose the command corresponding to your operating system and package manager.# Homebrew (macOS and Linux) brew install kubelogin # Krew (macOS, Linux, Windows and ARM) kubectl krew install oidc-login # Chocolatey (Windows) choco install kubelogin