DOCSEARCHbyAlgolia
Submit Docs

onedr0p cluster-template

repository·main·Indexed 25 days ago

https://github.com/onedr0p/cluster-template

A template for deploying production-ready, highly available Kubernetes clusters on bare-metal or virtual machines. It utilizes Talos Linux, Flux GitOps, and Cloudflare for networking, with configuration rendered via makejinja from a pydantic-validated cluster.toml file. Included components feature Cilium, Envoy Gateway, cert-manager, External-DNS, and SOPS for secret management.

Tokens
3.1K
Snippets
9
Records
22
Agent score
34%

What's inside cluster-template

  1. Overview of the Cluster Template

    main
    The cluster-template is a project designed to simplify the deployment of a single Kubernetes cluster on bare-metal or virtual machines. It uses makejinja to render configurations from a cluster.toml file (validated by pydantic). The template automates the deployment of Talos Linux, Kubernetes, and Flux, providing a modular foundation for managing Kubernetes environments.

  2. Reset the cluster

    main

    To destroy the Kubernetes cluster and return nodes to maintenance mode, use the talos reset command.

    CAUTION

    Resetting the cluster multiple times in a short period can lead to rate limiting by DockerHub or Let's Encrypt.

    just talos reset

  3. Configure Public and Private DNS access

    main

    Public DNS

    To make applications accessible from the internet, use the envoy-external gateway on your HTTPRoutes. The external-dns application in the network namespace manages public records. By default, only echo and flux-webhook are public; other applications must explicitly set the correct gateway in their HelmRelease.

    Home (Private) DNS

    To make applications private to your local network, use the envoy-internal gateway on your HTTPRoutes.

    Split DNS Requirement: For internal resolution to work, your home DNS server must be configured to forward queries for your ${cloudflare_domain} to ${gateways_dns} (conditional forwarding).

  4. Persistent Storage options for Kubernetes workloads

    main

    If your workloads require persistent storage with features like replication or connectivity to NFS, SMB, or iSCSI servers, consider the following projects:

    Cloud-native/Distributed Storage:

    • rook-ceph
    • longhorn
    • openebs

    CSI Drivers and Specialized Storage:

    • democratic-csi
    • csi-driver-nfs
    • csi-driver-smb
    • synology-csi
    • truenas-csi / tns-csi

  5. Alternative DNS solutions for Kubernetes

    main

    The cluster template uses k8s_gateway by default to provide DNS for applications. If you require broader support for various DNS providers or wish to offload DNS management to your router or an external device, consider using external-dns.

    Supported providers for external-dns include:

    • Pi-hole
    • UniFi
    • Adguard Home
    • Bind

  6. Alternative Secret management with External Secrets

    main
    While the template uses SOPS for managing secrets in a GitOps workflow, SOPS can become difficult to manage during secret rotation. For a more streamlined approach, you can use External Secrets. This allows you to leverage external providers (cloud-based or self-hosted) to manage secrets instead of relying solely on SOPS.

  7. Set up your local workstation environment

    main

    Follow these steps to set up your development environment using mise:

    1. Create a repository from the template using the GitHub CLI:
      export REPONAME="home-ops"
gh repo create $REPONAME --template onedr0p/cluster-template --public --clone
cd $REPONAME
    2. Install and activate Mise CLI.
    3. Install the required CLI tools via mise:
      mise trust
mise install
    4. (Optional) Logout of the GitHub Container Registry to avoid authorization issues:
      docker logout ghcr.io
helm registry logout ghcr.io
    export REPONAME="home-ops"
gh repo create $REPONAME --template onedr0p/cluster-template --public --clone
cd $REPONAME

# Install tools
mise trust
mise install

# Cleanup registry auth
docker logout ghcr.io
helm registry logout ghcr.io

  8. Bootstrap Talos, Kubernetes, and Flux

    main

    Once configuration is complete, run the bootstrap process. Note that errors like couldn't get current server API group list are normal during the initial rollout.

    1. Install Talos:
      just bootstrap talos
    2. Install Apps (Cilium, Flux, etc.):
      just bootstrap apps
    3. Monitor the rollout:
      kubectl get pods --all-namespaces --watch
    just bootstrap talos
just bootstrap apps
kubectl get pods --all-namespaces --watch

  9. Prepare nodes with Talos Linux

    main

    To prepare your hardware for the cluster:

    1. Visit the Talos Linux Image Factory.
    2. Select the bare-minimum system extensions (e.g., i915/intel-ucode for Intel or amdgpu/amd-ucode for AMD).
    3. Download the Talos Linux ISO (or RAW for SBCs) and note the schematic ID.
    4. Flash the image to a USB drive and boot your nodes.
    5. Verify nodes are reachable on the network using nmap (replace the subnet as needed):
    nmap -Pn -n -p 50000 192.168.1.0/24 -vv | grep 'Discovered'

  10. Configure Git Webhooks for Flux reconciliation

    main

    To enable Flux to reconcile immediately on git push instead of waiting for the polling interval, configure a webhook in your Git provider. If your provider cannot reach the cluster, set webhook_provider = "none" in cluster.toml to stick with polling.

    1. Get the Webhook Path: Retrieve the path from the flux-webhook receiver.
    2. Construct the URL: Combine https://flux-webhook.${cloudflare_domain}/ with the path obtained in step 1.
    3. Provider Setup:
      • GitHub: Add webhook via Settings/Webhooks. Use the URL, the secret from flux-webhook-token.txt, Content type application/json, and select the push event.
      • GitLab: Add webhook via Settings/Webhooks. Use the URL and secret. Set webhook_provider = "gitlab" in cluster.toml.
      • Gitea/Forgejo: Add a Gitea/Forgejo webhook. Use POST, application/json, and the secret. Keep webhook_provider = "github" in cluster.toml as they emulate GitHub.

  11. Tidy up the repository after configuration

    main

    Once the cluster is fully configured and you no longer need to run just configure, clean up the repository by removing the template directory and templating files to avoid clutter and Renovate warnings.

    just template tidy
git add -A
git commit -m "chore: tidy up :broom:"
git push

  12. Verify cluster installation status

    main

    After installation, perform these checks to ensure the core components are running correctly:

    1. Cilium: Check the status of the Cilium agent.
    2. Flux CD: Verify that Flux resources are up-to-date and ready. Use just kube reconcile to force a sync if needed.
    3. Connectivity: Test TCP connectivity to internal and external gateways using nmap.
    4. DNS: Verify DNS resolution for the echo service.
    5. Certificates: Check the status of the wildcard Certificate in the network namespace.
    # 1. Check Cilium status
kubectl -n kube-system exec ds/cilium --container cilium-agent -- cilium status

# 2. Check Flux status
flux check
flux get sources git flux-system
flux get ks -A
flux get hr -A

# 3. Check TCP connectivity (replace placeholders)
nmap -Pn -n -p 443 ${gateways_internal} ${gateways_external} -vv

# 4. Check DNS resolution (replace placeholders)
dig @${gateways_dns} echo.${cloudflare_domain}

# 5. Check wildcard Certificate
kubectl -n network describe certificates