DOCSEARCHbyAlgolia
Submit Docs

k3s-ansible

repository·master·Indexed 25 days ago

https://github.com/timothystewart6/k3s-ansible

An Ansible-based automation tool for building High Availability (HA) k3s Kubernetes clusters. It integrates kube-vip for control plane load balancing and MetalLB for service LoadBalancer support. The tool supports IPv6-first dual-stack networking, multiple CNI options including Calico and Cilium, and is compatible with Debian, Ubuntu, and Rocky Linux across x64, arm64, and armhf architectures.

Tokens
5.6K
Snippets
7
Records
11
Agent score
84%

What's inside k3s-ansible

  1. Configure system requirements for k3s-ansible

    master

    To use this playbook, ensure the following environment conditions are met:

    • Control Node: Must have Ansible 2.11+ installed.
    • Python Dependency: The netaddr package must be available to Ansible. If using pip to install Ansible, install netaddr into the same virtual environment.
    • SSH Access: server and agent nodes should have passwordless SSH access. If not, you must append --ask-pass --ask-become-pass to your Ansible commands.
    • Supported OS: Debian (tested on v11), Ubuntu (tested on v22.04), and Rocky (tested on v9).
    • Supported Architectures: x64, arm64, and armhf.

  2. Set up Molecule testing for k3s-ansible

    master

    To test the k3s-ansible playbook locally, you must set up a Python virtual environment and install the necessary dependencies. This ensures that Molecule, Ansible, and other requirements are isolated from your system Python.

    Prerequisites

    Ensure you have the following installed:

    • Python 3
    • Vagrant
    • VirtualBox

    VirtualBox Networking (Linux and macOS only)

    If you are on Linux or macOS, you must manually configure the 192.168.30.0/24 subnet used by the test cluster. Ensure /etc/vbox/networks.conf exists and contains:

    * 192.168.30.0/24
* fdad:bad:ba55::/64

    Note: Windows users can skip this step.

    # Create a virtualenv at ".env"
python3 -m venv .env

# Activate the virtualenv
source .env/bin/activate

# Install the required packages
python3 -m pip install -r requirements.txt

  3. Execute Molecule test scenarios

    master

    Once your virtual environment is active, you can use molecule commands to manage and test different cluster configurations.

    Available Scenarios

    • default: 3 control + 2 worker node cluster.
    • ipv6: Single control and single worker node, externally accessible via IPv6.
    • single_node: Single node for all cluster functionality.
    • calico: Single node using Calico CNI instead of Flannel.
    • cilium: Single node using Cilium CNI instead of Flannel.
    • kube-vip: Single node using kube-vip as the service loadbalancer instead of MetalLB.

    Molecule Commands

    • molecule create: Create virtual machines for the test cluster nodes.
    • molecule destroy: Delete the virtual machines.
    • molecule converge: Run the site playbook on the nodes.
    • molecule side_effect: Run the reset playbook on the nodes.
    • molecule verify: Verify that the cluster works correctly.
    • molecule test: Runs the full sequence: create -> converge -> verify -> side_effect -> destroy.

  4. Prepare a new cluster inventory

    master

    To set up a new cluster configuration, follow these steps:

    1. Create a new inventory directory by copying the sample:
      cp -R inventory/sample inventory/my-cluster
    2. Edit inventory/my-cluster/hosts.ini to define your hosts. Use the [master] group for control plane nodes and [node] for workers. If multiple hosts are in the [master] group, the playbook automatically configures k3s in HA mode with etcd.
    3. Copy ansible.example.cfg to ansible.cfg and update the inventory path to point to your new directory.
    4. (Optional) Edit inventory/my-cluster/group_vars/all.yml to customize environment variables like k3s_version (minimum 1.19.1) or apiserver_endpoint.
    [master]
192.168.30.38
192.168.30.39
192.168.30.40

[node]
192.168.30.41
192.168.30.42

[k3s_cluster:children]
master
node

  5. Install k3s-ansible as an Ansible Galaxy Collection

    master

    You can use this project as a collection within larger Ansible projects.

    1. Create or modify a collections/requirements.yml file in your project with the following content:
    2. Install the collection using ansible-galaxy.

    Once installed, all roles are available using the prefix techno_tim.k3s_ansible. (e.g., techno_tim.k3s_ansible.lxc).

    # collections/requirements.yml
collections:
  - name: ansible.utils
  - name: community.general
  - name: ansible.posix
  - name: kubernetes.core
  - https://github.com/timothystewart6/k3s-ansible.git
    type: git
    version: master

    Install command

    ansible-galaxy collection install -r ./collections/requirements.yml

  6. Remove a k3s cluster

    master

    To decommission the cluster and run the reset playbook, use the reset.yml file.

    Note: You should reboot the nodes after running this command because the Virtual IP (VIP) may not be fully destroyed during the reset process.

    ansible-playbook reset.yml -i inventory/my-cluster/hosts.ini

  7. Create a k3s cluster

    master

    Provision the HA Kubernetes cluster using the site.yml playbook and your specific inventory file. Once deployed, the control plane is accessible via the virtual IP address defined as apiserver_endpoint in your inventory/group_vars/all.yml.

    ansible-playbook site.yml -i inventory/my-cluster/hosts.ini

  8. Configure IPv6-first dual-stack networking

    master

    This configuration pattern implements an IPv6-first cluster that maintains dual-stack support for IPv4. In this setup:

    • The Kubernetes API server VIP uses an IPv6 address.
    • MetalLB pools include both IPv4 and IPv6 addresses.
    • Nodes and cluster-internal resources (pods and services) are accessible via both IPv4 and IPv6.

    Network Design Reference

    IPv6 Subnets (Prefix: fdad:bad:ba55):

    • External Access Subnet (fdad:bad:ba55:0::/64): Used for cluster components requiring external access.
      • API Server VIP: fdad:bad:ba55::333
      • MetalLB Services: fdad:bad:ba55::1b:0/112
      • Cluster Nodes: fdad:bad:ba55::de:0/112
      • Vagrant Host: fdad:bad:ba55::1
    • Pod Internal Subnet: fdad:bad:ba55:4200::/56
    • Service Internal Subnet: fdad:bad:ba55:4300::/108

    IPv4 Subnets:

    • Node/MetalLB Network: 192.168.123.0/24 (MetalLB range: 192.168.123.80-192.168.123.90)
    • Pod/Service Defaults: 10.42.0.0/16 (Pods) and 10.43.0.0/16 (Services)

  9. Install Ansible collections for k3s-ansible

    master

    Before running the playbook, you must install the required Ansible collections using the provided requirements file located in the ./collections/ directory.

    ansible-galaxy collection install -r ./collections/requirements.yml

  10. Copy Kube Config to local machine

    master

    To access your Kubernetes cluster locally, copy the k3s.yaml file from a master node to your local ~/.kube/config.

    If you encounter a 'Permission denied' error, you must temporarily change the permissions on the node, copy the file, and then revert the permissions.

    After copying, you must update the server address in your local config to point to the master node's IP address instead of 127.0.0.1.

  11. Reference Ansible Variables for k3s-ansible

    master

    The following variables are used by the various roles in the playbook. Note that some variables are required for specific roles.

    | Role(s) | Variable | Type | Default | Required | Description |
|---|---|---|---|---|---|
| `download` | `k3s_version` | string | ❌ | Required | K3s binaries version |
| `k3s_agent`, `k3s_server`, `k3s_server_post` | `apiserver_endpoint` | string | ❌ | Required | Virtual ip-address configured on each master |
| `k3s_agent` | `extra_agent_args` | string | `null` | Not required | Extra arguments for agents nodes |
| `k3s_agent`, `k3s_server` | `group_name_master` | string | `null` | Not required | Name othe master group |
| `k3s_agent` | `k3s_token` | string | `null` | Not required | Token used to communicate between masters |
| `k3s_agent`, `k3s_server` | `proxy_env` | dict | `null` | Not required | Internet proxy configurations |
| `k3s_agent`, `k3s_server` | `proxy_env.HTTP_PROXY` | string | ❌ | Required | HTTP internet proxy |
| `k3s_agent`, `k3s_server` | `proxy_env.HTTPS_PROXY` | string | ❌ | Required | HTTP internet proxy |
| `k3s_agent`, `k3s_server` | `proxy_env.NO_PROXY` | string | ❌ | Required | Addresses that will not use the proxies |
| `k3s_agent`, `k3s_server`, `reset` | `systemd_dir` | string | `/etc/systemd/system` | Not required | Path to systemd services |
| `k3s_custom_registries` | `custom_registries_yaml` | string | ❌ | Required | YAML block defining custom registries |
| `k3s_server`, `k3s_server_post` | `cilium_bgp` | bool | `~` | Not required | Enable cilium BGP control plane for LB services and pod cidrs. Disables the use of MetalLB |
| `k3s_server`, `k3s_server_post` | `cilium_iface` | string | ❌ | Not required | The network interface used for when Cilium is enabled |
| `k3s_server` | `extra_server_args` | string | `""` | Not required | Extra arguments for server nodes |
| `k3s_server` | `k3s_create_kubectl_symlink` | bool | `false` | Not required | Create the kubectl -> k3s symlink |
| `k3s_server` | `k3s_create_crictl_symlink` | bool | `true` | Not required | Create the crictl -> k3s symlink |
| `k3s_server` | `kube_vip_arp` | bool | `true` | Not required | Enables kube-vip ARP broadcasts |
| `k3s_server` | `kube_vip_bgp` | bool | `false` | Not required | Enables kube-vip BGP peering |
| `k3s_server` | `kube_vip_bgp_routerid` | string | `"127.0.0.1"` | Not required | Defines the router ID for the kube-vip BGP server |
| `k3s_server` | `kube_vip_bgp_as` | string | `"64513"` | Not required | Defines the AS for the kube-vip BGP server |
| `k3s_server` | `kube_vip_bgp_peeraddress` | string | `"192.168.30.1"` | Not required | Defines the address for the kube-vip BGP peer |
| `k3s_server` | `kube_vip_bgp_peeras` | string | `"64512"` | Not required | Defines the AS for the kube-vip BGP peer |
| `k3s_server` | `kube_vip_bgp_peers` | list | `[]` | Not required | List of BGP peer ASN & address pairs |
| `k3s_server` | `kube_vip_bgp_peers_groups` | list | `['k3s_master']` | Not required | Inventory group in which to search for additional `kube_vip_bgp_peers` parameters to merge |
| `k3s_server` | `kube_vip_iface` | string | `~` | Not required | Explicitly define an interface that ALL control nodes should use to propagate the VIP |
| `k3s_server` | `kube_vip_tag_version` | string | `v0.7.2` | Not required | Image tag for kube-vip |
| `k3s_server` | `kube_vip_cloud_provider_tag_version` | string | `main` | Not required | Tag for kube-vip-cloud-provider manifest when enable |
| `k3s_server`, `k3_server_post` | `kube_vip_lb_ip_range` | string | `~` | Not required | IP range for kube-vip load balancer |
| `k3s_server`, `k3s_server_post` | `metal_lb_controller_tag_version` | string | `v0.14.3` | Not required | Image tag for MetalLB |
| `k3s_server` | `metal_lb_speaker_tag_version` | string | `v0.14.3` | Not required | Image tag for MetalLB |
| `k3s_server` | `metal_lb_type` | string | `native` | Not required | Use FRR mode or native. Valid values are `frr` and `native` |
| `k3s_server` | `retry_count` | int | `20` | Not required | Amount of retries when verifying that nodes joined |
| `k3s_server` | `server_init_args` | string | ❌ | Not required | Arguments for server nodes |
| `k3s_server_post` | `bpf_lb_algorithm` | string | `maglev` | Not required | BPF lb algorithm |
| `k3s_server_post` | `bpf_lb_mode` | string | `hybrid` | Not required | BPF lb mode |
| `k3s_server_post` | `calico_blocksize` | int | `26` | Not required | IP pool block size |
| `k3s_server_post` | `calico_ebpf` | bool | `false` | Not required | Use eBPF dataplane instead of iptables |
| `k3s_server_post` | `calico_encapsulation` | string | `VXLANCrossSubnet` | Not required | IP pool encapsulation |
| `k3s_server_post` | `calico_natOutgoing` | string | `Enabled` | Not required | IP pool NAT outgoing |
| `k3s_server_post` | `calico_nodeSelector` | string | `all()` | Not required | IP pool node selector |
| `k3s_server_post` | `calico_iface` | string | `~` | Not required | The network interface used for when Calico is enabled |
| `k3s_server_post` | `calico_tag` | string | `v3.27.2` | Not required | Calico version tag |
| `k3s_server_post` | `cilium_bgp_my_asn` | int | `64513` | Not required | Local ASN for BGP peer |
| `k3s_server_post` | `cilium_bgp_peer_asn` | int | `64512` | Not required | BGP peer ASN |
| `k3s_server_post` | `cilium_bgp_peer_address` | string | `~` | Not required | BGP peer address |
| `k3s_server_post` | `cilium_bgp_neighbors` | list | `[]` | Not required | List of BGP peer ASN & address pairs |
| `k3s_server_post` | `cilium_bgp_neighbors_groups` | list | `['k3s_all']` | Not required | Inventory group in which to search for additional `cilium_bgp_neighbors` parameters to merge |
| `k3s_server_post` | `cilium_bgp_lb_cidr` | string | `192.168.31.0/24` | Not required | BGP load balancer IP range |
| `k3s_server_post` | `cilium_exportPodCIDR` | bool | `true` | Not required | Export pod CIDR |
| `k3s_server_post` | `cilium_hubble` | bool | `true` | Not required | Enable Cilium Hubble |
| `k3s_server_post` | `cilium_mode` | string | `native` | Not required | Inner-node communication mode (choices are `native` and `routed`) |
| `k3s_server_post` | `cluster_cidr` | string | `10.52.0.0/16` | Not required | Inner-cluster IP range |
| `k3s_server_post` | `enable_bpf_masquerade` | bool | `true` | Not required | Use IP masquerading |
| `k3s_server_post` | `kube_proxy_replacement` | bool | `true` | Not required | Replace the native kube-proxy with Cilium |
| `k3s_server_post` | `metal_lb_available_timeout` | string | `240s` | Not required | Wait for MetalLB resources |
| `k3s_server_post` | `metal_lb_ip_range` | string | `192.168.30.80-192.168.30.90` | Not required | MetalLB ip range for load balancer |
| `k3s_server_post` | `metal_lb_controller_tag_version` | string | `v0.14.3` | Not required | Image tag for MetalLB |
| `k3s_server_post` | `metal_lb_mode` | string | `layer2` | Not required | Metallb mode (choices are `bgp` and `layer2`) |
| `k3s_server_post` | `metal_lb_bgp_my_asn` | string | `~` | Not required | BGP ASN configurations |
| `k3s_server_post` | `metal_lb_bgp_peer_asn` | string | `~` | Not required | BGP peer ASN configurations |
| `k3s_server_post` | `metal_lb_bgp_peer_address` | string | `~` | Not required | BGP peer address |
| `lxc` | `custom_reboot_command` | string | `~` | Not required | Command to run on reboot |
| `prereq` | `system_timezone` | string | `null` | Not required | Timezone to be set on all nodes |
| `proxmox_lxc`, `reset_proxmox_lxc` | `proxmox_lxc_ct_ids` | list | ❌ | Required | Proxmox container ID list |
| `raspberrypi` | `state` | string | `present` | Not required | Indicates whether the k3s prerequisites for Raspberry Pi should be set up (possible values are `present` and `absent`) |