Kubespray

Kubespray tests are organized into stages that correspond to build pipeline steps. The execution order follows: unit_tests -> part1 -> part2 -> part3 -> special.

• unit_tests: Covers linting, markdown validation, and Vagrant/Terraform validation.
• part1: Focuses on Molecule and All-in-One (AIO) tests.
• part2: Standard tests covering various layouts, Operating Systems (OS), Runtimes, and Network Plugins.
• part3: Includes upgrade jobs, Terraform jobs, and control plane recovery tests.
• special: Manual or other miscellaneous jobs.

Kubespray uses a local cache to speed up subsequent provisioning runs and minimize bandwidth usage.

Local Cache Configuration

When download_run_once is True, all downloaded files are stored in download_cache_dir (defaults to /tmp/kubespray_cache).

• Disk Usage: Expect approximately 800MB of disk space on the Ansible node for the cache.
• Forcing Cache: If you have a full cache and want to use it without triggering new downloads, set download_force_cache: True.

Remote Node Cache Management

By default, Kubespray deletes cached images from remote nodes after they are used to save space.

• Keep Remote Cache: Set download_keep_remote_cache: True to prevent deletion. This is useful during development to decrease provisioning times, but increases remote storage requirements from ~150MB to ~550MB (the combined size of all required images).

You can define various Kubernetes topologies by specifying the number of hosts for different classes. For each class, you can choose whether or not to allocate floating IP addresses.

Supported host classes:

• Control plane nodes with etcd
• Control plane nodes without etcd
• Standalone etcd hosts
• Kubernetes worker nodes

Important Constraint: The Ansible script will fail if the total number of etcd instances is an even number. This total includes both standalone etcd nodes and etcd replicas deployed within control plane nodes. For example, 3 control plane nodes with etcd + 3 standalone etcd nodes = 6 total etcd instances, which is an invalid configuration.

Kubespray supports three layers of variable precedence for customization. Use them according to your needs:

1. inventory group_vars: The most common method for setting group-level configurations.
2. inventory host_vars: Used for host-specific overrides (though group_vars is usually more practical).
3. extra vars: Used to override settings via the command line using ansible-playbook -e @foo.yml.

Warning: Extra vars have the highest precedence and always win. Use them sparingly; they are best for overriding internal Kubespray variables (like those in roles/vars/). Modifying internal variables is not part of the official Kubespray interface and may cause unexpected breakage if those variables change or disappear in future updates.

Kubespray supports several methods for provisioning the underlying cloud instances before running the deployment playbooks:

• Terraform: Use Terraform to manage infrastructure as code.
• Ansible: Use Ansible modules such as ec2 (for AWS) or gce (for Google Cloud Platform).
• Manual Creation: Manually create instances in your cloud environment and then provide their details to Kubespray via the inventory.

Kubespray and Kubeadm serve different primary functions in the Kubernetes ecosystem, though they are now integrated:

• Kubeadm focuses on Kubernetes cluster lifecycle management (e.g., self-hosted layouts, dynamic discovery). It is essentially a tool for managing the Kubernetes components themselves.
• Kubespray focuses on generic configuration management (the "OS operator" role) and initial cluster bootstrapping, including networking plugins.

Integration Note: Since version v2.3, Kubespray uses kubeadm internally for cluster creation. This allows Kubespray to leverage kubeadm's lifecycle management expertise while Kubespray handles the underlying OS configuration and infrastructure setup.

Flannel supports the following backends:

• vxlan
• host-gw
• wireguard

VXLAN Workaround: If you encounter issues with the VXLAN backend, you may need to disable offloading using ethtool. The current workaround is:

ethtool --offload flannel.1 rx off tx off

As of Kubespray 2.18.0, containerd is the default container engine.

Warning: Migrating container engines is not officially supported by Kubespray. While a migration procedure exists, it is not guaranteed to work for all scenarios. The recommended and safest approach is to reset and redeploy the entire cluster with the new container engine.

The private registry addon provides a place to store private Docker images within your cluster. It runs as a Pod in the kube-system namespace.

Because the registry does not currently support SSL or authentication, it would normally trigger Docker's "insecure registry" errors. To bypass this, a proxy is deployed as a DaemonSet on every node. This proxy exposes port 5000 on the host via a hostPort. Since Docker accesses the registry via localhost:5000, it treats the connection as secure.

The KubeVirt Image Builder uses an Ansible playbook to automate the creation of VM disk images for Kubespray CI testing.

The workflow is as follows:

1. Downloads upstream cloud images.
2. Converts images to qcow2 format.
3. Resizes the images by adding 8GB.
4. Wraps the disk in a Docker image using the kubevirt/registry-disk-v1alpha base.
5. Pushes the resulting image to quay.io/kubespray/vm-<os-name>:<tag>.

Trusted CI jobs can override the target registry for staged publishing using Cloud Build authentication, which bypasses the need for docker login.

# Example of pushing a single image to the staging registry in CI
make push-single-staging image_name=ubuntu-2404