K3s Lightweight Kubernetes Distribution

K3s uses a combination of Dependabot and Updatecli to automate the discovery and Pull Request (PR) creation for third-party dependency and container image updates. This automation reduces manual overhead and helps mitigate security vulnerabilities (CVEs) by ensuring dependencies and images are kept up to date.

Tooling Strategy

• Dependabot: Used for its simplicity and native GitHub integration. It handles major package ecosystems and Dockerfile image bumps with minimal configuration via .github/dependabot.yml.
• Updatecli: Used for complex orchestration and updating unstructured formats (e.g., environment variables or regex-based matches) that Dependabot cannot handle. It is implemented as a GitHub Action.

Benefits for Users

By automating these updates, K3s aims to ship code and container images with fewer known security issues in their underlying dependencies.

K3s is a fully conformant, production-ready Kubernetes distribution packaged as a single binary (less than 100 MB). It is designed to be lightweight for Edge, IoT, and CI environments.

Key Differences

• Single Binary: Wraps Kubernetes and all necessary components into one launcher.
• Storage: Uses sqlite3 as the default storage backend, though etcd3, MariaDB, MySQL, and Postgres are supported via Kine.
• Reduced Footprint: Achieved by running many components inside a single process and removing in-tree storage drivers and in-tree cloud providers (replaced by CSI and CCM).
• Networking: Uses Flannel for CNI and Kube-router for network policy. It eliminates the need to expose worker node ports for the kubelet API by using a websocket tunnel to the control plane.
• Bundled Components: Includes Containerd, CoreDNS, Metrics Server, Traefik (Ingress), Klipper-lb (Load Balancer), Helm-controller, and Local-path-provisioner.

K3s uses two primary types of tokens for cluster operations and node joining. Understanding the difference is critical for security and cluster management:

• --token (Server Token): The default token used during initial cluster startup. If not specified, K3s generates a random value. This token is used as the passphrase for the PBKDF2 function to generate the encryption key for cluster bootstrap data. Crucially, all server nodes must use the same token value, and this value cannot be changed once the cluster has started.
• --agent-token (Agent Token): Used to join new agents (worker nodes) to the cluster. By default, it matches the --token value. Unlike the server token, the agent token can be changed after the cluster has started, but changing it requires a coordinated reconfiguration and restart of all server nodes in the cluster.

Internally, these tokens act as passwords for HTTP Basic authentication to the K3s supervisor during the agent bootstrap process. Servers use the username server, while agents use the username node.

To prevent the size limitations of Kubernetes ConfigMaps, K3s manages etcd snapshot metadata using a cluster-scoped Custom Resource (CR) for each individual snapshot.

Key behaviors of the new system:

• Individual Resources: Instead of one large list in a ConfigMap, every snapshot has its own distinct Custom Resource.
• Metadata Redundancy: Metadata is written to the Custom Resource and also stored alongside the snapshot files on disk or in S3 (using the same basename as the snapshot file).
• Server Token Security: A hash of the server token is stored as an annotation on the Custom Resource and within S3 metadata. This allows K3s to determine if a server token rollback is required during a restore process.
• Transition Period: K3s supports a transition period where both the old ConfigMap and the new Custom Resources are used. During this time, older metadata might be removed from the ConfigMap to save space if the new Custom Resources already reference those snapshots.

K3s uses a distributed testing strategy across GitHub Actions and Drone CI, categorized by the scale and resource requirements of the tests. While the project is moving towards consolidating most testing into GitHub Actions, the distribution is as follows:

GitHub Actions (Standard/Fast Tests)

• Unit Tests: White-box testing of individual components and functions.
• Integration Tests: Black-box testing of functionalities across multiple packages.
• Smoke Tests: Basic functionality checks, including:
  • Install tests: Verifying K3s installation across various operating systems.
• Docker Basic Tests: Running clusters in containers to verify basic functionality.
• E2E Tests: Standard end-to-end tests (single node).

Drone CI (Resource-Intensive Tests)

• Docker Conformance Tests: Running clusters in containers to validate Kubernetes conformance (Sonobuoy) across multiple database backends.
• Large E2E Tests: Multi-node configuration and administration tests (2+ nodes).

Note: Legacy performance tests using Terraform have been removed from the CI pipeline.

K3s follows a structured lifecycle for deprecating and removing CLI flags to ensure users have sufficient time to migrate to new configuration options. If you encounter a warning about a flag, you should plan to migrate to the suggested replacement within one or two minor releases.

The Deprecation Lifecycle

1. To Be Deprecated: The flag is still fully operational, but the next patch release will include a warning message stating that the flag will be deprecated in the next minor release.
2. Deprecated: The flag is marked as deprecated in documentation and is hidden in the code. It remains operational and will still trigger a warning when used.
3. Nonoperational (Fatal Error): The flag is no longer functional. Using it will cause a fatal error. The error message will explicitly instruct you on which new flag or configuration replaces it.
4. Removed: The flag is completely removed from the documentation and the codebase.

K3s uses different voting thresholds depending on the importance of the decision. Most decisions are handled via "lazy consensus," but formal votes are taken when necessary.

Supermajority

Used for critical governance tasks like adding/removing Maintainers or modifying the Charter.

• Definition: At least two-thirds (2/3) of the active Maintainers.
• Calculation: Based on the number of votes cast, excluding abstentions. An "abstain" vote is equivalent to not voting and does not count toward the total used to calculate the 2/3 threshold.
• Quorum: A vote requires a quorum of more than 50% of active Maintainers to be valid.

Simple Majority

Used for less critical decisions, such as requesting CNCF resources.

• Definition: More than half (> 50%) of the votes cast.

Voting Requirements Summary

Note: For critical decisions, the CNCF Technical Oversight Committee (TOC) should be consulted.

The K3s project follows a structured community ladder that allows individuals to progress from users to project leaders. Understanding these roles helps you identify how to contribute and how to advance within the project.

Community Ladder

User $\rightarrow$ Contributor $\rightarrow$ Reviewer $\rightarrow$ Maintainer

Role Definitions

• Users: Engage with the community via Slack, GitHub, or mailing lists.
• Contributors: Regularly contribute to documentation, code, issue triage, or proposal discussions. Contributors are the pool from which Maintainers are drawn.
• Reviewers: Knowledgeable members who review code for quality and correctness in specific subprojects. They are responsible for project quality control and are expected to be responsive to review requests.
• Maintainers: The project leaders (the Maintainer Council). They have write access to the GitHub repository, manage releases, triage issues, set project direction, and oversee the overall health of the project.

K3s supports different modes of Certificate Authority (CA) management depending on how the cluster was initially bootstrapped:

1. External CA (Non-disruptive): If the cluster was started using user-provided certificates signed by an external root/intermediate CA, you can perform non-disruptive renewals. This requires only a service restart and no changes to node configuration.
2. Autogenerated CA (Disruptive): If the cluster was started with K3s-autogenerated self-signed CAs, renewal or replacement is considered disruptive. This requires updating K3s CLI flags, configuration files, or environment variables before restarting the service. Note that a temporary cluster outage may occur while configuration changes propagate, as nodes may temporarily lack a common root of trust.

Key Constraints to Note:

• Server CA Pinning: When joining a cluster via a K10 token, nodes validate the Cluster Server CA by matching a SHA256 hash in the token against the CA bundle downloaded from /v1-k3s/cacert.
• Bootstrap Immutability: Once certificates are written to the datastore during initial startup, they are considered authoritative. Replacing files on disk may result in errors or replacement depending on file timestamps.