DOCSEARCHbyAlgolia
Submit Docs

k0smotron Documentation

repository·main·Indexed 20 days ago

https://github.com/k0sproject/k0smotron

A Kubernetes control plane manager that uses k0s to run Kubernetes control planes as workloads within a management cluster (Mothership). It integrates with Cluster API (CAPI) to provide declarative provisioning, scaling, and upgrading of clusters across diverse infrastructures, including remote machines via SSH. k0smotron is optimized for CI/CD, edge computing, and multi-cloud workloads by separating the control plane from the worker plane.

Tokens
184.7K
Snippets
439
Records
658
Agent score
71%

What's inside k0smotron

  1. Overview of k0smotron capabilities

    main

    k0smotron is a Kubernetes control plane manager designed to manage k0s clusters. It integrates with Cluster API (CAPI) to provide lifecycle management (LCM) for Kubernetes clusters across various infrastructures.

    Core Capabilities:

    • Embedded Control Planes: Runs k0s control planes as pods within a management cluster, reducing operational overhead for installation, configuration, and upgrades.
    • Cluster API Integration: Operates as a control plane, bootstrap, and infrastructure provider for CAPI.
    • Lifecycle Operations: Supports provisioning, scaling (worker nodes and control planes), and smooth upgrades with minimal downtime.
    • k0smotron Anywhere: A Remote Machine Provider that allows provisioning clusters on remote machines via SSH, useful for environments lacking a native CAPI provider.

    Common Use Cases:

    • CI/CD: Declarative, on-demand creation of temporary clusters for testing to prevent cluster sprawl.
    • Edge Computing: Offloads the control plane to dedicated hardware while keeping worker nodes ephemeral, reducing the resource footprint on edge devices.
    • Multi-cloud LCM: Provides a unified management experience by running a single control plane management cluster in one cloud while distributing worker nodes across multiple different cloud providers.

  2. Use k0smotron for CI/CD, Edge, and Multi-cloud workloads

    main

    k0smotron is optimized for several specific deployment patterns:

    • CI/CD: Create temporary, on-demand clusters for integration and end-to-end testing by simply creating or deleting a custom resource. This eliminates the need for long-lived 'snowflake' clusters.
    • Edge Computing: Offload the resource-intensive task of running a control plane from low-resource edge infrastructure to a centralized Mothership cluster. This allows you to focus edge resources on actual workloads.
    • Multi-cloud: Run your management cluster (Mothership) in one cloud provider while distributing workloads across various other cloud providers, streamlining multi-cloud operations.

  3. Choose between Standalone and Cluster API modes

    main

    k0smotron can be deployed in two primary operational modes depending on your management requirements:

    1. Standalone Mode: k0smotron manages only k0s control planes running within the management cluster. This mode is designed for streamlined control and monitoring of k0s clusters.
    2. Cluster API (CAPI) Integration: k0smotron integrates with Cluster API to manage Kubernetes clusters across various infrastructure providers, extending management capabilities to a wider range of deployments.

  4. What is k0smotron and how does it work?

    main

    k0smotron is a Kubernetes operator designed to manage the lifecycle of k0s control planes within an existing Kubernetes cluster (referred to as the 'Mothership').

    By running control planes as workloads inside a Mothership cluster, you leverage the underlying cluster's high availability and auto-healing capabilities. k0smotron uses the Kubernetes Custom Resource (CR) pattern to manage these control planes, automatically provisioning the necessary low-level constructs like Pods and ConfigMaps required to run and manage the k0s control planes.

  5. Check control plane initialization status

    main

    The initialization object in K0smotronControlPlane.status tracks the setup progress of the control plane.

    • controlPlaneInitialized: A boolean indicating whether the control plane is initialized.

    Note on initialization.initialized vs conditions: In the context of the Cluster API contract, the initialized field (denoting if the API Server can accept requests) is used to orchestrate provisioning and is never updated after provisioning is complete. For monitoring the ongoing operational health and state of the control plane, always use the conditions field.

  6. Handle AWS Userdata size limits in AWSMachineTemplate

    main

    AWS imposes a 16kb limit on userdata. Because k0smotron generates certificates and other files, the userdata might exceed this limit.

    To prevent deployment failures, you can compress the userdata by setting uncompressedUserData: false within the AWSMachineTemplate.spec.template.spec section of your manifest.

    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachineTemplate
metadata:
  name: k0s-aws-test-mt
  namespace: default
spec:
  template:
    spec:
      uncompressedUserData: false
      # ... other fields

  7. Use ClusterTrustBundle projections

    main

    The clusterTrustBundle source (part of projected.sources) allows a pod to access the .spec.trustBundle field of ClusterTrustBundle objects in an auto-updating file.

    Note: This is an Alpha feature, gated by the ClusterTrustBundleProjection feature gate.

    ClusterTrustBundle objects can be selected by:

    1. Name
    2. A combination of signer name and a label selector.

  8. How podCertificate projected volumes work

    main

    The podCertificate source projects an auto-rotating credential bundle (private key and certificate chain) into a pod. This allows the pod to use the credentials as a TLS client or server.

    Implementation Details

    1. Generation: Kubelet generates a private key and sends a PodCertificateRequest to a named signer.
    2. Issuance: Once the signer approves, Kubelet writes the key and certificate chain to the pod filesystem.
    3. Lifecycle: The pod will not start until certificates have been issued for all podCertificate volumes in its spec. Kubelet handles rotation based on the BeginRefreshAt timestamp provided by the signer.

    Usage Formats

    You can choose between two formats:

    • Credential Bundle (Recommended): Uses credentialBundlePath to project a single PEM file. The first entry is the PKCS#8 private key, followed by the certificate chain. This is preferred because it allows applications to read the credentials atomically.
    • Separate Files: Uses keyPath and certificateChainPath. Warning: If your application reads these files during a rotation, the private key and leaf certificate might not match. Your application must implement logic to check for consistency and re-read if necessary.

  9. High availability and scaling constraints for NATS storage

    main

    For multi-replica clusters, the embedded NATS servers form a cluster with JetStream replication across all pods.

    Critical Constraints:

    • Replica count is immutable: You cannot change the replica count after the cluster is created. NATS JetStream uses RAFT for consensus and does not support adding or removing members without risking quorum loss. Set spec.replicas to its final value before creation.
    • Quorum: For a 3-replica cluster, quorum requires at least 2 nodes, allowing the cluster to tolerate 1 node failure.
    • Pod Management: All pods start simultaneously using a Parallel pod management policy to ensure the NATS cluster forms before kine initializes.

  10. Target resources for patching using kind and component

    main

    To ensure predictable and maintainable customization, patches target resources using a target block. This avoids brittle name-based or template-based targeting.

    Each patch entry must specify:

    • kind: The Kubernetes Kind (e.g., StatefulSet, Service, ConfigMap).
    • component: The value of the app.kubernetes.io/component label applied to the generated resource.

    Warning: A (kind, component) combination is not always unique. If multiple resources share the same kind and component (e.g., two different ConfigMaps both labeled component: monitoring), the patch will be applied to all matching resources.

  11. Configure k0s control plane update strategies

    main

    The spec.updateStrategy field in the K0sControlPlane resource determines how upgrades are rolled out:

    StrategyBehavior
    InPlace (default)Updates k0s on existing machines without replacing them, using k0s autopilot.
    RecreateCreates new machines first, then removes old ones.
    RecreateDeleteFirstRemoves old machines first, then creates new ones.

    Constraints:

    • The Recreate strategy is not supported for clusters running in --single mode.
    • The RecreateDeleteFirst strategy requires at least 3 control plane nodes to maintain quorum.

  12. Define K0s worker configurations using K0sWorkerConfigTemplate

    main

    The K0sWorkerConfigTemplate is a Cluster API resource (bootstrap.cluster.x-k8s.io/v1beta1) used to define a reusable template for k0s worker configurations. It allows you to specify how k0s workers should be provisioned, including binary versions, installation directories, and custom user data.

    To use it, define a spec.template which contains a K0sWorkerConfigSpec. This spec governs the actual state of the worker nodes.

    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: K0sWorkerConfigTemplate
metadata:
  name: my-worker-template
spec:
  template:
    metadata:
      labels:
        env: production
    spec:
      version: "v2.0.0"
      k0sInstallDir: /usr/local/bin