DOCSEARCHbyAlgolia
Submit Docs

k0rdent Cluster Manager (KCM)

repository·main·Indexed 41 days ago

https://github.com/k0rdent/kcm

An enterprise-grade, multi-cluster Kubernetes management solution for managing cluster lifecycles across private and public clouds, including AWS, Azure, GCP, and vSphere. KCM provides a hierarchical template system using ClusterTemplate and ServiceTemplate, supports dry-run validation for ClusterDeployments, and integrates with Sveltos and Flux.

Tokens
5.5K
Snippets
12
Records
23
Agent score
51%

What's inside KCM

  1. Use Dry Run mode for ClusterDeployments

    main

    The dryRun flag in a ClusterDeployment allows you to validate your configuration before any infrastructure is actually provisioned.

    • Automatic Dry Run: If you do not provide a spec.config block, KCM will populate the object with defaults from the Template and automatically mark it as dryRun: true.
    • Validation: Use dryRun: true to ensure your configuration is valid. A configuration is considered valid when the Template reaches the TemplateReady condition in its status.conditions.
    • Execution: Once validation passes, set spec.dryRun to false to trigger the actual infrastructure provisioning.

  2. How KCM CI/CD workflows function

    main

    KCM uses GitHub Actions for CI/CD, organized into several distinct phases:

    1. Release (release.yml): Triggered by a release, it builds and uploads Helm charts and images to ghcr.io/k0rdent/kcm.
    2. Build and Test (build_test.yml):
      • Lint and Unit Test: Runs controller unit tests and linters.
      • Build and Push Artifacts: Builds Helm charts and controller/telemetry images, uploading them to ghcr.io/k0rdent/kcm/staging. This job is conditional on the test-e2e label and authorization.
      • E2E Tests: Runs concurrently after successful builds. Includes E2E Controller (always runs in a kind cluster), E2E Cloud (AWS/Azure), and E2E Onprem (vSphere).
    3. Cleanup: Automatically runs for AWS and Azure tests to remove resources using make dev-aws-nuke and make dev-azure-nuke.

  3. Configure vSphere CCM and CSI credentials

    main

    CCM (Cloud Controller Manager)

    cloud-provider-vsphere expects configuration passed via a ConfigMap, where the credentials reside in a referenced Secret. The configuration is a YAML file.

    KCM applies the following defaults for vSphere CCM:

    • insecureFlag: set to true (to omit certificate management).
    • port: set to 443 (HTTPS).
    • Multi-vcenter labels: defaults to k8s-region and k8s-zone.

    CSI (Container Storage Interface)

    CSI expects a single Secret containing configuration in .ini format. The options and defaults are similar to the CCM configuration.

  4. Configure Azure CCM/CSI credentials

    main

    Azure CCM/CSI controllers require an azure.json file provided via a Kubernetes Secret or a host file system.

    The KCM (2A) controller automatically creates a Secret named azure-cloud-provider in the kube-system namespace. The name of this secret is passed to the controllers via Helm values. Most parameters are derived from CAPZ objects.

  5. Deploy KCM and a managed cluster

    main

    Follow these steps to deploy the KCM management cluster and a managed cluster on your chosen provider:

    1. Set Provider: Set DEV_PROVIDER (e.g., export DEV_PROVIDER=azure).
    2. Configure Parameters: Edit the provider-specific file (e.g., config/dev/aws-clusterdeployment.yaml) to set the name, machine sizes, and counts.
    3. Deploy Management Cluster: Run make dev-apply.
    4. Apply Provider Credentials: Run make dev-creds-apply.
    5. Deploy Managed Cluster: Run make dev-mcluster-apply.
    6. Monitor Deployment: Use clusterctl describe to watch progress:
      export KUBECONFIG=~/.kube/config
./bin/clusterctl-<version> describe cluster <clusterdeployment-name> -n kcm-system --show-conditions all
    7. Retrieve Kubeconfig: Once deployed, extract the managed cluster's kubeconfig:
      kubectl --kubeconfig ~/.kube/config get secret -n kcm-system <clusterdeployment-name>-kubeconfig -o=jsonpath={.data.value} | base64 -d > kubeconfig
    # Example: Monitoring a deployment
export KUBECONFIG=~/.kube/config
./bin/clusterctl-<version> describe cluster <clusterdeployment-name> -n kcm-system --show-conditions all

# Example: Retrieving the managed cluster kubeconfig
kubectl --kubeconfig ~/.kube/config get secret -n kcm-system <clusterdeployment-name>-kubeconfig -o=jsonpath={.data.value} | base64 -d > kubeconfig

  6. Install k0rdent Cluster Manager (KCM) via Helm

    main

    To install KCM with support for custom configurations, use the Helm chart. It is recommended to use Helm over Kubernetes manifests to allow for deployment customization.

    Prerequisites:

    • An existing management cluster (Kubernetes version 1.28.0 or higher).
    • kubectl CLI installed locally.
    • helm CLI installed locally.
    • Set your KUBECONFIG environment variable to point to your management cluster.

    Installation Command:

    export KUBECONFIG=<path-to-management-kubeconfig>
helm install kcm oci://ghcr.io/k0rdent/kcm/charts/kcm --version <kcm-version> -n kcm-system --create-namespace

  7. Run KCM E2E tests locally

    main

    End-to-end (E2E) tests can be executed using make test-e2e. You must modify test/e2e/config/config.yaml before running.

    Running Non-Hosted Templates (Local/Standalone)

    Use this for standalone, adopted, or remote tests where you want to run everything locally using local images and registries.

    make KIND_CLUSTER_NAME="kcm-test" CLUSTER_DEPLOYMENT_PREFIX="<your-cluster-name-prefix>" GINKGO_LABEL_FILTER="<optional>" REGISTRY_REPO="oci://127.0.0.1:5001/charts" VERSION=1.0.0 IMG=localhost/kcm/controller:1.0.0 IMG_TELEMETRY=localhost/kcm/telemetry:1.0.0 kind-deploy registry-deploy set-kcm-version dev-push test-e2e

    Running Hosted Templates

    Requires a prior CI run to ensure manifests, images, and charts are available in the registry.

    make KIND_CLUSTER_NAME="kcm-test" CLUSTER_DEPLOYMENT_PREFIX="<your-cluster-name-prefix>" GINKGO_LABEL_FILTER="<optional>" IMG="ghcr.io/k0rdent/kcm/staging/controller:<version-from-ci>" IMG_TELEMETRY="ghcr.io/k0rdent/kcm/staging/telemetry:<version-from-ci>" VERSION=<version-from-ci> REGISTRY_REPO="oci://127.0.0.1:5001/charts" set-kcm-version test-e2e

    Filtering Tests

    Use GINKGO_LABEL_FILTER to run specific subsets of tests (e.g., only AWS tests) to save time. To see all available labels, run ginkgo labels ./test/e2e.

    Example for AWS only:

    make GINKGO_LABEL_FILTER="provider:aws" [OTHER_VARIABLES] test-e2e
    # Example: Running non-hosted tests locally
make KIND_CLUSTER_NAME="kcm-test" CLUSTER_DEPLOYMENT_PREFIX="my-prefix" GINKGO_LABEL_FILTER="" REGISTRY_REPO="oci://127.0.0.1:5001/charts" VERSION=1.0.0 IMG=localhost/kcm/controller:1.0.0 IMG_TELEMETRY=localhost/kcm/telemetry:1.0.0 kind-deploy registry-deploy set-kcm-version dev-push test-e2e

  8. Configure OpenStack credentials via clouds.yaml

    main

    CAPO requires a clouds.yaml file provided as a Kubernetes Secret. Typically, you create a Secret (e.g., openstack-cloud-config) in the kcm-system namespace.

    When a new cluster is deployed, KCM parses this Secret to build a cloud.conf file, which is then mounted inside the CCM and/or CSI pods. This allows them to manage resources like load balancers and floating IPs.

    clouds:
  my-openstack-cloud:
    auth:
      auth_url: <your_auth_url>
      application_credential_id: <your_credential_id>
      application_credential_secret: <your_credential_secret>
    region_name: <your_region>
    interface: <public|internal|admin>
    identity_api_version: 3
    auth_type: v3applicationcredential

  9. Cleanup KCM installation

    main

    To completely remove KCM and its associated resources, follow these steps in order. Warning: Ensure all ClusterDeployment objects are deleted before proceeding with Management deletion.

    1. Delete the Management object:
      kubectl delete management.k0rdent kcm
    2. Uninstall the Helm release:
      helm uninstall kcm -n kcm-system
    3. Remove the namespace:
      kubectl delete ns kcm-system
    kubectl delete management.k0rdent kcm
helm uninstall kcm -n kcm-system
kubectl delete ns kcm-system

  10. Create a ClusterDeployment

    main

    Follow these steps to deploy a new Kubernetes cluster using KCM:

    1. Prepare Credentials: Create a Credential object containing the necessary access keys. Refer to the Credential system docs for details.
    2. Select a Template: Identify the ClusterTemplate you wish to use. You can list available templates with:
      kubectl get clustertemplate -n kcm-system
    3. Define the Deployment: Create a ClusterDeployment YAML file. Note: Use dryRun: true initially to validate your configuration without provisioning infrastructure.
    4. Apply the Configuration:
      kubectl create -f clusterdeployment.yaml
    5. Monitor Progress: Check the status of the deployment:
      kubectl -n <clusterdeployment-namespace> get clusterdeployments <clusterdeployment-name> -o=yaml
      Or watch the provisioning process using clusterctl:
      clusterctl describe cluster <clusterdeployment-name> -n <clusterdeployment-namespace> --show-conditions all
    6. Retrieve Kubeconfig: Once the cluster is ready, extract the kubeconfig:
      kubectl get secret -n kcm-system <clusterdeployment-name>-kubeconfig -o go-template='{{.data.value|base64decode}}' > kubeconfig
    apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: <cluster-name>
  namespace: <cluster-namespace>
spec:
  template: <template-name>
  credential: <credential-name>
  dryRun: <true/false>
  config:
    <cluster-configuration>

  11. Install KCM for development

    main

    To set up a development environment for KCM, clone the repository and install the required CLI tools using make. This setup allows you to create managed clusters on various providers (AWS, Azure, vSphere, etc.) for testing purposes.

    # Clone and enter the repository
git clone https://github.com/K0rdent/kcm.git && cd kcm

# Install required CLIs
make cli-install

  12. Adopt an existing cluster for development

    main

    To use an existing cluster instead of provisioning a new one, set DEV_PROVIDER to adopted and export the cluster's kubeconfig as a base64 encoded string in the KUBECONFIG_DATA environment variable.

    export DEV_PROVIDER=adopted
export KUBECONFIG_DATA=$(cat kubeconfig | base64 -w 0)