Kamaji Documentation

Kamaji uses a 'Tenant Control Plane' model to manage Kubernetes clusters at scale. Instead of provisioning dedicated machines for every cluster, Kamaji runs the control plane components (kube-apiserver, kube-scheduler, and kube-controller-manager) as pods within a central Management Cluster.

Key Characteristics

• Upstream Compatibility: Control plane components are standard CNCF-compliant Kubernetes binaries. Kamaji uses kubeadm for setup and lifecycle management.
• Resource Efficiency: Running control planes as pods allows for high-density multi-tenancy on shared infrastructure.
• High Availability: Control plane pods are managed via Kubernetes Deployments, providing self-healing, rolling updates, and autoscaling capabilities.
• Declarative Management: Clusters are managed via the TenantControlPlane custom resource, making it suitable for GitOps and Infrastructure as Code (IaC) workflows.

Kamaji is a Kubernetes Control Plane Manager that implements the Hosted Control Plane concept. Instead of running Kubernetes Control Plane components on dedicated machines, Kamaji runs them as Pods within a management cluster.

Core Abstractions

Kamaji extends the Kubernetes API using Custom Resource Definitions (CRDs) to manage two primary resources:

1. TenantControlPlane (short-name: tcp): A Namespace-scoped resource that defines the desired state of a specific Kubernetes Control Plane. It allows you to configure:

   • Kubernetes configuration values.
   • Pod options (limits, requests, tolerations, node selectors, etc.).
   • Exposure methods (e.g., ClusterIP, LoadBalancer, or NodePort).
   • Core addons like kube-proxy, CoreDNS, and konnectivity.

2. Datastore: A cluster-scoped resource that acts as the backing store for one or more TenantControlPlane instances. It manages the state of the managed clusters.

Key Benefits

• Resource Optimization: Decoupling the control plane from the datastore allows for high density and reduced hardware requirements (up to 60% savings).
• Fast Provisioning & Updates: Control planes can be ready in ~16 seconds, and updates use Blue/Green deployment to ensure zero downtime and avoid mixed versions.
• Automated Management: Handles certificate rotation via kubeadm, manages core addons, and provides auto-healing for the TenantControlPlane objects.

Kamaji allows for modular template architecture. You can host shared templates (like KamajiControlPlaneTemplate and KubeadmConfigTemplate) in a central namespace and reference them in provider-specific ClusterClass resources. This ensures that the control plane logic remains consistent while only the infrastructure templates (AWS, Azure, etc.) change.

# Example: AWS Cluster Class referencing shared Kamaji templates
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
  name: kamaji-aws-class
spec:
  controlPlane:
    ref:
      apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
      kind: KamajiControlPlaneTemplate
      name: kamaji-controlplane
      namespace: cluster-templates  # Shared template
  
  infrastructure:
    ref:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      kind: AWSClusterTemplate
      name: aws-cluster-template  # AWS-specific
  
  workers:
    machineDeployments:
    - class: default-worker
      template:
        bootstrap:
          ref:
            apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
            kind: KubeadmConfigTemplate
            name: kubeadm
            namespace: cluster-templates  # Shared template
        infrastructure:
          ref:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
            kind: AWSMachineTemplate
            name: aws-worker-template  # AWS-specific

The ClusterClass allows for highly flexible cluster deployments through two mechanisms:

Variable System

Variables define parameters that can be customized during cluster creation. Each variable includes an OpenAPI v3 schema for validation, requirement status, default values, and type constraints (e.g., minimum, maximum, enum).

JSON Patching System

Patches apply the values provided via variables to the base templates at cluster creation time.

• Standard Patching: Uses jsonPatches with operations like replace or add to map a variable to a specific path in a template.
• Conditional Patching: Uses the enabledIf field to apply a patch only if a specific condition is met (e.g., if a string is not empty).

This allows a single ClusterClass to serve multiple use cases (e.g., small vs. large worker nodes) by simply changing the input variables.

# Example of a variable definition
variables:
- name: machineSpecs
  required: true
  schema:
    openAPIV3Schema:
      type: object
      properties:
        numCPUs:
          type: integer
          minimum: 2
          maximum: 64
          default: 4

# Example of a JSON patch applying a variable
patches:
- name: machineResources
  definitions:
  - selector:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: VSphereMachineTemplate
      matchResources:
        machineDeploymentClass:
          names: ["default-worker"]
    jsonPatches:
    - op: add
      path: /spec/template/spec/numCPUs
      valueFrom:
        variable: machineSpecs.numCPUs