DOCSEARCHbyAlgolia
Submit Docs

KWOK (Kubernetes WithOut Kubelet)

repository·main·Indexed 25 days ago

https://github.com/kubernetes-sigs/kwok

A toolkit for simulating large-scale Kubernetes clusters by emulating Kubernetes API resource lifecycles without actual Kubelets. It includes the core kwok engine for simulating nodes and pods, and the kwokctl CLI for managing clusters using runtimes like kind or compose. Key features include pod and node emulation via stage policies, resource usage simulation using annotations and Metrics Resources, and specialized stages for scheduling and autoscaling tests.

Tokens
33.2K
Snippets
83
Records
203
Agent score
85%

What's inside KWOK

  1. Overview of KWOK design goals

    main

    KWOK (Kubernetes Without Kubelet) is designed to simulate Kubernetes clusters with low cost and high fidelity. It is intended to be used as a real cluster for various testing scenarios, including:

    • Scheduler Testing: Testing schedulers with a large number of Nodes and Pods.
    • CRD Controller Testing: Testing custom controllers using real API interactions instead of fake clients.
    • Control Plane Performance Testing: Testing the performance and scalability of the control plane at a low resource cost.

    Key characteristics of KWOK include:

    • Low-cost simulation of any cluster.
    • High fidelity (uses the cluster like a real one).
    • Customizable emulation.
    • Environment agnostic (runs in any environment).
    • Fast startup and ease of use.

  2. Overview of KWOK (Kubernetes WithOut Kubelet)

    main

    KWOK (Kubernetes WithOut Kubelet) is a toolkit designed to simulate the lifecycle of fake nodes, pods, and other Kubernetes API resources. It allows developers to create clusters with thousands of nodes in seconds with a very low resource footprint.

    Core Components

    • kwok: The core engine responsible for simulating the lifecycle of fake Kubernetes API resources.
    • kwokctl: A CLI tool used to streamline the creation and management of clusters where nodes are simulated by kwok.

    Key Use Cases

    • Scaling Tests: Measure how applications or controllers scale with large numbers of nodes and pods.
    • Load Generation: Generate high loads by creating many pods or services with varying resource requests/limits.
    • Failure Simulation: Simulate node failures or network partitions by modifying node conditions or deleting nodes.
    • Development & Learning: Test Kubernetes controllers, features, or API versions without the overhead of real infrastructure.

  3. Identify the KWOK core components

    main

    KWOK consists of two primary tools:

    1. kwok: The core engine responsible for simulating the lifecycle of fake nodes, pods, and other Kubernetes API resources.
    2. kwokctl: A CLI tool used to streamline the creation and management of clusters where nodes are simulated by kwok.

  4. Understand the `kwok` resource controller

    main

    kwok is a resource controller designed to simulate the lifecycle of fake nodes, pods, and other Kubernetes API resources. It functions similarly to kube-controller-manager and can run in any environment, connecting to a kube-apiserver via in-cluster configuration or a kubeconfig file.

    kwok implements several types of controllers:

    • Resource Lifecycle Simulation Controllers: These simulate the lifecycle of Kubernetes resources (typically by updating status fields that a real kubelet would normally report).
      • Node Controller: Selects nodes to simulate and updates their status fields.
      • Pod Controller: Simulates the lifecycle of pods scheduled to selected nodes by updating pod status fields.
      • Stage Controller: A generalized controller used to simulate the lifecycle of Kubernetes resource types other than v1.Node and v1.Pod.
    • Node Lease Controller: A dedicated controller that reports node heartbeats by creating and renewing node lease objects for managed nodes.

  6. Scale clusters using Cluster Autoscaler with KWOK

    main

    You can use KWOK to simulate cluster autoscaling scenarios by deploying a metric server. This allows you to observe and trigger a Cluster Autoscaler (CA) within a KWOK-simulated environment.

    To implement this, you need to use the KWOK provider for the Cluster Autoscaler. Detailed implementation details and the provider source can be found in the official Cluster Autoscaler repository.

    https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/kwok

  7. Deploy Istio on KWOK

    main

    This guide demonstrates how to run Istio on a KWOK cluster by simulating nodes and then migrating Istio controllers to a real node to ensure they function correctly within the simulated environment.

    1. Set up Cluster

    Create a KWOK cluster using the kind runtime:

    kwokctl create cluster --runtime kind

    2. Create Node

    Apply a KWOK node definition to the cluster:

    kubectl apply -f https://kwok.sigs.k8s.io/examples/node.yaml

    3. Deploy Istio

    Install Istio using istioctl:

    istioctl install -y

    4. Migrate Controllers to Real Node

    To ensure Istio controllers function properly, patch the istiod and istio-ingressgateway deployments to pin them to a real node (e.g., kwok-kwok-control-plane):

    kubectl patch deploy istiod -n istio-system --type=json -p='[{"op":"add","path":"/spec/template/spec/nodeName","value":"kwok-kwok-control-plane"}]'
kubectl patch deploy istio-ingressgateway -n istio-system --type=json -p='[{"op":"add","path":"/spec/template/spec/nodeName","value":"kwok-kwok-control-plane"}]'

    5. Create Pod and Inject Sidecar

    Enable Istio sidecar injection for the default namespace and deploy a sample application (Bookinfo):

    kubectl label namespace default istio-injection=enabled
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo.yaml
    # Set up Cluster
kwokctl create cluster --runtime kind

# Create Node
kubectl apply -f https://kwok.sigs.k8s.io/examples/node.yaml

# Deploy Istio
istioctl install -y

# Migrate Controllers to Real Node
kubectl patch deploy istiod -n istio-system --type=json -p='[{"op":"add","path":"/spec/template/spec/nodeName","value":"kwok-kwok-control-plane"}]'
kubectl patch deploy istio-ingressgateway -n istio-system --type=json -p='[{"op":"add","path":"/spec/template/spec/nodeName","value":"kwok-kwok-control-plane"}]'

# Create Pod and Inject Sidecar
kubectl label namespace default istio-injection=enabled
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo.yaml

  9. Scale custom resources into a specific namespace

    main

    When scaling custom resources, you can target a specific namespace using the --namespace flag. Ensure the target namespace exists in the cluster before running the scale command.

    kubectl create namespace my-namespace
kwokctl scale deployment --replicas 5 --namespace my-namespace --param '.replicas=3' --config ./deployment-resource.yaml

  10. Install KWOK using Golang

    main

    Use go install to install the latest version of kwok and kwokctl directly from the repository. This method requires jq to parse the latest release tag from the GitHub API.

    # KWOK repository
KWOK_REPO=kubernetes-sigs/kwok
# Get latest
KWOK_LATEST_RELEASE=$(curl "https://api.github.com/repos/${KWOK_REPO}/releases/latest" | jq -r '.tag_name')

go install sigs.k8s.io/kwok/cmd/{kwok,kwokctl}@${KWOK_LATEST_RELEASE}

  11. Simulate autoscaling scenarios with KWOK

    main

    KWOK can be used to simulate various Kubernetes autoscaling scenarios to test system behavior without requiring actual heavy workloads or physical infrastructure. Supported scenarios include:

    • Horizontal Pod Autoscaling (HPA): Testing how the cluster responds to changes in pod replica counts.
    • Vertical Pod Autoscaling (VPA): Testing how the cluster responds to changes in pod resource requests/limits.
    • Cluster Autoscaling (CA): Testing how the cluster responds to node scaling requirements.
    • Karpenter Autoscaling: Testing Karpenter-specific provisioning and scaling logic.