Cluster Capacity Analysis Framework

The framework estimates how many instances of a specific pod can be scheduled in a cluster. It uses a scheduler (created by a scheduler factory) and a loopback fake client that intercepts requests and operates over local caches.

The Estimation Lifecycle:

1. Capture State: The framework captures the current state of the cluster.
2. Local Simulation: It translates API requests into local cache operations. This ensures the actual running cluster is not modified by the simulated scheduling.
3. Estimation: The framework runs the scheduling logic against the local cache and provides the results via available channels.

Key Assumptions:

• The pod is assumed to run immediately after scheduling.
• There is no interaction with Kubelet (no container running, image pulling, or volume handling).
• There is no interaction with the Apiserver (all interactions use local caches).
• There is no interaction with controllers.

You can run cluster-capacity as a Kubernetes Job to allow for automated, repeated analysis without manual intervention.

Prerequisites

1. Build a container image: Use the provided Dockerfile in the root directory.

   $ docker build -t cluster-capacity-image .

2. Setup RBAC: Apply the necessary permissions using the provided configuration.

   $ kubectl apply -f config/rbac.yaml

3. Prepare the Pod Spec: Create a ConfigMap containing the pod specification file you want to analyze.

   $ kubectl create configmap cluster-capacity-configmap --from-file pod.yaml

Job Configuration

When defining the Job, you must set the environment variable CC_INCLUSTER to true. This informs the tool that it is running inside a cluster as a pod.

Example Job specification (cluster-capacity-job.yaml):

apiVersion: batch/v1
kind: Job
metadata:
  name: cluster-capacity-job
spec:
  parallelism: 1
  completions: 1
  template:
    metadata:
      name: cluster-capacity-pod
    spec:
        containers:
        - name: cluster-capacity
          image: cluster-capacity-image
          imagePullPolicy: "Never"
          volumeMounts:
          - mountPath: /test-pod
            name: test-volume
          env:
          - name: CC_INCLUSTER
            value: "true"
          command:
          - "/bin/sh"
          - "-ec"
          - "/bin/cluster-capacity --podspec=/test-pod/pod.yaml --verbose"
        restartPolicy: "Never"
        serviceAccountName: cluster-capacity-sa
        volumes:
        - name: test-volume
          configMap:
            name: cluster-capacity-configmap

Execution

Run the job with:

$ kubectl create -f cluster-capacity-job.yaml

Check the results in the job logs:

$ kubectl logs jobs/cluster-capacity-job

Namespaces can have limitations like limit ranges, resource quota, or node selectors that affect schedulability. To account for these, the pod specification must be extended (e.g., setting spec.nodeSelector or spec.containers[i].resources).

The genpod binary automates this by generating an extended pod specification based on the current limit ranges and node selectors of a specific namespace.

Step 1: Generate the pod spec

$ genpod --kubeconfig <path to kubeconfig> --master <API server address> --namespace <NAMESPACE>

Step 2: Run analysis with the generated spec Use the generated file (e.g., genpod.yaml) and set the --resource-space-mode to ResourceSpacePartial to include admission controller constraints like quotas.

$ genpod --kubeconfig <path to kubeconfig> --master <API server address> --namespace NAMESPACE

$ ./cluster-capacity --kubeconfig <path to kubeconfig> --master <API server address> --podspec=genpod.yaml --apiserver-config config/apiserver.yaml --verbose --resource-space-mode ResourceSpacePartial

If you do not have direct access to a kubeconfig (e.g., you only have web UI access), you can run the framework as a pod within the cluster.

The following example demonstrates a pod that uses the genpod binary to create a specification for the cluster-capacity namespace and then executes the framework.

Note: Once the framework pod is running, you can update the pod specification for the estimation by sending a POST request.

apiVersion: v1
kind: Pod
metadata:
  name: cluster-capacity
  labels:
    name: cluster-capacity
spec:
  containers:
  - name: cluster-capacity
    image: docker.io/gofed/cluster-capacity:latest
    command:
    - "/bin/sh"
    - "-ec"
    - |
      echo "Generating pod"
      /bin/genpod --namespace=cluster-capacity >> /pod.yaml
      cat /pod.yaml
      echo "Running cluster capacity framework"
      /bin/cluster-capacity --period=1 --podspec=/pod.yaml --default-config /config/default-scheduler.yaml
    ports:
    - containerPort: 8081

Configuration for the framework is split into two categories:

Analysis Configuration

Used to define how the estimation is performed:

• kubeconfig: Path to the kubeconfig file.
• apiserver-config: Address of the running Apiserver.
• podspec: The pod specification to be scheduled.
• maxLimit: The number of pod instances that, when reached, cause the analysis to stop prematurely.
• period: The time interval between consecutive analysis runs.
• resource-space-mode: The mode of resource space exploration.

Scheduler Configuration

The framework expects a scheduler created by the default scheduler factory. By default, it looks for a configuration file at config/default-scheduler.yaml. To match the behavior of your actual cluster, it is recommended to use the same scheduler configuration, including enabled predicates and priority functions.

port: 10251
address: 0.0.0.0
algorithmprovider: DefaultProvider
policyconfigfile: ""
enableprofiling: false
contenttype: application/vnd.kubernetes.protobuf
kubeapiqps: 50
kubeapiburst: 100
schedulername: default-scheduler
hardpodaffinitysymmetricweight: 1
failuredomains: kubernetes.io/hostname,topology.kubernetes.io/zone,topology.kubernetes.io/region
leaderelection:
  leaderelect: true
  leaseduration:
    duration: 15s
  renewdeadline:
    duration: 10s
  retryperiod:
    duration: 2s