DOCSEARCHbyAlgolia
Submit Docs

Kubernetes Descheduler

repository·master·Indexed 26 days ago

https://github.com/kubernetes-sigs/descheduler

A Kubernetes tool that improves cluster health by evicting pods that are no longer optimally placed according to defined policies. It includes a Descheduling Framework (KEP-753) that allows developers to implement custom out-of-tree strategies and plugins via interfaces such as PreSortPlugin, DeschedulePlugin, BalancePlugin, EvictPlugin, and SortPlugin. The tool can be installed via Helm as a CronJob or Deployment and configured using DeschedulerPolicy objects.

Tokens
19.9K
Snippets
46
Records
84
Agent score
87%

What's inside Descheduler

  1. Overview of Descheduler for Kubernetes

    master

    Descheduler is a tool for Kubernetes clusters that identifies pods that can be moved to more desirable nodes based on a configurable policy and evicts them. This helps address issues such as:

    • Under or over-utilized nodes.
    • Changes in node taints or labels that invalidate original scheduling decisions.
    • Pod/node affinity requirements no longer being satisfied.
    • Node failures causing pod redistribution.
    • New nodes being added to the cluster.

    Note: Descheduler evicts pods but does not schedule their replacements; it relies on the default Kubernetes scheduler to handle the rescheduling of evicted pods.

  2. Understand the Descheduling Cycle Workflow

    master

    The descheduling framework follows a specific execution order during a cycle:

    1. Capture State: The current cluster state is captured.
    2. Deschedule Phase: All Deschedule extension points from all profiles are executed. These plugins process pods independently.
    3. Balance Phase: All Balance extension points from all profiles are executed. These plugins focus on distribution and balancing.

    Important Notes for Developers:

    • Low-level extension points: PreSort, Filter, and Sort are not called directly by the framework. Instead, they must be invoked inside the implementation of Deschedule or Balance plugins. This allows strategy owners to decide exactly when to apply filtering or sorting.
    • Concurrency: There is no parallelism between plugins; only one plugin executes at a time.
    • Eviction Limits: If a limit on the number of evicted pods (per node, namespace, or cluster) is reached, the descheduling cycle terminates prematurely.

  3. Customization Points for Descheduler Developers

    master

    Developers looking to extend the Descheduler can target several key customization points:

    • Custom Strategies: Implement individual eviction algorithms. This is the primary entry point for custom behavior.
    • Custom Pod/Node/Resource Logic: Extend or abstract internal types like PodEvictor, PodLister, and NodeLister. These control the filtering and sorting of Kubernetes resources. For example, you could implement logic to control pod eviction based on the availability of PVCs on a node.
    • Extending the Config API: Custom features will require custom configuration options. The Descheduler API is being redesigned to be extensible for generic additions while maintaining stability.
    • Event-triggered Evictions: While the Descheduler typically runs on a periodic interval (via Deployment or CronJob), there is a move toward making it more reactive by triggering evictions based on cluster events.

  4. Understand the Descheduling Framework (KEP-753)

    master

    The Descheduler is transitioning from a centralized component to a developer-facing, importable library known as the Descheduling Framework. This framework is designed to provide a stable platform for writing custom pod eviction logic, similar in concept to the Kubernetes Scheduler Framework.

    Key objectives of the framework include:

    • API Stability: Allowing third-party developers to implement custom features without breaking existing behaviors.
    • External Development: Enabling decentralized development of new descheduling features.
    • Strategy Ownership: Allowing third-party providers to specialize in and maintain their own features independently of the core Descheduler maintainers.

  5. Configure High Availability (HA) mode

    master

    To run multiple replicas of the descheduler (e.g., in a Deployment) without them running simultaneously, you must enable High Availability mode using Kubernetes leader election.

    How to enable HA:

    • CLI: Set the --leader-elect flag.
    • Helm: Set the flag --set=leaderElection.enabled=true.

    Best Practices for HA:

    • Set the replica count to greater than 1.
    • Consider configuring a podAntiAffinity rule to ensure descheduler pods are scheduled across different nodes/zones.

  6. Run unit and end-to-end tests

    master

    Testing requires a local Kubernetes cluster (via kind) and a Linux environment for the dev image. Follow these steps to build the environment and run tests:

    1. Build the Linux dev image.
    2. Create a multi-node kind cluster.
    3. Load the docker image into kind.
    4. Configure KUBECONFIG to point to the kind cluster.
    5. Execute unit and e2e tests using make.
    GOOS=linux make dev-image
make kind-multi-node
kind load docker-image <image name>
kind get kubeconfig > /tmp/admin.conf
export KUBECONFIG=/tmp/admin.conf
make test-unit
make test-e2e

  7. Select documentation version for your Descheduler release

    master

    To ensure compatibility, always use the documentation corresponding to your specific Descheduler version. If you are using a published release (e.g., registry.k8s.io/descheduler/descheduler:v0.36.0), refer to the documentation in that version's release branch.

    Descheduler VersionDocs link
    v0.36.xrelease-1.36
    v0.35.xrelease-1.35
    v0.34.xrelease-1.34
    v0.33.xrelease-1.33
    v0.32.xrelease-1.32
    v0.31.xrelease-1.31
    v0.30.xrelease-1.30

    The master branch is in-development and may contain information that is not compatible with previous versions.

  8. Load Descheduler image into a kind cluster

    master

    Because multi-arch container images cannot be pulled directly by kind from a registry, you must manually pull the image to your local Docker daemon and then load it into the cluster.

    kind create cluster
docker pull registry.k8s.io/descheduler/descheduler:v0.20.0
kind load docker-image registry.k8s.io/descheduler/descheduler:v0.20.0

  9. Install Descheduler using Kustomize

    master

    You can use kustomize build to install Descheduler. Replace release-1.34 with the version tag relevant to your deployment.

    Run as a Job:

    kustomize build 'github.com/kubernetes-sigs/descheduler/kubernetes/job?ref=release-1.34' | kubectl apply -f -

    Run as a CronJob:

    kustomize build 'github.com/kubernetes-sigs/descheduler/kubernetes/cronjob?ref=release-1.34' | kubectl apply -f -

    Run as a Deployment:

    kustomize build 'github.com/kubernetes-sigs/descheduler/kubernetes/deployment?ref=release-1.34' | kubectl apply -f -
    kustomize build 'github.com/kubernetes-sigs/descheduler/kubernetes/job?ref=release-1.34' | kubectl apply -f -
kustomize build 'github.com/kubernetes-sigs/descheduler/kubernetes/cronjob?ref=release-1.34' | kubectl apply -f -
kustomize build 'github.com/kubernetes-sigs/descheduler/kubernetes/deployment?ref=release-1.34' | kubectl apply -f -

  12. Build and run the descheduler binary

    master

    To build the descheduler from source, use the make command. Once built, you can run the binary by providing a path to your kubeconfig and your policy configuration file. Use the --v 1 flag to enable logging.

    # Build
cd descheduler
make

# Run
./_output/bin/descheduler --client-connection-kubeconfig <path to kubeconfig> --policy-config-file <path-to-policy-file> --v 1