Karpenter Documentation

Karpenter is a node autoscaler designed to provision compute capacity for pods and remove unused capacity. To maintain a simple and stable API, the project follows specific boundaries regarding its responsibilities:

What Karpenter Does

• Provisions compute capacity to allow pods to run.
• Removes capacity that is no longer needed.
• Emits Kubernetes events, status conditions, and Prometheus metrics.
• Provides configuration to express desired behavior (e.g., consolidation policies or disruption budgets).

What Karpenter Does NOT Do (Out-of-Scope)

• Notification Delivery: Karpenter does not send webhooks, Slack, email, or PagerDuty alerts. You must use external tooling to consume its emitted signals.
• Pod Scheduling: Karpenter provisions nodes, but it does not decide which pod runs on which node; that is the responsibility of the kube-scheduler.
• Workload Management: Karpenter does not scale pods or manage workload replicas. Use tools like HPA, KEDA, or VPA for workload lifecycle management.
• Provider-Specific Logic: The kubernetes-sigs/karpenter repository is provider-neutral. Cloud-specific features (e.g., AWS, Azure) are located in their respective provider repositories.
• Internal Implementation Exposure: Karpenter configuration is designed for users to express what they want (desired state), not how the internal mechanics (like thread concurrency) should operate.

A NodeOverlay is a Karpenter API resource used to inject alternative assumptions into the scheduling simulation. By default, Karpenter relies on cloud provider APIs for instance type offerings and base prices. NodeOverlay allows users to fine-tune these simulations for advanced use cases that the standard API cannot express, such as custom pricing models or extended resource capacities.

Use Cases

• Complex Pricing: Adjusting for vendor fees, negotiated discounts, capacity type discounts, or carbon-offset costs.
• Resource Capacity Management: Accounting for extended resources (like GPUs or HugePages) or custom system overhead that isn't captured by standard CPU/Memory metrics.

How it works

NodeOverlay uses a requirements field to match specific instance types or NodePools. When multiple overlays match an offering, they are merged using spec.weight to resolve conflicts. Changes to price, priceAdjustment, or capacity are integrated into Karpenter's scheduling and consolidation cycles automatically.

apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
  name: default
spec:
  weight: ...
  requirements:
    ... 
  price: 
  priceAdjustment: ... 
  capacity: 
    ...
status:
  condition:
  - lastTransitionTime: "..."
    message: ""
    reason: ...
    status: "True"
    type: Ready

If multiple NodeOverlay resources have the same weight and conflicting configurations, Karpenter follows a Fail Open approach to maintain system availability:

1. Alphabetical Resolution: Conflicts are resolved using alphabetical ordering of the overlay names.
2. Status Reporting: The overlay's Ready status condition is set to False with a reason of conflict and a message indicating which overlay it conflicted with.
3. Operational Continuity: Karpenter continues provisioning and consolidation operations even if the overlay configurations do not match the user's intent.

Note: If an unapplied overlay defines extended resources (like HugePages), pods requiring those resources will fail to provision capacity, consistent with standard Karpenter behavior when required resources are unavailable.

kind: NodeOverlay
metadata:
  name: memory-1
spec:
  weight: 90
  requirements:
  - key: karpenter.k8s.aws/instance-memory
    operator: Gt
    values: ["m5.large", "m5.2xlarge"]
  capacity:
    memory: 50Mi
status:
  condition:
  - lastTransitionTime: "..."
    message: "conflict with overlay capacity-1"
    reason: "conflict"
    status: "False"
    type: Ready
---
kind: NodeOverlay
metadata:
  name: capacity-1
spec:
  weight: 90
  requirements:
  - key: node.kubernetes.io/instance-type
    operator: In
    values: ["c5.large", "c5.2xlarge", "m5.2xlarge"]
  capacity:
    memory: 100Mi
status:
  condition:
  - lastTransitionTime: "..."
    message: ""
    reason: ...
    status: "True"
    type: Ready

Karpenter can use dry-run pod creation to determine what a pod's resources will be after recreation (e.g., after a VPA update). This method uses the Kubernetes API server's admission chain (POST /api/v1/namespaces/{ns}/pods?dryRun=All) to simulate pod creation without persisting changes.

To use this mechanism, pods or their templates must be annotated to signal that their requests are volatile.

Workflow:

1. Annotation: Apply autoscaling.k8s.io/volatile-requests: "true" to pods or templates.
2. Simulation: A background controller identifies annotated pods, groups them by owner (e.g., ReplicaSet), and submits dry-run creates.
3. Caching: The predicted Ceiling() result is cached, keyed by (namespace, owner-uid). Karpenter uses this cache during consolidation simulation.
4. Validation: Before eviction, Karpenter re-runs a fresh dry-run. If the result differs from the simulation, the decision is rejected.

Requirements:

• Requires create pods RBAC cluster-wide (note: RBAC does not distinguish between dry-run and real creates).
• Pods must have an owner (e.g., ReplicaSet) to be processed; unowned pods are skipped.