repository·main·Indexed 27 days ago
https://github.com/aws/karpenter-provider-awsAn open-source Kubernetes node provisioning project that automates scaling based on workload requirements to optimize performance and cost. It manages the node lifecycle by evaluating pod scheduling constraints and provisioning nodes that satisfy those requirements. The project includes OCI Helm charts for installation, support for AWS Launch Templates, and tools such as allocatable-diff, kompat, and kubereplay.
Karpenter is an open-source node provisioning project for Kubernetes designed to improve cluster efficiency and cost. It automates the node lifecycle by:
nodeSelector, affinities, tolerations, and topology spread constraints).One proposed solution for managing node ownership and in-flight capacity is to use a Machine CRD as a representation of a scheduling decision. In this model, Karpenter creates a Machine CR instead of a Node object or launching instances directly during the provisioning loop.
Key behaviors of the Machine CRD model:
CreateFleet response into the Machine's status.spec.providerID with the Machine's status.providerID.karpenter.sh/termination) is migrated to the Machine to orchestrate cordoning, draining, and deletion. For compatibility, the Node finalizer remains but checks for the presence of a Machine and triggers Machine deletion if found..metadata.labels and .status.allocatable until the actual Node registers and initializes its own .status.allocatable.When working with or extending Karpenter metrics, keep the following architectural design choices in mind:
Karpenter's security model is defined by three primary actors with distinct responsibilities and permission levels:
NodePools and EC2NodeClasses, and manage the cloud identities (IAM) and permissions for both Karpenter and the resulting Nodes.Deployments, ReplicaSets, StatefulSets, or Jobs. They should not have permissions to modify the Karpenter deployment, pods, or service accounts.Karpenter's current model of creating Node objects immediately after launching VM instances leads to several operational challenges. Understanding these issues is critical for troubleshooting provisioning behavior:
Karpenter must create the Node object to capture in-flight details (node name, allocatable resources, capacity) for its scheduler. This creates a dependency on the cloud provider API or the kubelet's registration speed, which can slow down provisioning. If resolution fails, Karpenter may lack capacity knowledge, leading to over-provisioning.
ownerReference to link nodes to their Provisioners. If a Provisioner is deleted during a provisioning loop after the VM has launched but before the ownerReference is reconciled, the resulting nodes and VM instances will be orphaned.Because Karpenter creates nodes without initial capacity details, Kubernetes 1.23+ control planes may report NotReady or unhealthy events due to insufficient image filesystem capacity. These errors typically resolve once the kubelet begins reporting ephemeral-storage.
When Karpenter (or any third party) registers a node, the kubelet only updates a fixed set of labels. This prevents users from applying dynamic labels via KubeletConfig or userData in AWS, as the kubelet's updates may be ignored or overwritten.
Karpenter is migrating its node ownership model to use a Machine CRD to represent scheduling decisions. This CRD acts as the primary record for in-flight nodes before they register as Kubernetes Nodes.
labels, annotations, and taints (including startupTaints). This state is reconciled onto the Kubernetes Node once the Node registers.Machine resources on the cluster, rather than waiting for the Node to appear in the API server.userData) without interfering with Karpenter's ability to reconcile labels, annotations, and taints.kube-apiserver QPS and etcd storage is expected to be negligible. The model performs approximately 2 write operations per created node, which is comparable to or less than the current Node-based reconciliation flow.Karpenter treats pod preferences (node affinity, pod affinity, pod anti-affinity, and pod topology) as requirements when determining if a pod can schedule on a node or be shifted to a new node.
When constructing requirements, Karpenter initially treats preferred affinities as required. If these cannot be met, Karpenter relaxes the preferences one-at-a-time in ascending order of weight (lowest weight relaxed first) and retries.
Warning: Karpenter does not interpret preferred affinities as required when constructing topology requirements. If a preference is strictly necessary, you must use required affinities.
An alternative solution for managing node ownership is to use an In-Memory Node Store that relies on the cloud provider's instance data rather than the Kubernetes API server for in-flight capacity.
Key behaviors of the In-Memory Store model:
ListMachines() method is added to the CloudProvider interface to re-populate Karpenter-owned instances onto the API server (similar to the Kubernetes List/Watch mechanism).userData does not support annotations, scheduling decisions (labels, taints, and annotations) must be stored in cloud provider tags.Karpenter interacts with core Kubernetes components as follows:
Karpenter's NodePools are designed to coexist with static capacity management solutions such as EKS Managed Node Groups and EC2 Auto Scaling Groups. Users can adopt several management models:
NodePools.NodePool-managed) and static (Managed Node Groups/ASGs) capacity.Most users are expected to use a mixed approach initially, transitioning to full NodePool management over time.
NodeClaims are immutable, cluster-scoped resources used by Karpenter to manage the lifecycle of Kubernetes Nodes with the underlying cloud provider. They serve as requests for capacity and map one-to-one with a cloud provider instance and a Kubernetes Node.
Key characteristics of NodeClaims:
NodePool and references a single NodeClass.NodePool name as a prefix.NodePool template. Karpenter also applies a termination finalizer to ensure proper cleanup.