DOCSEARCHbyAlgolia
Submit Docs

AWS Node Termination Handler

repository·main·Indexed 23 days ago

https://github.com/aws/aws-node-termination-handler

AWS Node Termination Handler (NTH) manages the graceful shutdown of Kubernetes nodes in response to AWS EC2 termination events, such as Spot interruptions or maintenance. It ensures workloads are drained and cordoned before instances disappear. The tool supports IMDS and Queue Processor modes and can be deployed via a Helm chart for Kubernetes versions v1.16 and above.

Tokens
18.8K
Snippets
17
Records
48
Agent score
82%

What's inside aws-node-termination-handler

  1. Overview of AWS Node Termination Handler

    main

    The AWS Node Termination Handler (NTH) ensures that the Kubernetes control plane responds gracefully to events that cause EC2 instances to become unavailable. This prevents application code from failing to stop gracefully, delays in recovering availability, or scheduling work on nodes that are about to be terminated.

    NTH handles the following event types:

    • EC2 maintenance events
    • EC2 Spot interruptions
    • Auto Scaling Group (ASG) Scale-In events
    • ASG AZ Rebalance events
    • EC2 Instance Termination via API or Console

    When a termination event is detected, NTH uses the Kubernetes API to cordon the node (preventing new work from being scheduled) and drain the node (removing existing work).

  2. Scaling the Queue Processor

    main

    By default, the Helm chart deploys a single instance of the AWS Node Termination Handler. While a single instance is usually sufficient for responsiveness, you may consider deploying multiple replicas in the following scenarios:

    • Responsiveness: If NTH is taking too long to process numerous concurrent events or drain Pods.
    • Availability: To mitigate delays if the NTH deployment itself is being drained or restarted.

    Important Considerations for Multiple Replicas:

    • No Load Balancing: Instances do not load balance; each instance will greedily consume and respond to events.
    • No Log Aggregation: Logs from multiple instances are not aggregated.
    • Duplicate Responses: Multiple instances may respond to the same event if processing takes longer than 20 seconds. This is expected behavior; only the first response will take effect.

  3. Choose between IMDS Processor and Queue Processor modes

    main

    The AWS Node Termination Handler (NTH) operates in one of two mutually exclusive modes. You must choose the mode that best fits your infrastructure requirements.

    IMDS Processor Mode

    • Mechanism: Monitors the EC2 Instance Metadata Service (IMDS) paths for termination signals.
    • Configuration: Enabled by default if enableSqsTerminationDraining is set to false.
    • Granularity: Allows fine-grained control over which specific IMDS paths are monitored (e.g., Spot Interruption, Rebalance Recommendation, or Scheduled Events).
    • Best for: Simpler setups where you want NTH to poll metadata directly from the instance.

    Queue Processor Mode

    • Mechanism: Consumes termination events from an Amazon SQS queue.
    • Configuration: Enabled by setting enableSqsTerminationDraining to true (Helm) or ENABLE_SQS_TERMINATION_DRAINING=true (Environment Variable).
    • Granularity: Does not support fine-grained path configuration via Helm. Instead, you control event types by modifying your Amazon EventBridge rules to filter which events are sent to the SQS queue.
    • Best for: Complex environments, multi-cluster setups, or when using ASG Lifecycle Hooks to handle AZ rebalances and more advanced event types.

    Note: If enableSqsTerminationDraining is true, IMDS monitoring is automatically disabled.

  4. Extend termination grace periods with Lifecycle Heartbeats

    main

    In Queue Processor mode, NTH can issue heartbeats to an Auto Scaling Group (ASG) to extend the Terminating:Wait state. This is useful for workloads that require long draining times.

    Requirements & Constraints:

    • You must configure a termination lifecycle hook on your ASG.
    • The heartbeat timeout in ASG must be longer than the Heartbeat Interval configured in NTH.
    • Global Timeout: The maximum duration an instance can stay in a wait state is 48 hours or 100 times the heartbeat timeout, whichever is smaller.
    • Concurrency: Heartbeats are limited by the number of workers configured in NTH; NTH can only issue heartbeats for up to the number of instances specified by the workers flag simultaneously.
    • Warning: Heartbeats are only supported in Queue Processor mode. Setting enableSqsTerminationDraining=false while specifying heartbeat flags will cause NTH to fail.

  5. Understand IMDS mode event annotations

    main

    When operating in IMDS mode (where enable-sqs-termination-draining is disabled) and emit-kubernetes-events is enabled, the handler automatically injects metadata gathered from the host's Instance Metadata Service (IMDS) into each event.

    Note: In Queue Processor mode, these default IMDS annotations are disabled, but you can still provide custom ones via kubernetes-events-extra-annotations.

    If you provide custom annotations via kubernetes-events-extra-annotations that collide with these defaults, your custom value will take precedence.

  6. Compare IMDS Monitor and Queue Processor modes

    main

    The AWS Node Termination Handler can be deployed using one of two operational modes. Choosing the right one depends on your infrastructure and permission requirements.

    Instance Metadata Service (IMDS) Monitor

    • Mechanism: Runs a small pod on each host to monitor specific IMDS paths (e.g., /spot or /events).
    • Action: Reacts to metadata changes by draining and/or cordoning the corresponding node.
    • Best for: Simple setups where you want a per-node monitoring agent.

    Queue Processor

    • Mechanism: Monitors an Amazon SQS queue populated by Amazon EventBridge events.
    • Events Monitored: ASG lifecycle events, EC2 status change events, Spot Interruption Termination Notice events, and Spot Rebalance Recommendation events.
    • Requirements: Requires AWS IAM permissions to monitor/manage the SQS queue and to query the EC2 API.
    • Best for: Centralized event processing across the cluster.

  7. Compare IMDS Processor and Queue Processor modes

    main

    AWS Node Termination Handler (NTH) provides two processing modes. Choosing the right one depends on your required event coverage and infrastructure capabilities.

    IMDS Processor

    • Deployment: Must be deployed as a Kubernetes DaemonSet.
    • Best for: Simpler setups that only need to monitor EC2 Instance Metadata.
    • Supported Events:
      • Spot Instance Termination Notifications
      • Scheduled Events
      • Instance Rebalance Recommendations
      • ASG Target Lifecycle State changes (monitors latest/meta-data/autoscaling/target-lifecycle-state for Terminated status).

    Queue Processor

    • Deployment: Must be deployed as a Kubernetes Deployment.
    • Infrastructure Requirements: Requires additional setup including an SQS queue and EventBridge rules.
    • Best for: Advanced workflows requiring lifecycle hooks, AZ rebalancing, or handling manual instance terminations.
    • Supported Events:
      • All IMDS events
      • ASG Termination Lifecycle Hooks (Scale-In, AZ Rebalance, Unhealthy Instances, etc.)
      • Instance State Change events
      • Ability to issue Lifecycle Heartbeats to extend termination grace periods.

  9. Build multi-architecture Docker images

    main

    To build images for all supported Linux architectures (linux/amd64 and linux/arm64), use the make build-docker-images target. This utilizes docker buildx to handle multiple platforms.

    To push a multi-arch image, use the manifest-tool to create a manifest from a specification file.

    # Build all supported Linux architectures
make build-docker-images

# Push a multi-arch image using manifest-tool
cat << EOF > manifest.yaml
image: ${USER}/aws-node-termination-handler:v1.0.0
manifests:
  -
    image: ${USER}/aws-node-termination-handler-amd64:v1.0.0
    platform:
      architecture: amd64
      os: linux
  -
    image: ${USER}/aws-node-termination-handler-arm64:v1.0.0
    platform:
      architecture: arm64
      os: linux
EOF
manifest-tool push from-spec manifest.yaml

  10. Install Queue Processor via kubectl

    main

    When using kubectl for Queue Processor mode, you must manually update the QUEUE_URL in the manifest before applying it.

    1. Download the manifest:
    curl -L https://github.com/aws/aws-node-termination-handler/releases/download/v1.25.6/all-resources-queue-processor.yaml -o all-resources-queue-processor.yaml
    1. Open all-resources-queue-processor.yaml and update the QUEUE_URL value.
    2. Apply the manifest:
    kubectl apply -f ./all-resources-queue-processor.yaml
    curl -L https://github.com/aws/aws-node-termination-handler/releases/download/v1.25.6/all-resources-queue-processor.yaml -o all-resources-queue-processor.yaml
# <open all-resources-queue-processor.yaml and update QUEUE_URL value>
kubectl apply -f ./all-resources-queue-processor.yaml

  11. Install IMDS Processor via Helm

    main

    To install the IMDS Processor mode using Helm, first authenticate to the ECR Public registry, then run the helm upgrade --install command.

    1. Authenticate to ECR Public

    aws ecr-public get-login-password \
  --region us-east-1 | helm registry login \
  --username AWS \
  --password-stdin public.ecr.aws

    2. Install with Configuration

    Replace $CHART_VERSION with your desired version.

    Zero Config (Default):

    helm upgrade --install aws-node-termination-handler \
  --namespace kube-system \
  oci://public.ecr.aws/aws-ec2/helm/aws-node-termination-handler --version $CHART_VERSION

    Enabling Specific IMDS Paths: Use enableSpotInterruptionDraining, enableRebalanceMonitoring, and enableScheduledEventDraining to control monitoring.

    helm upgrade --install aws-node-termination-handler \
  --namespace kube-system \
  --set enableSpotInterruptionDraining="true" \
  --set enableRebalanceMonitoring="true" \
  --set enableScheduledEventDraining="false" \
  oci://public.ecr.aws/aws-ec2/helm/aws-node-termination-handler --version $CHART_VERSION

    Running on Specific Nodes (Node Selector):

    helm upgrade --install aws-node-termination-handler \
  --namespace kube-system \
  --set nodeSelector.lifecycle=spot \
  oci://public.ecr.aws/aws-ec2/helm/aws-node-termination-handler --version $CHART_VERSION

    Webhook Configuration: Pass a URL directly via --set webhookURL or via a Kubernetes Secret using --set webhookURLSecretName.

    helm upgrade --install aws-node-termination-handler \
  --namespace kube-system \
  --set enableSpotInterruptionDraining="true" \
  --set enableRebalanceMonitoring="true" \
  --set enableScheduledEventDraining="false" \
  oci://public.ecr.aws/aws-ec2/helm/aws-node-termination-handler --version $CHART_VERSION