Amazon EKS Blueprints Addons

repository·main·Indexed 18 days ago

https://github.com/aws-ia/terraform-aws-eks-blueprints-addons

A Terraform module designed to simplify the deployment and management of various Kubernetes addons on Amazon EKS clusters. It supports a wide range of addons including ArgoCD, Karpenter, AWS Load Balancer Controller, Cert-manager, and External DNS, providing fine-grained control over AWS-managed addons and Helm-based deployments through boolean flags and configuration objects.

Tokens
28.2K
Snippets
88
Records
104
Agent score
64%

What's inside terraform-aws-eks-blueprints-addons

  1. Get started with Amazon EKS Blueprints Addons

    main

    Amazon EKS Blueprints Addons is a collection of Terraform modules designed to simplify the installation and management of common EKS add-ons. It provides a standardized way to deploy essential Kubernetes components (such as Karpenter, AWS Load Balancer Controller, and various observability tools) onto an existing Amazon EKS cluster using Terraform.

    To use this project, you typically provide your existing EKS cluster details and configure the specific add-ons you wish to deploy via module inputs.

  2. Provision Helm charts with IRSA using the `terraform-aws-eks-blueprints-addon` module

    main

    If an add-on requires IAM Role for Service Account (IRSA) to function, the helm_releases variable in the main addons module may not be sufficient. Instead, use the standalone terraform-aws-eks-blueprints-addon module.

    This module is designed to provision a Helm chart alongside the necessary IAM roles and policies. You can use it to:

    1. Provision a Helm chart with full IRSA support.
    2. Provision only the IAM resources (roles/policies) and skip the Helm release.
    3. Wrap multiple such add-ons into a custom Terraform module to maintain a consistent pattern for unsupported add-ons.
  3. Deploy Bottlerocket Update Operator (BRUPOP) via EKS Blueprints Addons

    main

    You can deploy BRUPOP using the eks-blueprints-addons module.

    Important: BRUPOP requires cert-manager to be installed first so the API server can use a CA certificate for SSL communication with agents. When enabling both, you must set cert_manager.wait = true to ensure the Cert-Manager CRDs are present before BRUPOP attempts to deploy.

    Default Configuration

    Enables BRUPOP with standard settings.

    Custom Configuration

    You can customize the Helm charts for the bottlerocket_update_operator and the bottlerocket_shadow (which provides the CRDs). This allows you to change the namespace, chart version, or specific Helm set values like the scheduler_cron_expression.

    ### Default Deployment
    ```hcl
    module "eks_blueprints_addons" {
      source  = "aws-ia/eks-blueprints-addons/aws"
      version = "~> 1.13"
    
      cluster_name      = module.eks.cluster_name
      cluster_endpoint  = module.eks.cluster_endpoint
      cluster_version   = module.eks.cluster_version
      oidc_provider_arn = module.eks.oidc_provider_arn
    
      enable_cert_manager = true
      cert_manager = {
        wait = true
      }
      enable_bottlerocket_update_operator = true
    }

    Customized Deployment

    enable_bottlerocket_update_operator           = true
    
    bottlerocket_update_operator = {
      name          = "brupop-operator"
      description   = "A Helm chart for BRUPOP"
      chart_version = "1.3.0"
      namespace     = "brupop"
      set           = [{
        name  = "scheduler_cron_expression"
        value = "0 * * * * * *" # Check every hour
        }]
    }
    
    bottlerocket_shadow = {
      name          = "brupop-crds"
      description   = "A Helm chart for BRUPOP CRDs"
      chart_version = "1.0.0"
    }
  4. Configure Bottlerocket OS for BRUPOP updates

    main

    To allow the Bottlerocket Update Operator (BRUPOP) to manage updates, your nodes must run the Bottlerocket OS and include a specific label. The label bottlerocket.aws/updater-interface-version=2.0.0 is required for the BRUPOP Agent to query and perform updates. Nodes without this label will be ignored by the agent.

    Using Managed Node Groups

    When using the Terraform Amazon EKS module, add the label to the [settings.kubernetes.node-labels] section of your bootstrap_extra_args.

    Using Karpenter

    When using Karpenter, add the label to the [settings.kubernetes.node-labels] section within the userData field of your EC2NodeClass.

    ### Managed Node Groups Example
    ```hcl
    module "eks" {
      source  = "terraform-aws-modules/eks/aws"
      version = "~> 19.21"
      # ...
      eks_managed_node_groups = {
        bottlerocket = {
          platform = "bottlerocket"
          ami_type       = "BOTTLEROCKET_x86_64"
          # ...
          enable_bootstrap_user_data = true
          bootstrap_extra_args = <<-EOT
                [settings.host-containers.admin]
                enabled = false
                [settings.host-containers.control]
                enabled = true
                [settings.kernel]
                lockdown = "integrity"
                [settings.kubernetes.node-labels]
                "bottlerocket.aws/updater-interface-version" = "2.0.0"
                [settings.kubernetes.node-taints]
                "CriticalAddonsOnly" = "true:NoSchedule"
              EOT
        }
      }
    }

    Karpenter Example

    apiVersion: karpenter.k8s.aws/v1beta1
    kind: EC2NodeClass
    metadata:
      name: bottlerocket-example
    spec:
      amiFamily: Bottlerocket
      userData:  |
        [settings.kubernetes]
        "kube-api-qps" = 30
        "shutdown-grace-period" = "30s"
        "shutdown-grace-period-for-critical-pods" = "30s"
        [settings.kubernetes.eviction-hard]
        "memory.available" = "20%"
        [settings.kubernetes.node-labels]
         "bottlerocket.aws/updater-interface-version" = "2.0.0"
  5. Verify AWS FSx CSI Driver deployment

    main

    After deployment, verify that the controller and node components are running in the kube-system namespace using kubectl.

    Check the controller deployment:

    kubectl -n kube-system get deployment fsx-csi-controller

    Check the controller pods:

    kubectl -n kube-system get pods -l app=fsx-csi-controller

    Check the node daemonset:

    kubectl -n kube-system get daemonset fsx-csi-node

    Check the node pods:

    kubectl -n kube-system get pods -l app=fsx-csi-node
  6. Verify AWS Node Termination Handler deployment

    main

    After deploying the add-on, verify the installation by checking the status of the pods and ensuring the required AWS resources (SQS Queue and Event Rules) were created.

    1. Check Pod Status

    Ensure the pods are running in the designated namespace (default is aws-node-termination-handler).

    2. Verify SQS Queue

    Confirm that an SQS queue has been created to handle the termination events.

    3. Verify Event Rules

    Confirm that EventBridge rules are active for various event types, including:

    • EC2 Instance-terminate Lifecycle Action
    • AWS Health Event
    • EC2 Instance Rebalance Recommendation
    • EC2 Instance State-change Notification
    • EC2 Spot Instance Interruption Warning
    # Check pods
    kubectl get pods -n aws-node-termination-handler
    
    # Check SQS queues
    aws sqs list-queues
    
    # Check Event Rules
    aws event list-rules
  7. Enable and configure Fargate FluentBit

    main

    Amazon EKS on Fargate uses a built-in log router based on Fluent Bit. Instead of running a sidecar container, you configure the router via a dedicated ConfigMap deployed by this add-on.

    To enable it in the eks_blueprints_addons module, set enable_fargate_fluentbit = true. You can then configure specific features, such as CloudWatch Logs integration, using the fargate_fluentbit block.

    module "eks_blueprints_addons" {
    
      enable_fargate_fluentbit = true
      fargate_fluentbit = {
        flb_log_cw = true
      }
    }