terraform-aws-eks

EKS Hybrid Nodes allow running nodes on infrastructure outside of AWS (e.g., on-premises).

To set this up:

1. Use the eks_hybrid_node_role module to create the necessary IAM role.
2. In the eks module, define access_entries with type = "HYBRID_LINUX" using the hybrid node role ARN.
3. Configure remote_network_config with remote_node_networks and remote_pod_networks (the latter is required if running webhooks on Hybrid nodes).
4. Configure security_group_additional_rules to allow traffic from the remote network CIDRs.

locals {
  remote_network_cidr = "172.16.0.0/16"
  remote_node_cidr    = cidrsubnet(local.remote_network_cidr, 2, 0)
  remote_pod_cidr     = cidrsubnet(local.remote_network_cidr, 2, 1)
}

module "eks_hybrid_node_role" {
  source  = "terraform-aws-modules/eks/aws//modules/hybrid-node-role"
  version = "~> 21.0"
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = "example"
  kubernetes_version = "1.33"

  create_node_security_group = false
  security_group_additional_rules = {
    hybrid-all = {
      cidr_blocks = [local.remote_network_cidr]
      description = "Allow all traffic from remote node/pod network"
      from_port   = 0
      to_port     = 0
      protocol    = "all"
      type        = "ingress"
    }
  }

  compute_config = {
    enabled    = true
    node_pools = ["system"]
  }

  access_entries = {
    hybrid-node-role = {
      principal_arn = module.eks_hybrid_node_role.arn
      type          = "HYBRID_LINUX"
    }
  }

  vpc_id     = "vpc-1234556abcdef"
  subnet_ids = ["subnet-abcde012", "subnet-bcde012a", "subnet-fghi345a"]

  remote_network_config = {
    remote_node_networks = {
      cidrs = [local.remote_node_cidr]
    }
    remote_pod_networks = {
      cidrs = [local.remote_pod_cidr]
    }
  }
}

In v19.x, the default security group created by self-managed node groups has been removed. To upgrade without disruption, follow these steps while still using v18.x of the module:

1. Identify Usage: If you are using the default security group, create a new security group with the same rules and add its ID to vpc_security_group_ids.
2. Disable Default SG: Set create_security_group = false in your node group definition or self_managed_node_group_defaults.
3. Trigger Instance Refresh: Because changes to launch templates do not automatically cycle existing instances, you must add an instance_refresh block to force nodes to re-deploy. This ensures new instances launch without the old security group and old ones are terminated.
4. Safe Eviction: It is recommended to use aws-node-termination-handler to ensure pods are safely evicted during the refresh.

Once the node group has cycled and the security group is no longer attached to any instances, you can proceed to upgrade the module version to v19.x.

self_managed_node_group_defaults = {
  create_security_group = false
  instance_refresh = {
    strategy = "Rolling"
    preferences = {
      min_healthy_percentage = 66
    }
  }
}

Use the self-managed-node-group module to create an Auto Scaling Group (ASG) for EKS, including the necessary IAM role, security group, and launch template.

Important: If you are using this module outside of the main EKS module context, you MUST provide vpc_security_group_ids containing the cluster's security groups. Failure to do so will result in empty security groups on the nodes, preventing them from joining the cluster.

module "self_managed_node_group" {
  source = "terraform-aws-modules/eks/aws//modules/self-managed-node-group"

  name                = "separate-self-mng"
  cluster_name        = "my-cluster"
  kubernetes_version     = "1.31"
  cluster_endpoint    = "https://012345678903AB2BAE5D1E0BFE0E2B50.gr7.us-east-1.eks.amazonaws.com"
  cluster_auth_base64 = "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKbXFqQ1VqNGdGR2w3ZW5PeWthWnZ2RjROOTVOUEZCM2o0cGhVZUsrWGFtN2ZSQnZya0d6OGxKZmZEZWF2b2plTwpQK2xOZFlqdHZncmxCUEpYdHZIZmFzTzYxVzdIZmdWQ2EvamdRM2w3RmkvL1dpQmxFOG9oWUZkdWpjc0s1SXM2CnNkbk5KTTNYUWN2TysrSitkV09NT2ZlNzlsSWdncmdQLzgvRU9CYkw3eUY1aU1hS3lsb1RHL1V3TlhPUWt3ZUcKblBNcjdiUmdkQ1NCZTlXYXowOGdGRmlxV2FOditsTDhsODBTdFZLcWVNVlUxbjQyejVwOVpQRTd4T2l6L0xTNQpYV2lXWkVkT3pMN0xBWGVCS2gzdkhnczFxMkI2d1BKZnZnS1NzWllQRGFpZTloT1NNOUJkNFNPY3JrZTRYSVBOCkVvcXVhMlYrUDRlTWJEQzhMUkVWRDdCdVZDdWdMTldWOTBoL3VJUy9WU2VOcEdUOGVScE5DakszSjc2aFlsWm8KWjNGRG5QWUY0MWpWTHhiOXF0U1ROdEp6amYwWXBEYnFWci9xZzNmQWlxbVorMzd3YWM1eHlqMDZ4cmlaRUgzZgpUM002d2lCUEVHYVlGeWN5TmNYTk5aYW9DWDJVL0N1d2JsUHAKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ=="

  subnet_ids = ["subnet-abcde012", "subnet-bcde012a", "subnet-fghi345a"]

  // The following variables are necessary if you decide to use the module outside of the parent EKS module context.
  // Without it, the security groups of the nodes are empty and thus won't join the cluster.
  vpc_security_group_ids = [
    module.eks.cluster_primary_security_group_id,
    module.eks.cluster_security_group_id,
  ]

  min_size     = 1
  max_size     = 10
  desired_size = 1

  launch_template_name   = "separate-self-mng"
  instance_type          = "m5.large"

  tags = {
    Environment = "dev"
    Terraform   = "true"
  }
}

When upgrading from version 17.x to 18.x, several core variables have been renamed. Ensure your Terraform configuration is updated to use the new keys to avoid breaking changes.

Root Module Renames:

• create_eks $\rightarrow$ create
• subnets $\rightarrow$ subnet_ids
• cluster_create_security_group $\rightarrow$ create_cluster_security_group
• cluster_log_retention_in_days $\rightarrow$ cloudwatch_log_group_retention_in_days
• cluster_log_kms_key_id $\rightarrow$ cloudwatch_log_group_kms_key_id
• manage_cluster_iam_resources $\rightarrow$ create_iam_role
• cluster_iam_role_name $\rightarrow$ iam_role_name
• permissions_boundary $\rightarrow$ iam_role_permissions_boundary
• iam_path $\rightarrow$ iam_role_path
• pre_userdata $\rightarrow$ pre_bootstrap_user_data
• additional_userdata $\rightarrow$ post_bootstrap_user_data
• worker_groups $\rightarrow$ self_managed_node_groups
• workers_group_defaults $\rightarrow$ self_managed_node_group_defaults
• node_groups $\rightarrow$ eks_managed_node_groups
• node_groups_defaults $\rightarrow$ eks_managed_node_group_defaults

EKS Managed Node Group Sub-module Renames:

• create_eks $\rightarrow$ create
• worker_additional_security_group_ids $\rightarrow$ vpc_security_group_ids

Fargate Profile Sub-module Renames:

• fargate_pod_execution_role_name $\rightarrow$ name
• create_fargate_pod_execution_role $\rightarrow$ create_iam_role
• subnets $\rightarrow$ subnet_ids
• iam_path $\rightarrow$ iam_role_path
• permissions_boundary $\rightarrow$ iam_role_permissions_boundary

When providing a custom ami_id, the AWS EKS Managed Node Group service will not automatically inject the necessary bootstrap script. To resolve this, set enable_bootstrap_user_data = true. This allows the module to append the default bootstrap template to your user data. You can also use cloudinit_pre_nodeadm and cloudinit_post_nodeadm to inject additional configuration or scripts.

  eks_managed_node_groups = {
    custom_ami = {
      ami_id   = "ami-0caf35bc73450c396"
      ami_type = "AL2023_x86_64_STANDARD"

      # Adds the bootstrap script back in using the module's template
      enable_bootstrap_user_data = true

      cloudinit_pre_nodeadm = [{
        content      = <<-EOT
          ---
          apiVersion: node.eks.aws/v1alpha1
          kind: NodeConfig
          spec:
            kubelet:
              config:
                shutdownGracePeriod: 30s
        EOT
        content_type = "application/node.eks.aws"
      }]

      cloudinit_post_nodeadm = [{
        content      = <<-EOT
          echo "All done"
        EOT
        content_type = "text/x-shellscript; charset="us-ascii""
      }]
    }
  }

When upgrading from version 17.x to 18.x, several configuration keys and structures have changed. Ensure you update your module calls to reflect the new naming conventions for networking, managed node groups, and self-managed node groups.

Key Changes:

Networking

• Rename subnets to subnet_ids.

EKS Managed Node Groups

• Rename node_groups_defaults to eks_managed_node_group_defaults.
• Rename node_groups to eks_managed_node_groups.
• Inside node group definitions:
  • Rename min_capacity to min_size.
  • Rename max_capacity to max_size.
  • Rename desired_capacity to desired_size.
  • Rename k8s_labels to labels.
  • Rename additional_tags to tags.

Self-Managed Node Groups (formerly Worker Groups)

• The worker_groups_launch_template structure has been replaced by self_managed_node_groups.
• Use self_managed_node_group_defaults to set common settings like vpc_security_group_ids (replaces worker_additional_security_group_ids).
• Inside self_managed_node_groups definitions:
  • Use min_size, max_size, and desired_size (replaces asg_max_size, asg_desired_capacity, etc.).
  • Use bootstrap_extra_args instead of kubelet_extra_args.
  • Use instance_type (singular) instead of instance_types (plural) if defining a single type.
  • For mixed instance policies, use the use_mixed_instances_policy and mixed_instances_policy blocks.

module "cluster_after" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 18.0"

  cluster_name                    = local.name
  cluster_version                 = local.cluster_version
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  eks_managed_node_group_defaults = {
    ami_type  = "AL2_x86_64"
    disk_size = 50
  }

  eks_managed_node_groups = {
    node_group = {
      min_size     = 1
      max_size     = 10
      desired_size = 1

      instance_types = ["t3.large"]
      capacity_type  = "SPOT"

      update_config = {
        max_unavailable_percentage = 50
      }

      labels = {
        Environment = "test"
      }

      taints = [
        {
          key    = "dedicated"
          value  = "gpuGroup"
          effect = "NO_SCHEDULE"
        }
      ]

      tags = {
        ExtraTag = "example"
      }
    }
  }

  self_managed_node_group_defaults = {
    vpc_security_group_ids = [aws_security_group.additional.id]
  }

  self_managed_node_groups = {
    worker_group = {
      name = "worker-group"
      min_size      = 1
      max_size      = 5
      desired_size  = 2
      instance_type = "m4.large"
      bootstrap_extra_args = "--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'"
    }
  }

  fargate_profiles = {
    default = {
      name = "default"
      selectors = [{ namespace = "default" }]
    }
  }

  tags = {
    Environment = "test"
  }
}