DOCSEARCHbyAlgolia
Submit Docs

Amazon EKS Blueprints for Terraform

repository·main·Indexed 25 days ago

https://github.com/aws-ia/terraform-aws-eks-blueprints

A collection of opinionated Terraform patterns for deploying and bootstrapping Amazon EKS clusters with operational software. Included patterns cover Agones gaming controllers, AWS Neuron devices with EFA for machine learning, VPC CNI network policies, and blue-green upgrade strategies using ArgoCD for GitOps workload management.

Tokens
32.9K
Snippets
94
Records
171
Agent score
85%

What's inside terraform-aws-eks-blueprints

  1. Overview of Blue/Green Migration Pattern

    main

    This pattern demonstrates how to perform blue/green or canary application workload migrations between EKS clusters using Amazon Route 53 weighted routing.

    Key components include:

    • EKS Clusters: Two clusters (Blue and Green) sharing the same VPC.
    • Workload Deployment: Uses ArgoCD (via the ArgoCD add-on) to deploy GitOps workloads defined as Helm charts.
    • Traffic Management: Workloads are exposed via AWS Load Balancer Controller and External DNS. External DNS is configured in both clusters to share the same Route 53 Hosted Zone, allowing for weighted record updates to shift traffic between clusters.
    • GitOps Bridge: Uses the gitops-bridge-argocd-bootstrap module to pass Terraform metadata to ArgoCD, enabling dynamic configuration of ApplicationSets based on the cluster context.

  2. Get started with Amazon EKS Blueprints for Terraform

    main

    Amazon EKS Blueprints for Terraform provides a set of modular, production-ready Terraform modules designed to simplify the deployment of Amazon EKS clusters and their associated ecosystem (such as networking, compute, and add-ons).

    To use this project, you can consume the modules via the Terraform Registry or by referencing the repository directly. The blueprints are designed to follow AWS best practices for security, scalability, and reliability.

  5. Multi-Node vLLM Infrastructure Components

    main

    The Multi-Node vLLM pattern consists of the following architectural components:

    • Default Node Group: Supports standard addons and components that do not require GPUs or EFA.
    • GPU Node Group (g6e.8xlarge):
      • EFA Enabled: All EFA network interfaces are enabled for high-speed collective communication.
      • Placement Group: Instances are provisioned in a single availability zone within a placement group to minimize latency.
      • Taints/Labels: Uses the taint nvidia.com/gpu:NoSchedule to isolate workloads and provides labels for node selection.
      • RAID-0 NVMe Storage: NVMe instance stores are mounted in a RAID-0 array; kubelet and containerd are configured to use this as ephemeral storage.
    • NVIDIA Device Plugin: Deployed via Helm to expose GPUs to pods.
    • EFA Device Plugin: Deployed via Helm to expose EFA interfaces to pods.
    • Container Image: A custom Dockerfile builds an image containing necessary collective communication libraries for EFA-based multi-node inference.

  6. Understand the v5 Project Structure

    main

    The EKS Blueprints project has been reorganized into several specialized repositories to improve modularity:

    • terraform-aws-eks-blueprints: Contains only example and blueprint implementations (architectural patterns); it no longer contains modules.
    • terraform-aws-eks-blueprints-addon: A module for creating Terraform-based addons (IRSA + Helm chart).
    • terraform-aws-eks-blueprints-addons: A repository containing a select set of supported addons.
    • terraform-aws-eks-blueprints-teams: The updated version of the previous aws-eks-teams sub-module.
    • terraform-aws-eks-ack-addons: Contains ACK-based addons.
    • data-on-eks (under awslabs/): Contains data-related patterns that were previously inside the main blueprints repository.

  7. Features of the Stateful Workload Pattern

    main

    The Amazon EKS Cluster for Stateful Workloads pattern groups together features commonly required for managing stateful workloads. You can enable only the features required for your specific use case.

    Velero

    Provides tools to back up and restore Kubernetes cluster resources and persistent volumes. It supports cluster backups, resource migration, and replication between production and development/testing clusters.

    EBS & EFS CSI Drivers

    • EBS CSI Driver: Adds a gp3 backed storage class and sets it as the cluster default (while keeping gp2 available).
    • EFS CSI Driver: Provides a standard implementation for Amazon EFS.

    EKS Managed Nodegroup with Multiple Volumes

    An EKS managed nodegroup configured with multiple EBS volumes. A dedicated second volume is used for the containerd runtime to prevent the root volume from being exhausted. This configuration includes:

    • EBS encryption using a Customer Managed Key (CMK).
    • Use of gp3 storage class.
    • User data to map containerd directories to the second volume.

    EKS Managed Nodegroup with Instance Store(s)

    An EKS managed nodegroup utilizing EC2 instances with ephemeral instance stores. These are ideal for temporary data like buffers, caches, or scratch data. This configuration includes:

    • EBS encryption using a Customer Managed Key (CMK).
    • Use of gp3 storage class.
    • User data to handle mounting of the instance stores.

  8. Understand the architectural shift in EKS Blueprints v5

    main

    EKS Blueprints v5 is moving away from a Terraform-centric 'push' model for managing Kubernetes addons toward a GitOps-centric 'pull' model.

    Key changes in approach:

    • From Terraform 'Push' to GitOps 'Pull': Instead of Terraform sending requests to the EKS API Server from outside the VPC (which often requires enabling public EKS endpoints), v5 encourages using operators or controllers (like ArgoCD or FluxCD) running inside the cluster. These controllers pull resource definitions from Git and reconcile state from within the cluster's internal network.
    • Addon Management: v5 addresses the limitations of wrapping every Helm chart in a Terraform module. Instead of maintaining a massive 'umbrella' module that obfuscates versioning and creates maintenance bottlenecks, the project is shifting toward supporting various tools and frameworks to meet diverse customer needs.
    • Dependency Handling: v5 acknowledges that Terraform struggles with complex dependency ordering for resources that live inside Kubernetes (where relationships aren't always visible in HCL). Kubernetes controllers/operators are better suited for this as they use continuous reconciliation loops to retry until dependencies are resolved.
    • Coexistence with existing tools: EKS Blueprints v5 is designed to augment, not replace, existing infrastructure. You can continue using clusters created via terraform-aws-eks or eksctl and use EKS Blueprints to manage addons, teams, and other supporting modules.

  9. Deploy an EKS Blueprint cluster with ArgoCD

    main

    This module deploys an EKS Blueprint configured to manage workloads using ArgoCD. The deployment includes an EKS cluster, managed node groups, platform and application teams, and a variety of Kubernetes add-ons (such as Karpenter, AWS Load Balancer Controller, and External DNS) managed via a hybrid approach of Terraform and ArgoCD.

    Infrastructure Components:

    • EKS Cluster: Managed Node Groups and dedicated team quotas (e.g., team-burnham, team-riker).
    • Managed Add-ons: CoreDNS, Kube Proxy, VPC CNI, EBS CSI Driver.
    • ArgoCD Managed Add-ons: Metrics server, Vertical Pod Autoscaler, AWS Load Balancer Controller, Karpenter, External DNS, AWS for FluentBit, CloudWatch Metrics, and Kubecost.
    • Workloads: Managed via GitOps in a dedicated repository.

  10. Destroy resources using Terraform

    main

    To clean up and remove the resources created by the EKS Blueprints, use the terraform destroy command. Depending on your requirements, you can destroy specific modules or the entire stack.

    Warning: Destroying specific modules (using -target) can lead to inconsistent state if those modules have dependencies. It is generally recommended to destroy the entire stack using terraform destroy -auto-approve to ensure all resources are removed correctly and dependencies are handled by Terraform.