DOCSEARCHbyAlgolia
Submit Docs

Oracle Cloud Infrastructure (OCI) Kubernetes Engine (OKE) Terraform Module

repository·main·Indexed 41 days ago

https://github.com/oracle-terraform-modules/terraform-oci-oke

A Terraform module for provisioning complete OCI Kubernetes Engine (OKE) environments, including networking, worker nodes, management hosts, and Kubernetes extensions. It provides pre-defined configuration templates for basic and enhanced clusters, OIDC discovery, and IAM policies. The module supports the deployment of various Kubernetes extensions such as ArgoCD, Cilium, Karpenter, and Prometheus, and includes examples for setting up multi-region Istio service meshes.

Tokens
20K
Snippets
45
Records
78
Agent score
49%

What's inside OCI OKE Terraform Module

  1. Use the OKE Utilities sub-module

    main

    The utilities sub-module provides helper resources to manage common OKE lifecycle and operational tasks. It includes functionality for:

    • Node readiness checks: Verifying that nodes are fully operational.
    • OCIR secret creation: Automating the creation of secrets required for Oracle Cloud Infrastructure Registry (OCIR) authentication.
    • Worker pool draining: Managing the graceful removal of nodes from a worker pool.

    For detailed configuration options and implementation details, refer to the Utilities section in the main module documentation.

  2. Use the IAM sub-module for OKE resources

    main

    The IAM sub-module is used to provision Identity and Access Management (IAM) resources required for Oracle Container Engine for Kubernetes (OKE). It automates the creation of:

    • IAM Dynamic Groups: To allow OKE resources (like worker nodes) to interact with other OCI services.
    • Policies: To grant necessary permissions to the dynamic groups.
    • Tag Namespaces (Optional): To support resource tagging and governance.

    For detailed configuration parameters and implementation patterns, refer to the central Identity & Access Management section in the main module documentation.

  3. Provision an OCI Kubernetes Engine (OKE) cluster with supporting infrastructure

    main

    The Terraform OKE module provisions a complete Kubernetes environment on Oracle Cloud Infrastructure (OCI). It automates the creation of networking, compute, and management resources required for a production-ready cluster.

    Key resources provisioned include:

    • Networking: A Virtual Cloud Network (VCN) with public/private subnets, network security groups, and gateways (Internet, NAT, Service, DRG).
    • OKE Cluster: Supports basic or enhanced clusters with configurable CNI, Kubernetes versions, and OIDC authentication.
    • Worker Nodes: Supports multiple modes including OKE-managed node pools, virtual nodes, self-managed instances, and HPC-optimized cluster networks.
    • Management Hosts: A bastion host for SSH access and an operator host pre-configured with kubectl, Helm, and optional tools (k9s, istioctl, stern, k8sgpt).
    • IAM & Security: IAM dynamic groups, policies, and optional tag namespaces.
    • Extensions: Kubernetes extensions deployed via Helm or YAML manifests.

    Detailed outputs like kubeconfig are provided when output_detail = true is set.

    # No code snippet provided in the overview section, but the module is used via Terraform.

  5. Choose between Public and Private exposure variants

    main

    You can customize the exposure of your cluster components using different topology variants:

    Control Plane Exposure

    • Public control plane topology: The Kubernetes API endpoint is reachable via a public address. This requires strict control of allowed CIDR ranges.
    • Private control plane topology: The Kubernetes API endpoint remains private within the VCN. This is preferred when administration should only occur from a bastion, operator, or connected private networks.

    Worker Node Exposure

    • Public workers topology: Worker nodes are assigned public IPs with direct outbound reachability. This simplifies bootstrapping and troubleshooting but increases the attack surface.
    • Private workers topology: Worker nodes use private addresses. This is the standard production posture, where outbound access is routed through NAT or service gateways.

  6. Configure network and access patterns

    main

    The module organizes the VCN into functional subnets and NSG boundaries. When designing your network, consider the following layouts:

    • Network Layout: Defines where the control plane, workers, pods, and load balancers reside and how traffic flows between them.
    • Load Balancer Layout: Consult this when deciding on the configuration for load_balancers, preferred_load_balancer, and your service exposure model. It distinguishes between public and internal load balancer subnet choices.
    • Bastion Access Layout: Defines the administrative access path into the VCN via a bastion host. Use this to validate SSH reachability to private resources like the operator or worker nodes.

  7. Configure Cluster Network or Compute Clusters for HPC/GPU Workloads

    main

    For high-performance computing (HPC) or GPU workloads requiring RDMA, you have two primary deployment models:

    1. Self-managed Cluster Network: Use mode = "cluster-network". This requires manual configuration of secondary_vnics and often specific image_ids.
    2. Managed Node Pool in a Compute Cluster: Use mode = "node-pool" combined with use_compute_cluster = true (to create a new cluster) or compute_cluster_id (to attach to an existing one). This allows you to use OKE-managed node pools instead of self-managed instances while still supporting RDMA.

    You can use host_group_id to constrain placement to a dedicated host group.

    # Example: Managed node pool in a compute cluster (HPC/GPU)
worker_pools = {
  oke-bm-gpu-rdma-np = {
    mode                = "node-pool"
    size                = 2
    shape               = "BM.GPU.B4.8"
    placement_ads       = [1]
    use_compute_cluster = true
  }
}

  8. Understand the default OKE network topology

    main

    The module deploys a VCN with a specific subnet architecture to isolate different layers of the Kubernetes stack. By default, the topology includes:

    SubnetPurposeAccess
    bastionBastion hostPublic
    operatorOperator hostPrivate
    cpKubernetes control planePrivate (or public)
    workersWorker nodesPrivate
    podsPod network (NPN CNI)Private
    int_lbInternal load balancersPrivate
    pub_lbPublic load balancersPublic

  9. Choose a worker management mode

    main

    The module supports several worker modes depending on your workload requirements. Select the mode that matches your operational needs:

    ModeDescriptionUse Case
    node-poolOKE-managed node poolsGeneral purpose workloads
    virtual-node-poolOKE-managed virtual nodesServerless, burstable workloads
    instanceSelf-managed compute instancesCustom node configuration
    instance-poolSelf-managed instance poolsScalable self-managed nodes
    cluster-networkSelf-managed cluster networksHPC/GPU with RDMA networking
    compute-clusterShared compute clustersMulti-nodepool HPC clusters

  10. Understand OKE deployment topologies

    main

    The module supports two primary deployment topologies based on Availability Domain (AD) requirements:

    • Default Multi-AD topology: A regional deployment spread across multiple availability domains. It separates the control plane, worker nodes, bastion, operator, and load balancer subnets to support both public entry points and private east-west traffic paths.
    • Default Single-AD topology: A baseline layout constrained to a single availability domain. Use this simpler topology when multi-AD placement is not required or unavailable in your target region.

  11. Understand Deployment Profiles

    main

    The module provides composable deployment profiles that allow you to deploy only the specific components required for your architecture. This enables modularity by separating network infrastructure, the OKE cluster, and worker pools.

    | Profile | Description |
|---------|-------------|
| `network-only` | VCN, subnets, NSGs, and gateways only |
| `cluster-workers-only` | OKE cluster and worker pools (requires existing network) |
| `workers-only` | Worker pools only (requires existing cluster) |
| `network-cluster-workers` | Full stack: network, cluster, and workers |