DOCSEARCHbyAlgolia
Submit Docs

OCI Free Cloud K8s

repository·main·Indexed 42 days ago

https://github.com/nce/oci-free-cloud-k8s

A Terraform and GitOps (FluxCD) repository for provisioning a zero-cost Kubernetes cluster on Oracle Cloud's Always Free tier using ARM-based VM.Standard.A1.Flex instances. Features include GatewayAPI with envoy-gateway, Longhorn storage, Teleport for GitHub SSO access, and a two-layer Terraform deployment structure (infra and config).

Tokens
2.4K
Snippets
4
Records
11
Agent score
5%

What's inside oci-free-cloud-k8s

  1. Overview of Oracle Cloud Kubernetes Free Tier Setup

    main

    This project provides a configuration to provision a Kubernetes cluster using Oracle Cloud's Always Free tier with zero monthly costs.

    Key Specifications:

    • Compute: Uses ARM-based VM.Standard.A1.Flex instances. The free tier provides 2 oCPUs and 12GB of memory (split between two worker nodes).
    • Storage: Each node has a 100GB boot volume, with ~60GB available for in-cluster storage via Longhorn.
    • Networking:
      • Ingress uses GatewayAPI via envoy-gateway on a Layer 7 Flexible Load Balancer (10Mbps).
      • teleport uses a Layer 4 Network Load Balancer.
    • Architecture Note: This setup uses ARM architecture only due to free tier limitations; x86 is not supported.

  2. Use GatewayAPI with Envoy Gateway for Ingress

    main

    The setup uses GatewayAPI with the envoy-gateway implementation instead of nginx-ingress.

    • Routing: All ingress resources are replaced by httproutes.
    • Security: envoy-gateway supports SecurityPolicies, which are used to protect httproutes with OIDC authentication.
    • Load Balancer Integration: Because Oracle provides a Layer 7 load balancer, the oci-controller requires the Security Group (SG) ID to be associated with the service. In this specific setup, the SG value is currently hardcoded in the EnvoyProxy CR because it is generated by Terraform but needed during the object creation phase.

  3. Understand the Terraform provisioning structure

    main

    The Terraform provisioning for this project is split into two distinct parts to mitigate long execution times and manage provider dependencies more effectively:

    1. infra/ (cluster-infra): Contains all resources required to provision the underlying infrastructure up to the point of having a functioning Kubernetes (k8s) API.
    2. config/ (cluster-config): Contains all resources that depend on an existing Kubernetes API (e.g., installing operators, configuring cluster-level settings).

    When provisioning, you should treat these as sequential stages: first establish the infrastructure, then apply the configuration.

  4. Access the Cluster via Teleport (GitHub SSO)

    main

    Teleport is used for secure Kubernetes API access. This setup uses GitHub as the Identity Provider (IdP) via SSO.

    Login via GitHub:

    # 1. Login to the Teleport proxy
tsh login --proxy teleport.nce.wtf:443 --auth=github-acme --user <YOUR_GITHUB_USER> teleport.nce.wtf

# 2. Login to the specific K8s cluster
tsh kube login oci

# 3. Verify access
k get po -n teleport

    Troubleshooting Certificate Errors: If you encounter a tls: failed to verify certificate: x509: certificate signed by unknown authority error during tsh kube login, it is often due to a stale session after a Teleport reinstallation. Perform a logout and login to refresh the client state.

    tsh login --proxy teleport.nce.wtf:443 --auth=github-acme --user nce teleport.nce.wtf
tsh kube login oci
k get po -n teleport

  5. Understand the Terraform Layout and Deployment Order

    main

    The infrastructure is managed in two distinct Terraform layers. You must deploy them in order:

    1. Infrastructure (terraform/infra/): Manages the core OCI resources required to reach the Kubernetes API endpoint. This must be run first.
    2. Kubernetes Configuration (terraform/config/): Manages OCI-specific Kubernetes configurations (like secrets). This relies on the output of the infra run.

    Important Notes:

    • The config layer requires a private *.tfvars file containing at least compartment_id.
    • Expected Failure: During initial provisioning, the config run might fail when attempting to create a ClusterSecretStore because external-secrets hasn't been deployed by FluxCD yet. If this happens, wait for FluxCD to deploy external-secrets, then rerun the config Terraform apply.

  6. Manually roll Kubernetes nodes after an upgrade

    main

    If the terraform apply command does not automatically roll your worker nodes, you must manually cordon, drain, and terminate instances to trigger node recreation. This is particularly important when using storage solutions like Longhorn that need to sync volumes to new nodes.

    1. Cordon and Drain the node

    Use kubectl to prepare the node for removal:

    k drain <k8s-node-name> --force --ignore-daemonsets --delete-emptydir-data
k cordon <k8s-node-name>

    2. Terminate the OCI Instance

    Deleting a node in Kubernetes does not automatically update the OKE Node Pool. You must terminate the specific instance via the OCI CLI or Web UI:

    1. Log in to the OCI Web UI.
    2. Navigate to OKE Cluster -> Node pool.
    3. Identify the correct instance using its private_ip and copy the OCID.
    4. Terminate the instance:
    oci compute instance terminate --force --instance-id <oidc.id>

    3. Verify Volume Sync

    After the node is recreated, wait for the node to reach Ready status and ensure Longhorn volumes have synchronized:

    k get -w volumes.longhorn.io -A

    Repeat these steps for all nodes in the pool.

    # cordon and drain
k drain <k8s-node-name> --force --ignore-daemonsets --delete-emptydir-data
k cordon <k8s-node-name>

# terminate instance to trigger node recreation
oci compute instance terminate --force --instance-id <oidc.id>

# wait for longhorn volumes to be healthy
k get -w volumes.longhorn.io -A

  7. Upgrade the OKE Kubernetes version

    main

    To upgrade the Oracle Cloud Infrastructure (OCI) Kubernetes Engine (OKE) cluster, follow these steps to ensure minimal disruption. It is recommended to only upgrade to versions identified by the available-kubernetes-upgrades command to avoid breaking the process.

    1. Identify available versions

    Run the following command from within the terraform/infra/ directory to find the recommended upgrade path:

    oci ce cluster get --cluster-id $(terraform output --raw k8s_cluster_id) | jq -r '.data."available-kubernetes-upgrades"'

    2. Update Terraform configuration

    Update your _variables.tf file with the target version. For example, to upgrade to v1.31.1:

    sed -i '' 's/default = "'$(terraform output --raw kubernetes_version)'"'/default = "v1.31.1"/' _variables.tf

    3. Apply the upgrade

    Run Terraform to upgrade the control plane and the node pool/images. This process typically takes about 10 minutes and should not roll the nodes automatically.

    terraform apply
    # get new cluster versions
oci ce cluster get --cluster-id $(terraform output --raw k8s_cluster_id) | jq -r '.data."available-kubernetes-upgrades"'

# update the cluster version
sed -i '' 's/default = "'$(terraform output --raw kubernetes_version)'"'/default = "v1.31.1"/' _variables.tf

# upgrade the controlplane and the nodepool & images
terraform apply

  8. Configure Client-Side Tooling and OCI Backend

    main

    To manage the infrastructure, you need terraform (v1.12 or higher), the oci-binary, and a configured OCI CLI (oci setup config).

    Terraform state is stored in Oracle Object Storage. You must first create a bucket with versioning enabled:

    oci os bucket create --name terraform-states --versioning Enabled --compartment-id <YOUR_COMPARTMENT_ID>

    Then, configure your ~/.oci/config to include both your default OCI credentials and AWS-compatible credentials (created via the OCI UI under User -> customer secret key) for S3-compatible backend access:

    [DEFAULT]
user=ocid1.user.xxx
fingerprint=ee:f4:xxx
tenancy=ocid1.tenancy.oc1.xxx
region=eu-frankfurt-1
key_file=/Users/xxxx.pem

[default]
aws_access_key_id = xxx
aws_secret_access_key = xxx
    oci os bucket create --name terraform-states --versioning Enabled --compartment-id xxx

  9. Access the Kubernetes Cluster via Kubeconfig

    main

    Once the terraform/infra deployment is complete, a .kube.config file is generated within the terraform folder. Use this file to interact with the cluster via kubectl.

    For more secure and regulated access, use the Teleport method described in the Teleport section.

  10. Configure FluxCD with GitHub Permissions

    main

    FluxCD is used for GitOps. To enable instant reconciliation via webhooks and allow FluxCD to annotate GitHub commit statuses, you must provide a GitHub Personal Access Token (PAT) with the following fine-grained scopes:

    • contents: read, write
    • commit statuses: read, write
    • webhooks: read, write

    Implementation:

    • Place the token in a private tfvars file to generate the FluxCD webhook URL.
    • Place the token in the OCI Vault (github-fluxcd-token) to allow FluxCD to annotate GitHub commit statuses based on Kustomization state.