ingress-nginx

repository·main·Indexed 12 days ago

https://github.com/kubernetes/ingress-nginx

A Kubernetes Ingress controller that uses NGINX as a reverse proxy and load balancer. This documentation covers installation via Helm, configuration of admission webhooks, autoscaling (including KEDA), Prometheus metrics, and networking settings for internal and external services.

Tokens
173.7K
Snippets
417
Records
744
Agent score
94%

What's inside ingress-nginx

  1. Overview of kube-webhook-certgen

    main

    kube-webhook-certgen is a utility designed to automate the setup of Kubernetes Admission Webhooks. It performs two primary functions:

    1. Certificate Generation: It generates a Certificate Authority (CA) and a leaf certificate with a long expiration (100 years).
    2. Webhook Patching: It automatically patches ValidatingWebhookConfiguration and MutatingWebhookConfiguration resources by setting the caBundle field with the generated CA.

    It is optimized for use within Helm provisioning processes, typically running as a Kubernetes Job via pre-install or post-install hooks.

    Security Note: This tool is intended for quick installation and configuration. For high-security environments requiring more robust certificate management, consider using cert-manager.

  2. Ways to customize NGINX configuration

    main

    You can customize the NGINX configuration in the Ingress NGINX Controller using three different methods depending on the scope and specificity required:

    1. Global Configuration via ConfigMap: Use a Kubernetes ConfigMap to apply settings across the entire NGINX instance. This is best for settings that affect all Ingress rules.
    2. Rule-specific Configuration via Annotations: Use Ingress resource annotations to apply specific configurations to a particular Ingress rule. This is ideal for per-service or per-host settings.
    3. Advanced Configuration via Custom Templates: Use custom NGINX templates when you need to implement settings that cannot be expressed via ConfigMap or Annotations. This is required for low-level NGINX directives such as open_file_cache or adjusting listen options like rcvbuf.
  3. Ingress NGINX Retirement Notice

    main

    The ingress-nginx project is being retired.

    Key Dates and Impact:

    • Best-effort maintenance: Continues until March 2026.
    • Post-March 2026: No further releases, bugfixes, or security updates will be provided.
    • Existing Deployments: Current deployments will not be broken. Existing project artifacts, such as Helm charts and container images, will remain available.

    Recommendation: If you are not currently using ingress-nginx, do not deploy it. Instead, identify and use a Gateway API implementation.

  4. What is kube-webhook-certgen?

    main

    kube-webhook-certgen is a utility designed to automate the setup of Kubernetes Admission Webhooks. It performs two primary functions:

    1. Certificate Generation: It generates a Certificate Authority (CA) and a leaf certificate/key pair with a long expiration (100 years).
    2. Webhook Patching: It automatically patches ValidatingWebhookConfiguration and MutatingWebhookConfiguration resources by setting their caBundle field with the generated CA.

    This tool is optimized for use within Helm provisioning processes, typically executed as a Kubernetes Job using pre-install or post-install hooks.

  5. Implement Custom Upstream Hashing

    main

    Use consistent hashing (Ketama algorithm) to map requests to specific backend servers based on a key (e.g., URI, Host, or custom text). This provides stickiness without relying on cookies or client IP.

    Subset Hashing: By enabling upstream-hash-by-subset, requests are mapped to a subset of nodes rather than a single node. A specific server is then chosen uniformly at random from that subset, balancing stickiness with load distribution.

    # Consistent hashing by URI
    nginx.ingress.kubernetes.io/upstream-hash-by: "$request_uri"
    
    # Subset hashing
    nginx.ingress.kubernetes.io/upstream-hash-by-subset: "true"
    nginx.ingress.kubernetes.io/upstream-hash-by-subset-size: "3"
  6. Configure Canary deployments with canary-* annotations

    main

    Canary deployments allow you to route a subset of traffic to a specific Ingress resource. This is controlled via several annotations:

    • nginx.ingress.kubernetes.io/canary: Enables canary functionality.
    • nginx.ingress.kubernetes.io/canary-weight: Specifies the weight of the canary traffic (e.g., 50 for 50% of traffic).
    • nginx.ingress.kubernetes.io/canary-by-header: Enables routing based on a specific header.
    • nginx.ingress.kubernetes.io/canary-tfo: (Internal/Advanced) related to TCP Fast Open.

    Key Behaviors:

    • If canary-weight is 0, all traffic goes to the mainline upstream.
    • If canary-weight is 100, all traffic goes to the canary upstream.
    • If a regex in a canary configuration causes an error, the controller typically routes traffic back to the mainline upstream.
    • Canary Ingresses should not be used as catch-all servers or for specific domains without a matching mainline Ingress.
    # Example: 50% Canary deployment
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-weight: "50"
  7. Enable SSL Passthrough

    main

    SSL Passthrough allows Ingress objects to pass encrypted traffic directly to backend services without NGINX decrypting it. This feature is disabled by default and must be enabled via the --enable-ssl-passthrough CLI flag.

    How it works:

    • It uses SNI (Server Name Indication) to read the virtual domain from the TLS negotiation.
    • It intercepts all traffic on the configured HTTPS port (default: 443) and hands it to a local TCP proxy.
    • If no hostname matches, the request is sent to the passthrough proxy port (default: 442), which proxies to the default backend.

    Warnings:

    • Performance: This introduces a non-negligible performance penalty because it bypasses NGINX's optimized processing.
    • Traffic Routing: Unlike HTTP backends, traffic to Passthrough backends is sent to the clusterIP of the backing Service instead of individual Endpoints.
  8. How cookie-based session affinity works

    main

    Mechanism

    1. Cookie Creation: When a client makes a request, the Ingress-Nginx Controller selects an upstream using consistent hashing and generates a cookie containing a randomly generated key corresponding to that upstream.
    2. Client Storage: The client receives the Set-Cookie header and stores the cookie.
    3. Subsequent Requests: The client sends the cookie back in subsequent requests. NGINX uses the key in the cookie to route the request to the same upstream.
    4. Handling Missing/Invalid Cookies: If a client sends a cookie that does not correspond to an existing upstream, NGINX selects a new upstream and issues a new cookie.

    Scaling and Failures

    • Scaling Up: If the backend pool grows, NGINX will continue sending requests to the original server even if it becomes overloaded (unless affinity-mode is set to balanced).
    • Scaling Down/Failures: When a backend server is removed, requests are re-routed to another upstream. Because the key's consistent hash changes, the cookie is effectively updated for the new routing.
    • Failure Handling: By default (nginx.ingress.kubernetes.io/session-cookie-change-on-failure: false), NGINX will still attempt to send the request to the upstream pointed to by the sticky cookie even if the previous attempt failed. Setting this to true will cause NGINX to change the sticky cookie to point to a different upstream if the previous attempt failed.
  9. Configure Ingress Controller authentication to the Kubernetes API Server

    main

    The Ingress controller requires authentication to communicate with the Kubernetes API server. You can use one of three methods:

    1. Service Account (Recommended): The controller uses the automatically provided service account token. The token is located at /var/run/secrets/kubernetes.io/serviceaccount/token.
    2. Kubeconfig file: For environments without service accounts, provide a path to a kubeconfig file using the --kubeconfig flag. This flag replaces the need for --apiserver-host.
    3. --apiserver-host flag: Use this to specify an unsecured API server or reach a remote cluster via kubectl proxy. Do not use this in production.
    # Example using kubeconfig flag
    # Add this to the deployment args:
    --kubeconfig=/etc/kubernetes/kubeconfig.yaml
  10. Pass Real IP and support WebSockets in Ingress

    main

    When configuring Ingress, you can ensure that the backend services receive the original client information and support persistent connections like WebSockets by using specific proxy headers.

    Key Headers used by the controller:

    • Real IP: proxy_set_header X-Real-IP $remote_addr; ensures the backend knows the actual client IP.
    • WebSockets:
      • proxy_set_header Upgrade $http_upgrade;
      • proxy_set_header Connection $connection_upgrade; These allow the connection to be upgraded from HTTP to a WebSocket protocol.
    • Forwarded Headers:
      • X-Forwarded-For: The list of IPs the request has passed through.
      • X-Forwarded-Host: The original Host header sent by the client.
      • X-Forwarded-Proto: The original protocol (e.g., https).