HAProxy Kubernetes Ingress Controller Documentation

repository·master·Indexed 21 days ago

https://github.com/haproxytech/kubernetes-ingress

Documentation for the HAProxy Kubernetes Ingress Controller, a resource that manages and routes external traffic to services within a Kubernetes cluster. Includes guides on installation via YAML or Helm, building from source, implementing Custom Resource Definitions (CRDs), and configuring custom annotations using ValidationRules and Common Expression Language (CEL) for schema validation.

Tokens
37K
Snippets
113
Records
142
Agent score
72%

What's inside HAProxy Kubernetes Ingress Controller

  1. Explore HAProxy Kubernetes Ingress Controller documentation

    master

    The HAProxy Kubernetes Ingress Controller (v3.2) provides several configuration and operational surfaces for managing ingress traffic in Kubernetes. Key documentation areas include:

    • Controller options: Configuration settings for the controller itself.
    • Custom resource definitions (CRDs): Information on using custom resources to manage HAProxy configurations.
    • Annotations: Ingress and Service annotations used to fine-tune HAProxy behavior.
    • Prometheus: Instructions for configuring and using Prometheus metrics for monitoring.
    • Lifecycle: Details regarding the lifecycle of the controller and its components.
    • Gateway API (Experimental): Support for the Kubernetes Gateway API.
    • Supervisor (Pebble): Information regarding the Pebble supervisor.
  2. Configure mTLS and Client Certificate Authentication

    master

    To enable client certificate authentication (mTLS), you must have ssl-offloading enabled. These annotations are available on the configmap.

    • client-ca: The path to a Secret (namespace/name) containing the CA certificate in the tls.crt key. You can concatenate multiple CAs in the same key.
    • client-crt-optional: If true, HAProxy accepts the connection even if certificate verification fails. If false (default), HAProxy enforces strict verification.
    • server-ca: Sets the CA for backend servers to enable HAProxy to check backend certificates when sending encrypted traffic to Kubernetes applications. Available on service, configmap, and ingress.
    # Client mTLS configuration
    client-ca: exp/client-ca-secret
    client-crt-optional: true
    
    # Backend TLS verification
    server-ca: "ns1/ca"
  3. Understand HAProxy annotation prefixes and hierarchy

    master

    HAProxy Kubernetes Ingress controller supports multiple annotation prefixes. The following prefixes are interchangeable for the same annotation:

    • ingress.kubernetes.io
    • haproxy.org
    • haproxy.com

    Example: haproxy.com/ssl-redirect and haproxy.org/ssl-redirect are functionally identical.

    Annotation Hierarchy and Priority

    Annotations follow a hierarchical lookup pattern. If a value is not defined at a specific level, the controller looks at the next level up: default $\leftarrow$ ConfigMap $\leftarrow$ Ingress $\leftarrow$ Service

    Priority Rules:

    • Service annotations have the highest priority.
    • Global annotations can only be used in a ConfigMap.
    • Ingress annotations can be used in an Ingress resource or a ConfigMap (to configure all ingress resources).
    • Service annotations can be used in a Service, an Ingress (to configure all services used in that Ingress), or a ConfigMap (to configure all services in use).
  4. Understand Ingress eligibility and matching rules

    master

    The HAProxy Ingress Controller decides which Ingress resources to process based on several inputs.

    Key Concepts

    • Valid IngressClass: An IngressClass where the controller field matches the controller's expected pattern (see below).
    • Matching IngressClass: An Ingress resource's ingressClassName field points to a valid IngressClass.
    • Matching Default IngressClass: A valid IngressClass exists in the cluster with the is-default-class annotation enabled.

    Controller Matching Logic

    The controller's identity is determined by the --ingress.class CLI parameter:

    • If --ingress.class is empty: The controller looks for controller: haproxy.org/ingress-controller.
    • If --ingress.class is set (e.g., prod): The controller looks for controller: haproxy.org/ingress-controller/prod.

    Eligibility Rules

    Controller StateIngress Resource ConditionResult
    --ingress.class is emptyNo ingressClassName AND no default IngressClassAccept
    --ingress.class is emptyNo ingressClassName AND a matching default IngressClass existsAccept
    --ingress.class is emptyHas a ingressClassName pointing to a matching IngressClassAccept
    --ingress.class is setNo ingressClassName AND --EmptyIngressClass is enabledAccept
    --ingress.class is setNo ingressClassName AND a matching default IngressClass existsAccept
    --ingress.class is setHas a ingressClassName pointing to a matching IngressClassAccept
    AnyDoes not meet above criteriaIgnore
  5. Determine Precedence for Backend Custom Annotations

    master

    When multiple custom annotations for the same key are defined, the controller follows a specific priority order. The highest priority source wins:

    1. Service Annotations: Highest priority. Applied only to the specific service.
    2. Ingress Annotations: Applied to services used in the ingress. Warning: This is disabled by default. To enable, use the flag --enable-custom-annotations-on-ingress. Using this can cause inconsistencies if different ingresses target the same service with different values.
    3. ConfigMap Annotations: Lowest priority. Applied to every backend.

    Precedence Summary: Service > Ingress > ConfigMap

  6. Configure Cookie Persistence (Sticky Sessions)

    master

    Sticky sessions allow a client to be consistently routed to the same backend pod. You can enable this via two different methods depending on whether you want to use dynamic cookie keys (recommended for multi-replica controllers) or static server names.

    Important Resolution Logic: These annotations are resolved at the service level, falling back to the configmap default. Setting them on an ingress is ignored to prevent non-deterministic backend configurations. The service value takes precedence over the configmap default.

    • cookie-persistence: Enables dynamic cookies (default). Uses a dynamic cookie key to support sticky sessions across multiple Ingress Controller replicas. Available on configmap, service.
    • cookie-persistence-no-dynamic: Enables sticky sessions using the server name instead of a dynamic key. Available on configmap, service.
    # Using dynamic cookies
    cookie-persistence: "mycookie"
    
    # Using non-dynamic cookies (server name)
    cookie-persistence-no-dynamic: "mycookie"
  7. Configure SSL Offloading and Certificate Management

    master

    HAProxy can offload SSL by looking up Kubernetes secrets. Certificates can be defined in the Ingress object via spec.tls[].secretName, via controller arguments, or via the ssl-certificate annotation in a ConfigMap.

    client-strict-sni

    If true, HAProxy only accepts TLS connections where the SNI matches an existing certificate. If false (default), HAProxy serves the default certificate if no match is found. Available on: configmap

    generate-certificates-signer

    Specifies a kubernetes.io/tls type secret containing a CA certificate used to sign automatically generated certificates. This enables HAProxy's generate-certificates feature on the HTTPS frontend bind line. Available on: configmap

    ssl-certificate

    Sets the name of the Kubernetes secret containing the TLS key and certificate. This replaces the default certificate and is used for QUIC binding. Available on: configmap

    # Example for client-strict-sni
    client-strict-sni: true
    
    # Example for CA signing
    generate-certificates-signer: "default/ca-signing-cert"
    
    # Example for default SSL certificate
    ssl-certificate: "default/tls-secret"
  8. Define GoLang types for a Custom Resource

    master

    When defining GoLang types for a CR (e.g., a Global resource), you must follow these rules:

    1. Embed Metadata: The main resource type and its corresponding List type must embed metav1.TypeMeta and metav1.ObjectMeta to be compatible with the Kubernetes API.
    2. Reuse HAProxy Models: Use the config field in the Spec to hold the HAProxy configuration model. This allows the CR to leverage existing HAProxy configuration logic.
    3. Manual DeepCopy for External Models: Because the Kubernetes deepcopy-gen tool cannot generate deep-copy methods for types in external packages (like github.com/haproxytech/client-native/v6/models), you must implement the DeepCopyInto method manually. This is typically done by using MarshalBinary and UnmarshalBinary from the HAProxy Model.
    4. Required Tags: Use // +genclient and // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object to signal the code generator.
    package v3
    
    import (
      "github.com/haproxytech/client-native/v6/models"
      metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    )
    
    // +genclient
    // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
    
    type Global struct {
      metav1.TypeMeta   `json:",inline"`
      metav1.ObjectMeta `json:"metadata,omitempty"`
    
      Spec GlobalSpec `json:"spec"`
    }
    
    type GlobalSpec struct {
      Config *models.Global `json:"config"`
    }
    
    func (in *GlobalSpec) DeepCopyInto(out *GlobalSpec) {
      *out = *in
      if in.Config != nil {
        b, _ := in.Config.MarshalBinary()
        _ = out.Config.UnmarshalBinary(b)
      }
    }
    
    // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
    
    type GlobalList struct {
      metav1.TypeMeta `json:",inline"`
      metav1.ListMeta `json:"metadata"`
    
      Items []Global `json:"items"`
    }
  9. Configure TCP backends using Backend CRs

    master

    To customize the behavior of the backend generated by a TCP CR (such as load balancing algorithms or server checks), you can use a Backend Custom Resource.

    When a Backend CR is used, its settings will override the default HAProxy configuration generated for the TCP CR's service. For example, setting balance: leastconn in a Backend CR will change the balance keyword in the resulting HAProxy backend section.

    apiVersion: ingress.v3.haproxy.org/v3
    kind: Backend
    metadata:
      name: mybackend
      namespace: haproxy-controller
    spec:
      abortonclose: disabled
      balance:
        algorithm: leastconn
      default_server:
        check-sni: example.com
        resolve-prefer: ipv4
        sni: str(example.com)
        verify: none
      mode: http
      name: toto
  10. Understand Custom Annotations

    master

    Custom annotations are a mechanism to expose HAProxy configuration options to developers while maintaining strict validation and security controlled by an administrator. Unlike regular annotations or config snippets, custom annotations use Custom Resource Definitions (CRDs) in the background to enforce schema validation and access control.

    Key Characteristics

    • Granularity: Unlike standard CRDs, custom annotations can be scoped to specific HAProxy sections, services, ingresses, or namespaces.
    • Validation: They use Common Expression Language (CEL) to define validation rules, ranging from simple to strict, to prevent misconfigurations.
    • Security: They create a separation of concerns between Administrators (who define and limit the available annotations via CRDs) and Developers/Teams (who consume the pre-defined annotations).
    • Reliability: Compared to config snippets, custom annotations provide better protection against typos and invalid syntax.
    • Monitoring: If a custom annotation fails validation, it is still included in the HAProxy configuration but appears as a comment on the relevant frontend or backend, including an error message explaining the failure.
  11. Implement percentage-based traffic splitting for Canary releases

    master

    To split traffic by a fixed percentage, apply the route-acl annotation to your staging (canary) service using the rand(100) function.

    For example, to route 25% of traffic to a staging backend, use rand(100) lt 25. The HAProxy controller translates this into a backend selection rule similar to: use_backend default-<service-name>-<port> if { var(txn.host) <host>} { rand(100) lt 25 }.

    Important Ingress Rules Note: When defining multiple backends (production and staging) in a single Ingress resource, ensure they do not share the same path. If they share a path, the latter definition will overwrite the former. To avoid collision and confusion, it is recommended to use separate Ingress resources for staging and production rules.

    # Staging Service with 25% Canary weight
    apiVersion: v1
    kind: Service
    metadata:
      name: echo-staging
      annotations:
        route-acl: "rand(100) lt 25"
    spec:
      ports:
        - name: http
          port: 80
          protocol: TCP
          targetPort: http
      selector:
        app: echo-staging
    ---
    # Ingress routing to both
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: echo
    spec:
      rules:
      - host: echo.haproxy.local
        http:
          paths:
            - path: /
              pathType: ImplementationSpecific
              backend:
                service:
                  name: echo-prod
                  port:
                    name: http
            - path: /staging # Use a distinct path to avoid overwriting
              pathType: ImplementationSpecific
              backend:
                service:
                  name: echo-staging
                  port:
                    name: http