DOCSEARCHbyAlgolia
Submit Docs

EgressGateway

repository·main·Indexed 43 days ago

https://github.com/spidernet-io/egressgateway

A Kubernetes networking solution that provides stable, configurable Egress IP addresses for Pods to solve IP instability caused by Pod rescheduling. It supports dual-stack (IPv4/IPv6), high availability, and granular policy control by Destination CIDR. Compatible with CNI solutions including Calico, Flannel, Weave, Spiderpool, and Cilium (native mode). Includes the egctl CLI for managing Virtual IP (VIP) resources.

Tokens
67.1K
Snippets
145
Records
206
Agent score
54%

What's inside EgressGateway

  1. Key features and capabilities of EgressGateway

    main

    EgressGateway provides several advanced networking capabilities for Kubernetes environments:

    • Dual-stack Support: Solves IPv4 and IPv6 connectivity issues for seamless communication across protocol stacks.
    • High Availability: Ensures egress node high availability to prevent single-point failures from affecting connectivity.
    • Granular Policy Control: Supports fine-grained filtering of egress policies, including filtering by Destination CIDR.
    • Application-level Control: Allows filtering of specific egress applications (Pods) for precise outbound traffic management.
    • Multi-instance Support: Supports multiple egress gateway instances to handle communication between different network partitions or clusters.
    • Namespaced Egress IPs: Supports assigning specific egress IPs to individual namespaces.
    • Automatic Traffic Detection: Automatically detects cluster traffic to inform egress gateway policies.
    • Namespace Defaults: Supports defining default egress instances for namespaces.
    • Environment Compatibility: Works with low kernel versions, making it suitable for diverse Kubernetes deployment environments.

  2. What is EgressGateway

    main

    EgressGateway is an open-source solution for Kubernetes (k8s) clusters designed to provide stable, predictable Egress IP addresses for Pods accessing external services.

    In standard Kubernetes networking, a Pod's egress IP is typically determined by the node it resides on (Overlay network) or its own IP (Underlay network). This causes IP instability when Pods are rescheduled. EgressGateway solves this by allowing administrators to configure specific egress IP addresses at either the tenant (namespace) level or the cluster level. When Pods access the external network, EgressGateway ensures they consistently use the configured Egress IP, facilitating easier IP management, network fault diagnostics, and external traffic control.

  3. What is EgressGateway and why use it

    main

    EgressGateway is an open-source egress gateway for Kubernetes clusters designed to solve the problem of unstable egress IP addresses. In standard Kubernetes networking (Overlay or Underlay), a Pod's egress IP changes whenever it is rescheduled, making IP-based traffic management and network diagnostics difficult from outside the cluster.

    EgressGateway provides a stable egress IP for tenant-level or cluster-level workloads, ensuring that when Pods access external networks, they use a consistent, pre-configured IP address regardless of the underlying CNI or Pod scheduling.

  4. Configure EgressPolicy to control Pod egress traffic

    main

    The EgressPolicy Custom Resource Definition (CRD) is a tenant-level resource used to specify which Pods use an Egress policy when accessing specific destination CIDRs, and which IP addresses the Egress traffic should use.

    By defining an EgressPolicy, you can route traffic from selected Pods through a specific egressGatewayName and assign them specific egress IPs (IPv4, IPv6, or the Node IP).

    apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressPolicy
metadata:
  namespace: "default"
  name: "policy-test"
spec:
  egressGatewayName: "eg1"
  egressIP: 
    ipv4: ""
    ipv6: ""
    useNodeIP: false
    allocatorPolicy: default
  appliedTo: 
    podSelector: 
      matchLabels: 
        app: "shopping"
    podSubnet: 
    - "172.29.16.0/24"
    - 'fd00:1/126'
  destSubnet: 
    - "10.6.1.92/32"
    - "fd00::92/128"
  priority: 100

  5. Configure Gateway Node Selection and EIP Allocation

    main

    EgressGateway uses specific logic to decide which nodes act as gateways and which EIPs are assigned to them.

    Gateway Node Selection Modes

    When an EgressPolicy needs to select a gateway node, it uses one of these modes:

    • Average selection: Selects the node currently serving the fewest number of gateway roles.
    • Minimum node selection: Attempts to reuse the same node as a gateway node whenever possible.
    • Limit selection: A node can only serve as a gateway for a limited number of EgressPolicies (defaults to 5). The controller prefers nodes below this limit; if all nodes are at the limit, it performs a random selection.

    EIP Allocation Logic

    EIPs are assigned to EgressGateways based on these rules:

    • Random allocation: Selects any available EIP at random.
    • Preferred use of unallocated EIPs: Prioritizes unused EIPs. If all EIPs are currently assigned, it falls back to random allocation of a used EIP.
    • Limit selection: An EIP can be used by a maximum number of EgressPolicies (defaults to 5). The controller prefers EIPs that have not reached this limit.

    EIP Recycling

    If an EIP is no longer in use, it is reclaimed by deleting the EIP field within the eips list.

  6. How Egress Ignore CIDR works in the Data Plane

    main

    When an EgressGatewayPolicy is configured with an empty destSubnet, the EgressGateway agent uses an ipset named egress-ingore-cidr to filter traffic.

    Instead of matching a specific destination, the iptables rule uses the negation operator (!) to match any destination IP that is NOT contained within the egress-ingore-cidr set. This ensures that internal cluster traffic (Service IPs, Node IPs, Pod IPs) is ignored by the gateway, while all other external traffic is marked and routed through the Egress Gateway.

    # Example of the generated iptables rule using the negation match
iptables -A EGRESSGATEWAY-MARK-REQUEST -t mangle -m conntrack --ctdir ORIGINAL \
-m set --match-set !egress-ingore-cidr dst  \
-m set --match-set $IPSET_RULE_SRC_NAME src  \
-j MARK --set-mark $NODE_MARK -m comment --comment "rule uuid: mark request packet"

  7. How tenant-level default EgressGateways work

    main

    Setting a default EgressGateway at the tenant (namespace) level simplifies the use of EgressPolicy by removing the need to manually specify the spec.egressGatewayName in every policy.

    Priority Logic:

    • Tenant-level default: If a namespace is labeled with a default EgressGateway, that gateway is used first.
    • Cluster-level default: If no tenant-level default is configured, the system falls back to the cluster-wide default EgressGateway.

    This allows administrators to enforce specific egress paths for all workloads within a specific namespace automatically.

  8. Use EgressClusterPolicy for cluster-level egress rules

    main

    The EgressClusterPolicy CRD defines egress policy rules at the cluster level. It is similar to the EgressPolicy CRD but includes the spec.appliedTo.namespaceSelector attribute, allowing you to target pods across multiple namespaces using a namespace selector combined with a pod selector.

    To apply a policy to specific pods within selected namespaces, use the appliedTo field to define both a namespaceSelector and a podSelector. The policy will only apply to pods that match the podSelector AND reside within a namespace that matches the namespaceSelector.

    apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressClusterPolicy
metadata:
  name: "policy-test"
spec:
  priority: 100
  egressGatewayName: "eg1"
  egressIP:
    ipv4: ""
    ipv6: ""
    useNodeIP: false
  appliedTo:
    podSelector:
      matchLabels:
        app: "shopping"
    podSubnet:
      - "172.29.16.0/24"
      - 'fd00:1/126'
    namespaceSelector:   # (1)
      matchLabels:
        app: "shopping"
  destSubnet:
    - "10.6.1.92/32"
    - "fd00::92/128"

  9. Egress IP Support Overview

    main
    The Egress IP proposal introduces enhancements to the EgressGateway CRD and EgressGatewayEgressPolicy to allow for fine-grained control over outbound traffic. Instead of relying solely on the node's IP address, users can now define specific Egress IP (EIP) ranges. This allows different services or tenants to be assigned different EIPs, enabling flexible network planning (e.g., different firewall policies or bandwidth limits per EIP) and improved security by isolating outbound traffic per tenant.

  10. EgressGateway Custom Resource Definitions (CRDs)

    main

    The Egress IP implementation uses several Kubernetes Custom Resource Definitions to manage tunnels, gateway nodes, and egress policies:

    • EgressTunnel: A cluster-level resource (one per Kubernetes Node) that records tunnel NIC information (IPv4, IPv6, MAC, and parent NIC details) used for cross-node communication. It includes a globally unique mark (e.g., 0x26000000) used for traffic tagging.
    • EgressGateway: A cluster-level resource used to define a set of nodes that act as egress gateways. It specifies the range of available Egress IPs (EIPs) and the nodeSelector used to pick gateway nodes.
    • EgressGatewayPolicy: A tenant-level resource that specifies which Pods follow a specific egress policy and which IP address they should use. It can target Pods via podSelector (labels) or podSubnet (network segments).
    • EgressEndpointSlice: A dependent resource of EgressGatewayPolicy that aggregates matched endpoints to improve scalability when using podSelector.

  11. Understand the EgressGateway Datapath architecture

    main

    The EgressGateway datapath categorizes rule application into three distinct scopes based on the node's role in the network:

    1. All nodes: Rules applied to every node in the cluster to handle traffic retagging, label preservation, and the creation of aggregate chains (e.g., EGRESSGATEWAY-MARK-REQUEST, EGRESSGATEWAY-NO-SNAT, EGRESSGATEWAY-SNAT-EIP).
    2. Non-Egress Gateway nodes: Nodes where Pods are scheduled. These nodes use ipsets to identify policy-matched traffic, apply a NODE_MARK to ensure traffic is tunneled, and use policy routing (ip rule) to direct traffic to the correct table.
    3. Egress Gateway nodes: Nodes that act as the exit point. These nodes use ipsets to identify traffic and apply Source Network Address Translation (SNAT) using a specific Egress IP ($EIP).

    Traffic matching an EgressGatewayPolicy is retagged when a node first becomes a gateway node or during a node join.

  12. How EgressGateway node selection and EIP allocation works

    main

    The system uses a two-step process to map policies to physical infrastructure.

    1. Gateway Node Selection

    When a policy is created, the controller selects a gateway node based on the EgressGateway selection policy:

    • Average Selection (AverageSelecton): Chooses the node currently serving the fewest gateway policies.
    • Least Node Selection: Attempts to consolidate policies onto the same node.
    • Limit Selection: A node can only serve a maximum number of policies (default is 5). Once the limit is reached, the controller selects a different node.

    2. EIP Allocation

    Once a node is selected, an EIP is assigned using one of these strategies:

    • Random: Picks a random IP from the pool.
    • Unassigned First: Prioritizes IPs that are not currently in use.
    • Limit Selection: An EIP can be shared by a maximum number of policies (default is 5).

    3. EIP Reclamation

    When a policy is deleted or no longer matches, the assigned EIP is automatically reclaimed by removing it from the EgressGateway.status.nodeList.eips field.