EdgeMesh

EdgeMesh is a data plane component for KubeEdge clusters. It provides simple service discovery and traffic proxying for applications, effectively masking complex network structures in edge computing scenarios.

Key benefits include:

• High Availability: Uses LibP2P to connect edge nodes. It distinguishes between LAN communication (direct) and cross-LAN communication (direct tunnels via hole punching if successful, otherwise relayed via a relay).
• High Reliability (Offline Scenarios): Metadata is delivered via the KubeEdge edge-cloud channel, removing the need to access the cloud apiserver. It includes a lightweight node-level DNS server so service discovery does not depend on cloud CoreDNS.
• Extreme Lightweightness: Only one edgemesh-agent is deployed per node, saving edge resources compared to a full stack of CoreDNS + Kube-Proxy + CNI.

The EdgeMesh CNI implementation aims to provide a complete, independent CNI plugin capable of managing complex edge-to-cloud and edge-to-edge network topologies.

Key Objectives:

• IP Address Management (IPAM): Ensure Pod IP uniqueness across the entire cluster by integrating with Edgecore to List/Watch Pods and network segments.
• Local Network Support: Provide standard CNI functionality (bridge creation, IP allocation, routing) for containers within the same subnet.
• Cross-Subnet P2P Service: Identify Pod IP traffic destined for different subnets and intercept it to provide P2P connectivity.
• Performance Optimization: Use technologies like eBPF (instead of massive iptables rule sets) to handle traffic interception and forwarding efficiently without degrading node performance.

Target Outcome: Containers in the cluster can communicate directly via Pod IP addresses regardless of whether they are located at the edge or in the cloud.

The Container Network Interface (CNI) is a standardized, plugin-based network solution for containers. It abstracts the differences between upper-layer network requirements and underlying network provider implementations.

Core CNI Requirements

According to the CNI specification, a CNI implementation must:

• Provide a file format for network administrators to define configurations.
• Provide a protocol/API for the Container Runtime Interface (CRI) to invoke.
• Execute network configuration based on the provided configuration file when called by CRI.
• Reserve the capability to invoke other plugins.
• Standardize the data format returned to the CRI.

Typical CNI Components

• Main Plugins: Create network devices (e.g., bridge, ipvlan, macvlan, ptp for Veth Pairs).
• IPAM (IP Address Management): Manages IP allocation (e.g., dhcp, host-local, static).
• Meta Plugins: Perform auxiliary tasks like tuning (sysctl parameters), portmap (iptables-based port mapping), bandwidth (traffic limiting), or firewall (iptables/firewalld rules).

When both kube-proxy and edgemesh are deployed in a Kubernetes cluster, EdgeMesh uses a Hybrid Proxy mechanism to ensure compatibility.

EdgeMesh achieves this by inserting the KUBE-PORTALS-CONTAINER chain into the PREROUTING and OUTPUT chains of the host's nat iptables. To successfully hijack service traffic for cloud-edge communication, this chain must be positioned in front of the KUBE-SERVICES chain.

If edgemesh-agent starts after kube-proxy, it places its rules ahead of the standard Kubernetes service rules, allowing EdgeMesh to take precedence over kube-proxy for ClusterIP traffic.

$ iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 1379K packets, 395M bytes)
 pkts bytes target     prot opt in  out     source               destination
   52 13980 KUBE-PORTALS-CONTAINER  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */
1375K  394M KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
...

EdgeMesh operates through an edgemesh-agent which contains several core functional modules. Since version v1.12.0, the capabilities of the former edgemesh-server have been merged into the tunnel module of the edgemesh-agent. This allows any agent with relay capability to automatically act as a relay server for other nodes.

The core components of the edgemesh-agent are:

• Proxier: Configures kernel iptables rules to intercept requests and route them to the EdgeMesh process.
• DNS: A built-in DNS resolver that resolves node-level DNS requests into service cluster IPs.
• LoadBalancer: Forwards requests to backend instances using various load balancing strategies.
• Controller: Fetches metadata (Services, Endpoints, Pods, etc.) by accessing the Kubernetes or KubeEdge apiserver.
• Tunnel: Implements LibP2P-based communication, utilizing automatic relay, MDNS, and hole punching to enable cross-subnet communication.

When designing or troubleshooting edge networking with EdgeMesh, be aware of two primary limitations addressed by the CNI proposal:

1. Lack of Cross-Subnet Traffic Forwarding in Mainstream CNI

Mainstream CNI plugins generally lack the ability to forward traffic across different subnets; they rely on Layer 3 (L3) reachability. If the underlying network layers (switches/routers) do not have routes between subnets, the CNI cannot bridge that gap. Even if Kubernetes sees the nodes as part of a cluster, if the physical/underlying network cannot route between those node IPs, communication will fail.

2. EdgeMesh PodIP Forwarding Limitation

While EdgeMesh supports cross-subnet traffic forwarding at the ClusterIP level (by intercepting Service traffic via iptables), it currently lacks PodIP traffic forwarding capabilities.

• How it works currently: EdgeMesh intercepts traffic destined for a Service via iptables rules and routes it through EdgeMesh's P2P service.
• The Problem: Traffic sent directly to a PodIP is not intercepted by EdgeMesh's iptables rules. Consequently, the traffic attempts to use the underlying NodeIP/network device, which lacks the specific routing entries to reach the destination Pod in a different subnet, leading to 'destination unreachable' errors.

Currently, EdgeMesh provides cross-subnet traffic forwarding specifically for Service (SVC) level traffic, rather than direct Pod IP traffic.

It works by hijacking Service traffic using iptables rules. By default, a port is exposed on 169.254.96.16, and iptables rules are configured to intercept traffic destined for a Service IP and redirect it to the EdgeMesh application layer at 169.254.96.16:40505.

Limitations:

• It does not support direct Pod IP traffic forwarding.
• Relying solely on iptables for all Pod IP traffic would cause rule explosion and degrade system performance.
• It relies on existing CNI plugins for basic Pod networking, which may fail in complex edge scenarios where nodes are in different subnets (L2 connectivity is often required by mainstream CNIs).

# Example of iptables NAT rules hijacking Service traffic to EdgeMesh
# Traffic to 10.106.117.151 (Service IP) on port 12345 is redirected to 169.254.96.16:40505
[root@k8s-node1 ~]# iptables -t nat -nvL --line | grep  10.106.117.151
9       10   600 DNAT       tcp  --  *      *       0.0.0.0/0            10.106.117.151       /* testzone/hostname-svc:http-0 */ tcp dpt:12345 to:169.254.96.16:40505

The EdgeMesh CNI is designed to provide complete network resource management and P2P connectivity, especially in complex edge scenarios where containers and nodes reside in different subnets. Its core requirements include:

1. Cluster-scale IP address partitioning and management

• Network Resource Management: Creating bridges, network cards, and allocating IP addresses within a node while ensuring synchronization with KubeEdge to prevent address conflicts.
• Unique Cluster Addresses: Using List/Watch on Pods and cluster network segment resources via Edgecore to ensure every container in the cluster (cloud and edge) has a unique IP address.

2. Pod IP hierarchical traffic forwarding

• Intra-subnet (Same LAN) Forwarding: Implementing standard CNI functions to manage network resources and routing rules for containers within the same subnet.
• Inter-subnet (Cross-LAN) P2P Services: Identifying Pod IPs in different subnets and intercepting/forwarding traffic to provide P2P connectivity. This is intended to move away from heavy iptables rules toward more efficient technologies like eBPF to maintain high forwarding efficiency.

EdgeMesh security relies on a multi-step handshake with CloudCore to establish mutual authentication:

1. CA Retrieval: EdgeMesh requests the CA certificate from CloudCore. The validity of the CA certificate is verified by hashing it and comparing it against a cahash contained within the provided token.
2. CSR Submission: EdgeMesh generates a local key pair and submits a Certificate Signing Request (CSR) to CloudCore. This request includes a token in the HTTP header.
3. Certificate Issuance: CloudCore verifies the token and uses the CA certificate to sign the CSR, returning a valid certificate to EdgeMesh.
4. mTLS Establishment: Once certificates are obtained, edgemesh-server and edgemesh-agent (or agents with each other) establish mutual authentication TLS using the go-libp2p-tls-ca library.

Note on Certificate Types: Unlike standard EdgeCore, EdgeMesh requires certificates with both x509.ExtKeyUsageClientAuth and x509.ExtKeyUsageServerAuth usages.

By default, EdgeMesh targets Service-level (SVC) traffic rather than individual PodIP traffic. It exposes a port on 169.254.96.16 and uses iptables NAT rules to hijack Service traffic, forwarding it to the EdgeMesh application layer.

In a typical setup, traffic destined for a Cluster IP is matched via a specific iptables chain (e.g., KUBE-SVC-PX4OGZX33ST3IKD4) and DNAT'ed to the EdgeMesh proxy address 169.254.96.16:40505.

# Example of iptables NAT rules used by EdgeMesh to hijack Service traffic
# Traffic to Cluster IP 10.106.117.151 on port 12345 is DNAT'ed to 169.254.96.16:40505

[root@k8s-node1 ~]# iptables -t nat -nvL --line | grep  10.106.117.151
11       0     0 KUBE-MARK-MASQ  tcp  --  *      *      !10.244.0.0/16        10.106.117.151       /* testzone/hostname-svc:http-0 cluster IP */ tcp dpt:12345
12       0     0 KUBE-SVC-PX4OGZX33ST3IKD4  tcp  --  *      *       0.0.0.0/0            10.106.117.151       /* testzone/hostname-svc:http-0 cluster IP */ tcp dpt:12345
9       10   600 DNAT       tcp  --  *      *       0.0.0.0/0            10.106.117.151       /* testzone/hostname-svc:http-0 */ tcp dpt:12345 to:169.254.96.16:40505
9        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            10.106.117.151       /* testzone/hostname-svc:http-0 */ tcp dpt:12345 to:169.254.96.16:40505