Terway CNI Plugin

IP Prefix mode is a prefix-based IP Address Management (IPAM) mode provided by Terway. Unlike the traditional mode that allocates individual IP addresses, Prefix mode allocates CIDR prefixes (e.g., /28) to Elastic Network Interfaces (ENIs). Each /28 prefix contains a continuous block of 16 IP addresses.

Key Differences

Use Cases

• Large-scale clusters: When a massive number of Pods need IP addresses.
• High-density deployment: When running many Pods per node.
• Fast startup: Reduces API calls to improve Pod startup speed.
• IPv6 Dual-stack: Scenarios requiring both IPv4 and IPv6 addresses.

Terway implements a protection mechanism to ensure HighPerformance (HP) ENIs are reserved for RDMA Pods. This involves two main components:

1. ENI Blacklist Tag Mechanism

Terway identifies ENIs to skip using a blacklist system:

• Default Hardcoded Blacklist: Automatically recognizes creator=alibabacloud-erdma-controller and terway.alibabacloud.com/excluded=true.
• ConfigMap Supplement: Users can extend the blacklist via ConfigMap.

2. The Unschedulable State

When an ENI matches the blacklist, Terway performs the following:

• Marking: The ENI is retained in the Custom Resource (CR) but marked as Unschedulable.
• Scheduling: During IP allocation, Terway skips any ENI marked Unschedulable.
• Lifecycle Management:
  • Unschedulable ENIs are skipped during IP release/deletion processes (the card and its IPs are not deleted automatically).
  • Unschedulable ENIs are excluded from IP water level calculations and deletion quotas.
  • If a node only has Unschedulable ENIs with idle IPs, the node will report InsufficientIP.

A PodNetworking resource is a cluster-level Custom Resource (CR) used to describe a network plane configuration, including settings like vSwitch and SecurityGroup.

Key Concepts:

• Relationship: Pods and PodNetworking resources have a many-to-many relationship. A Pod can reference 1 or more network planes.
• Lifecycle: PodNetworking configurations are applied only at the time of Pod creation. Modifying an existing PodNetworking resource will not affect Pods that have already been created.
• Routing: In a multi-network setup, only one network plane can be set as the default route; other planes can be configured with specific detailed routes.
• Limitations:
  • ACS (Alibaba Cloud Serverless) does not support custom route configurations (defaultRoute and routes).
  • Multi-network functionality currently supports Trunk ENI or Dedicated ENI, but does not support Lingjun nodes.

apiVersion: network.alibabacloud.com/v1beta1
kind: PodNetworking
metadata:
  name: example
spec:
  allocationType:
    type: Fixed
    releaseStrategy: TTL
    releaseAfter: "1h"
  securityGroupIDs:
  - sg-bpxxxx
  vSwitchOptions:
  - vsw-bpxxxx

When an ENI is marked as Unschedulable (due to blacklist tags or being a HighPerformance ENI without RDMA requirements), Terway follows a "show but don't manage" policy to ensure coexistence with other controllers:

• IP Allocation: Skipped in assignIPFromLocalPool and assignEniPrefixWithOptions.
• Capacity/Quota: Skipped in validateENI (no new IPs added) and countTotalIdleIPs (not counted in watermarks).
• Reclamation/GC: The releaseUnUsedIP and handleStatus paths explicitly skip Unschedulable ENIs. Terway will never Detach, Delete, or UnAssign IPs from these cards.
• Evacuation: Existing Pods on these ENIs are left untouched. The ENI is naturally evacuated as existing Pods exit.

Note: If you have a HighPerformance ENI that is not managed by a controller and you want Terway to use it, you must manually unbind it from the node via the ECS Console, as Terway will not automatically claim unowned HighPerformance ENIs to avoid race conditions.

Terway supports three primary networking modes depending on your connectivity and scale requirements:

1. VPC Mode (Overlay)

• Mechanism: Uses an overlay network where the Pod CIDR is different from the host VPC CIDR. Connectivity is achieved via Aliyun VPC routing tables.
• Connectivity: Pods connect to the host via veth devices. Traffic is routed through the host's network namespace using iptables and tc for load balancing and traffic control.
• Best for: Standard overlay networking where Pod IPs are distinct from host IPs.

2. ENI Mode (Direct VPC Access)

• Mechanism: Binds an Aliyun Elastic Network Interface (ENI) directly to the node. Pods use this ENI to communicate.
• Connectivity: The Pod's network interface is essentially the ENI moved into the Pod's network namespace. Pods can communicate directly with VPC and public resources.
• Constraint: Limited by the number of ENIs supported by the specific ECS instance type.

3. ENI Multi-IP Mode (High Density)

• Mechanism: Leverages multiple secondary IP addresses on a single ENI to increase Pod density.
• Implementation Options:
  • Veth Policy Routing: Uses veth pair to connect the Pod to the host. Requires policy routing on the host to ensure traffic for secondary IPs exits through the correct ENI.
  • IPvlan L3/L2: Uses Linux ipvlan (available in kernels 4.2+) to create virtual sub-interfaces from the ENI. This mode is simpler and offers better performance than veth policy routing.
• Best for: High-density Pod deployments requiring direct VPC connectivity.

Terway uses the MergePatch mechanism (RFC 7396) via the json-patch library to merge dynamic node configurations into the default configuration.

Key Behaviors:

• Partial Merging: For nested objects like vswitches, Terway performs a partial merge. If a dynamic config specifies a subset of keys in a nested object, the other keys from the default config are preserved.
• Removing Keys: To remove a key from a nested object during a merge, set its value to null in the dynamic configuration.

Examples:

Partial Merge Example: Default vswitches:

{
  "vswitches": {
    "cn-hangzhou-g": ["vsw-xxx"],
    "cn-hangzhou-i": ["vsw-yyy"]
  }
}

Dynamic vswitches:

{
  "vswitches": {
    "cn-hangzhou-g": ["vsw-10000"]
  }
}

Resulting configuration:

{
  "vswitches": {
    "cn-hangzhou-g": ["vsw-10000"],
    "cn-hangzhou-i": ["vsw-yyy"]
  }
}

Removing a Key Example: To remove cn-hangzhou-i from the resulting configuration, use null:

{
  "vswitches": {
    "cn-hangzhou-g": ["vsw-10000"],
    "cn-hangzhou-i": null
  }
}