Kube-OVN Documentation

Kube-OVN is a CNCF Sandbox project that integrates OVN-based Network Virtualization with Kubernetes. It is designed to provide advanced networking features that go beyond standard Kubernetes CNI capabilities, specifically focusing on multi-tenancy, VM support, and high-performance networking.

Key capabilities include:

Multi-Tenancy & Isolation

• VPC Support: Independent address spaces per tenant with dedicated infrastructure (EIPs, NAT gateways, security groups, and load balancers).
• Namespaced Subnets: Unique Subnets per Namespace (backed by Logical Switches), allowing for shared or isolated IP allocation.
• Subnet Isolation: Ability to deny traffic from outside a Subnet or whitelist specific IP ranges.
• Namespaced Gateways: Dedicated Egress gateways per Namespace.

Advanced Networking & Connectivity

• Vlan/Underlay Support: Support for underlay and VLAN modes for direct physical network connectivity and high performance.
• Non-Primary CNI Mode: Can operate as a secondary CNI alongside primary CNIs (like Cilium or Calico) using Network Attachment Definitions (NADs).
• Multi-Cluster Network: L3 connectivity between different Kubernetes or OpenStack clusters.
• BGP Support: Exposing Pod/Subnet IPs to external networks via BGP.
• DualStack IP Support: Support for IPv4-only, IPv6-only, or DualStack modes.
• Direct External Connectivity: Direct exposure of Pod IPs to the external network.

Workload & VM Support

• KubeVirt Integration: Seamless LiveMigration of VMs without network interruption.
• Static IP Addresses: Support for allocating specific static IP addresses to workloads.
• Pod NAT and EIP: Management of external traffic and External IPs similar to traditional VMs.
• IPAM for Multi NIC: Cluster-wide IPAM for use with other CNI plugins (e.g., macvlan, vlan) to leverage Kube-OVN's subnet and static IP functions.

Performance & Operations

• Embedded Load Balancers: High-performance distributed L2 Load Balancers that can replace kube-proxy.
• Distributed Gateways: Every node can act as a gateway for external connectivity.
• Dynamic QoS: On-the-fly configuration of traffic rate, priority, loss, and latency for Pods and Gateways.
• Hardware Offload: Offloading OVS flow tables to hardware to reduce CPU usage and boost performance.
• Traffic Mirroring: Duplicating network traffic for monitoring and diagnostics.
• Observability: Prometheus & Grafana integration for network quality metrics (latency, connectivity) and built-in troubleshooting tools.

In a split-cluster topology, the ovn-central control plane runs in a management cluster, while agents run in tenant clusters. To allow tenant clusters to reach the management cluster's OVN DB, you must expose the ovn-nb / ovn-sb / ovn-northd Services via LoadBalancer or NodePort.

Single-replica mode is a prerequisite for this pattern to prevent leader flapping during LoadBalancer health checks.

1. Configure Management Cluster

Set the service type to LoadBalancer in your Helm values or install.sh script.

2. Configure Tenant Clusters

Point the tenant data plane to the LoadBalancer IP using the OVN_DB_IPS configuration.

Security Warning

Exposing OVN DB over plain TCP is insecure. You should enable SSL (networking.ENABLE_SSL=true) and distribute the OVN certificates to the tenant clusters.

### Helm (Management Cluster)

```yaml
# values.yaml on the management cluster
OVN_CENTRAL_MODE: single

ovn-central:
  storage:
    storageClassName: my-csi
    size: 10Gi
  service:
    type: LoadBalancer
    loadBalancerIP: 10.99.99.99       # provider-dependent; omit to let LB pick
    externalTrafficPolicy: Local      # optional; preserves source IPs

install.sh (Management Cluster)

ENABLE_SINGLE_REPLICA_OVN=true \
  OVN_CENTRAL_STORAGE_CLASS=my-csi \
  OVN_CENTRAL_SERVICE_TYPE=LoadBalancer \
  OVN_CENTRAL_LB_IP=10.99.99.99 \
  OVN_CENTRAL_EXTERNAL_TRAFFIC_POLICY=Local \
  bash dist/images/install.sh

Wiring Tenant Clusters (Data Plane)

# data-plane values (or chart fork) — pseudo-config
OVN_DB_IPS: 10.99.99.99      # the LoadBalancer VIP from above

To install from source, you must first prepare your Kubernetes nodes by applying specific labels for OS type, control-plane role, and OVS datapath type. Then, use helm install with the MASTER_NODES parameter. If MASTER_NODES is not specified, Helm will automatically use the internal IPs of nodes labeled with kube-ovn/role=master.

# Prepare nodes
kubectl label node -lbeta.kubernetes.io/os=linux kubernetes.io/os=linux --overwrite
kubectl label node -lnode-role.kubernetes.io/control-plane  kube-ovn/role=master --overwrite
kubectl label node -lovn.kubernetes.io/ovs_dp_type!=userspace ovn.kubernetes.io/ovs_dp_type=kernel  --overwrite

# Standard install (single master)
helm install --debug kubeovn ./charts/kube-ovn --set MASTER_NODES=${Node0}

# High availability install (multiple masters)
helm install --debug kubeovn ./charts/kube-ovn --set MASTER_NODES=${Node0},${Node1},${Node2}

# Upgrade existing installation
helm upgrade --debug kubeovn ./charts/kube-ovn --set MASTER_NODES=${Node0},${Node1},${Node2}

⚠️ Breaking Change Warning

Migrating from v1 to v2 is a breaking migration. The v2 chart uses standard Kubernetes labels (app.kubernetes.io/name, app.kubernetes.io/part-of) for spec.selector.matchLabels. Because these selectors are immutable on Deployments and DaemonSets, a standard helm upgrade will fail.

Every workload must be deleted and recreated. Plan for a maintenance window.

Preparation

Before migrating, generate the v2 templates with a dry-run to compare them against your running resources and verify your values file (note that some keys have changed, e.g., networking.NET_STACK $\rightarrow$ networking.stack):

helm template kube-ovn ./charts/kube-ovn-v2 -f your-values.yaml > v2-manifests.yaml

Migration Strategy

Migrate components in the following order, waiting for each to become healthy before proceeding. You can verify pod health using:

kubectl get pods -n kube-system -l app.kubernetes.io/part-of=kube-ovn

Install the Kube-OVN Helm chart in the management cluster using installMode: controlPlaneOnly. This setup creates the OVN database and exposes it via a LoadBalancer.

Key Configuration Requirements:

• Set OVN_CENTRAL_MODE: single.
• Configure ovn-central.storage with a suitable storageClassName.
• Set ovn-central.service.type to LoadBalancer.
• It is strongly recommended to set networking.ENABLE_SSL: true.

Post-Installation Steps:

1. Retrieve the LoadBalancer's IP/hostname.
2. Export the kube-ovn-tls Secret from the management cluster to sync it to the tenant cluster.

# mgmt-values.yaml
namespace: kube-system

installMode: controlPlaneOnly
OVN_CENTRAL_MODE: single

ovn-central:
  storage:
    storageClassName: my-csi
    size: 10Gi
  service:
    type: LoadBalancer
    loadBalancerIP: 10.99.99.99        # provider-dependent; omit for auto
    externalTrafficPolicy: Local       # preserves tenant source IPs (optional)

networking:
  ENABLE_SSL: true                     # strongly recommended

# Install the chart
helm install --kube-context=mgmt kube-ovn ./charts/kube-ovn -f mgmt-values.yaml

# Pull the assigned VIP
kubectl --context=mgmt -n kube-system get svc ovn-nb -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

# Export the TLS Secret for syncing to the tenant cluster
kubectl --context=mgmt -n kube-system get secret kube-ovn-tls -o yaml > kube-ovn-tls.yaml

Follow these steps to build the kernel module and install it on your Kubernetes nodes.

1. Build the module

Navigate to the directory corresponding to your kernel version and run make all:

• For 3.x kernels: use the 3.x directory.
• For 4.x to 6.x kernels: use the 4.x-6.x directory.

make all

2. Install the module

Copy the resulting kube_ovn_fastpath.ko file to your Kubernetes nodes and load it using insmod:

insmod kube_ovn_fastpath.ko

3. Verify installation

Check dmesg for the following initialization logs to confirm success:

[timestamp] init_module,kube_ovn_fastpath_local_out
[timestamp] init_module,kube_ovn_fastpath_post_routing
[timestamp] init_module,kube_ovn_fastpath_pre_routing
[timestamp] init_module,kube_ovn_fastpath_local_in

4. Remove the module

To unload the module, use rmmod:

rmmod kube_ovn_fastpath.ko

IMPORTANT

To ensure the module survives a system reboot, you must manually add configuration to load the module at boot time.