DOCSEARCHbyAlgolia
Submit Docs

Whereabouts IPAM CNI Plugin

repository·master·Indexed 18 days ago

https://github.com/k8snetworkplumbingwg/whereabouts

Whereabouts is a cluster-wide IP Address Management (IPAM) CNI plugin that assigns IPv4 and IPv6 addresses across all nodes in a Kubernetes cluster using a shared data store (etcd or Kubernetes Custom Resources). It avoids the limitations of node-local assignment by tracking allocations cluster-wide and always assigning the lowest available address in a specified range. It supports DualStack configurations, Fast IPAM via preallocated node slices, and integration with Multus CNI through NetworkAttachmentDefinition.

Tokens
8.2K
Snippets
27
Records
38
Agent score
64%

What's inside Whereabouts

  1. What is Whereabouts?

    master
    Whereabouts is an IP Address Management (IPAM) CNI plugin designed to assign IP addresses cluster-wide. Unlike the host-local CNI plugin, which only manages IPs on a single node, Whereabouts tracks IP allocations across all nodes in a cluster using a shared data store (etcd or Kubernetes Custom Resources). It supports both IPv4 and IPv6 and always assigns the lowest available address in a specified range. It is designed with Kubernetes in mind but is not limited to it.

  2. Use Flatfile configuration to simplify CNI settings

    master

    To avoid repeating common parameters (like etcd settings or log_level) in every CNI configuration, you can use a flatfile. This is especially useful when using Multus CNI.

    Configuration Path Priority: Whereabouts searches for configuration in this order:

    1. The location specified by the configuration_path option in the primary CNI config.
    2. /etc/kubernetes/cni/net.d/whereabouts.d/whereabouts.conf
    3. /etc/cni/net.d/whereabouts.d/whereabouts.conf

    Overriding Behavior: Options in the flatfile are overridden by any options explicitly defined in the primary CNI configuration (e.g., in a NetworkAttachmentDefinition).

    // Example /etc/cni/net.d/whereabouts.d/whereabouts.conf
{
  "datastore": "kubernetes",
  "kubernetes": {
    "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
  },
  "log_file": "/tmp/whereabouts.log",
  "log_level": "debug"
}

  3. How Fast IPAM and NodeSlicePools work

    master

    In standard mode, Whereabouts uses a single cluster-wide lease for all IP allocations. In Fast IPAM mode, the system uses a NodeSlicePool CRD to manage slices of the network range.

    Key Concepts:

    • NodeSlicePool: A Custom Resource Definition (CRD) that manages how the total network range is divided into slices.
    • Per-Node IPPools and Leases: Instead of one global lock, Whereabouts creates an IPPool and a Lease for every unique combination of network name and node name.
    • Network Naming: If multiple NetworkAttachmentDefinitions share the same network name, they will share the same NodeSlicePool, IPPool, and Lease per node. If they have different network names, they will have independent slices.

    Example Naming Logic: If a node named trusted-otter joins a network named test-network, the resulting objects will be named test-network-trusted-otter.

  4. How IPPool storage and namespacing works with named networks

    master

    Whereabouts manages IP assignments using IPPool Custom Resources (CRs).

    Default Behavior (Legacy):

    • IPPool names are derived solely from the canonicalized CIDR range.
    • IPPool CRs are stored in the whereabouts namespace (typically kube-system).
    • This prevents configuring the same CIDR range twice, as they would collide on the same CR name.

    Named Network Behavior:

    • Naming: If network_name is provided in the IPAMConfig, the IPPool name is constructed by prepending the network name to the canonicalized CIDR (e.g., my-net-192-168-2-225-28).
    • Namespacing: Whereabouts attempts to store the IPPool CR in the same namespace as the Multus NetworkAttachmentDefinition. If no such definition exists, it falls back to the whereabouts namespace (e.g., kube-system).
    • Access: Cross-namespace access is permitted if Multus allows cross-namespace access to the NetworkAttachmentDefinition.

  5. How Whereabouts handles backward compatibility for IPAM config

    master

    Whereabouts maintains backward compatibility for SingleStack configurations by performing a union of old configuration fields and the new ipRanges array.

    If you provide the legacy top-level fields (like range or exclude) alongside ipRanges, Whereabouts will automatically convert the legacy fields into a new RangeConfiguration object and append it to the ipRanges list. This ensures that existing SingleStack configurations continue to work without modification.

  6. Run scale tests using `/scripts/scale-test.sh`

    master

    The scale test script requires a running Kubernetes cluster.

    1. Spin up a cluster using: ./hack/e2e-setup-kind-cluster -n 3.
    2. The script uses whereaboutsScaleNAD and scaleTestDeployment YAML files located in the /yamls directory.
    3. To adjust the number of pods used during the scale test, modify the replicas value in the scaleTestDeployment YAML file.
    ./hack/e2e-setup-kind-cluster -n 3

  7. Configure the IP Reconciler schedule

    master

    The frequency of the IP reconciler can be customized using a cron expression.

    For new clusters (via DaemonSet): Set the WHEREABOUTS_RECONCILER_CRON environment variable in your DaemonSet definition.

    For live clusters (via ConfigMap): Update the whereabouts-config ConfigMap in the namespace used by Whereabouts. Use kubectl edit configmap whereabouts-config to adjust the value. The schedule will update shortly after the change.

    # DaemonSet environment variable example
env:
  - name: WHEREABOUTS_RECONCILER_CRON
    value: "30 * * * *"

  8. Install Whereabouts with Helm 3

    master

    You can install Whereabouts using Helm 3 by templating the official OCI chart. Replace <WHEREABOUTS_VERSION> with the desired version:

    helm template whereabouts oci://ghcr.io/k8snetworkplumbingwg/whereabouts-chart --version <WHEREABOUTS_VERSION>

    This installation method includes the necessary CRDs and the Daemonset.

    helm template whereabouts oci://ghcr.io/k8snetworkplumbingwg/whereabouts-chart --version <WHEREABOUTS_VERSION>

  9. Run whereabouts with CNI's `docker-run.sh`

    master

    To run whereabouts using the CNI docker-run.sh script, ensure that the plugins are located in /opt/cni/bin and the configuration files are in /etc/cni/net.d. You must export CNI_PATH and NETCONFPATH so the script can locate the binaries and configurations.

    export CNI_PATH=/opt/cni/bin/
export NETCONFPATH=/etc/cni/net.d
CNI_PATH=$CNI_PATH ./docker-run.sh --rm busybox:latest ifconfig

  10. Install Whereabouts via Daemonset

    master

    To install Whereabouts as a Daemonset in a Kubernetes cluster (requires version 1.16 or later), clone the repository and apply the provided manifests:

    git clone https://github.com/k8snetworkplumbingwg/whereabouts && cd whereabouts
kubectl apply \
    -f doc/crds/daemonset-install.yaml \
    -f doc/crds/whereabouts.cni.cncf.io_ippools.yaml \
    -f doc/crds/whereabouts.cni.cncf.io_overlappingrangeipreservations.yaml \
    -f doc/crds/reconciler-deployment.yaml
    git clone https://github.com/k8snetworkplumbingwg/whereabouts && cd whereabouts
kubectl apply \
    -f doc/crds/daemonset-install.yaml \
    -f doc/crds/whereabouts.cni.cncf.io_ippools.yaml \
    -f doc/crds/whereabouts.cni.cncf.io_overlappingrangeipreservations.yaml \
    -f doc/crds/reconciler-deployment.yaml

  11. Run IP Reconciliation to clean up stranded IP addresses

    master

    Whereabouts includes an IP reconciliation tool designed to run as a Kubernetes CronJob. This utility scans currently allocated IP addresses and compares them against running pods, deallocating any 'stranded' IPs. Stranded IPs typically occur due to node failures (sudden power loss/reboot) or pods that were force-deleted (e.g., kubectl delete pod foo --grace-period=0 --force).

    A reference deployment for this job is available in /docs/ip-reconcilier-job.yaml.

  12. Enable Fast IPAM with Preallocated Node Slices (Experimental)

    master

    To improve performance in large-scale clusters and reduce allocation contention, you can enable Fast IPAM by specifying a node_slice_size.

    Requirements:

    1. You must run the Whereabouts controller (manifest in doc/crds/node-slice-controller.yaml).
    2. The Whereabouts daemonset and the controller must be in the same namespace as your NetworkAttachmentDefinition.

    Configuration: Adding the node_slice_size field triggers Fast IPAM mode. The value determines the size of the CIDR allocated per node.

    apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: whereabouts-fast-ipam
spec:
  config: '{
    "cniVersion": "0.3.0",
    "name": "whereaboutsexample",
    "type": "macvlan",
    "master": "eth0",
    "mode": "bridge",
    "ipam": {
      "type": "whereabouts",
      "range": "192.168.2.0/24",
      "node_slice_size": "/22"
    }
  }'
    apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: whereabouts-fast-ipam
spec:
  config: '{
    "cniVersion": "0.3.0",
    "name": "whereaboutsexample",
    "type": "macvlan",
    "master": "eth0",
    "mode": "bridge",
    "ipam": {
      "type": "whereabouts",
      "range": "192.168.2.0/24",
      "node_slice_size": "/22"
    }
  }'