Multus-CNI Documentation

By default, pod traffic routes over eth0 (the cluster-wide default network). To redirect the default route to a specific network attachment, use the default-route key within the JSON formatted k8s.v1.cni.cncf.io/networks pod annotation.

Warning: Changing the default route may impact the pod's ability to communicate over the cluster-wide default network.

To implement this, you must define a NetworkAttachmentDefinition and then reference it in your Pod's annotations with the desired gateway address.

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.2.0/24",
        "rangeStart": "192.168.2.200",
        "rangeEnd": "192.168.2.216",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.2.1"
      }
    }'
EOF

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: '[{
      "name": "macvlan-conf",
      "default-route": ["192.168.2.1"]
    }]'
spec:
  containers:
  - name: samplepod
    command: ["/bin/bash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: dougbtv/centos-network
EOF

To use Multus with a single primary CNI plugin (the default network), use the clusterNetwork option. This option is mutually exclusive with delegates.

Configuration Requirements:

• You must set clusterNetwork if you are not using delegates.
• You can optionally set defaultNetworks to specify additional NetworkAttachmentDefinition names that should be automatically attached to every pod.

Resolving clusterNetwork and defaultNetworks values: Multus resolves the strings provided in these fields in the following order:

1. A NetworkAttachmentDefinition custom resource name in the namespace defined by multusNamespace.
2. The name key within a CNI JSON configuration file located in the confDir.
3. An absolute file path to a CNI configuration file.
4. A path to a directory containing CNI JSON files (the alphabetically first file is used).

If no match is found, Multus will raise an error.

{
    "cniVersion": "0.3.1",
    "name": "node-cni-network",
    "type": "multus",
    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
    "confDir": "/etc/cni/multus/net.d",
    "cniDir": "/var/lib/cni/multus",
    "binDir": "/opt/cni/bin",
    "logFile": "/var/log/multus.log",
    "logLevel": "debug",
    "logOptions": {
        "maxAge": 5,
        "maxSize": 100,
        "maxBackups": 5,
        "compress": true
    },
    "capabilities": {
        "portMappings": true
    },    
    "namespaceIsolation": false,
    "clusterNetwork": "/etc/cni/net.d/99-flannel.conf",
    "defaultNetworks": ["sidecarCRD", "exampleNetwork"],
    "systemNamespaces": ["kube-system", "admin"],
    "multusNamespace": "kube-system",
    "auxiliaryCNIChainName": "cni-chain-config",
    "retryDeleteOnError": false
}

Multus needs a kubeconfig file to interact with the Kubernetes API. This must be generated on the master node and then copied to all other Kubernetes nodes. It is recommended to set permissions to 600 for security.

1. Generate on Master: Run the script provided in the documentation to extract the ServiceAccount token and CA from kube-system and create /etc/cni/net.d/multus.d/multus.kubeconfig.
2. Distribute: Use scp to copy the file to all nodes.
3. Secure: Run chmod 600 /etc/cni/net.d/multus.d/multus.kubeconfig on each node.

# Example of generating the kubeconfig on master
mkdir -p /etc/cni/net.d/multus.d
SERVICEACCOUNT_CA=$(kubectl get secrets -n=kube-system -o json | jq -r '.items[]|select(.metadata.annotations."kubernetes.io/service-account.name"=="multus")| .data."ca.crt"')
SERVICEACCOUNT_TOKEN=$(kubectl get secrets -n=kube-system -o json | jq -r '.items[]|select(.metadata.annotations."kubernetes.io/service-account.name"=="multus")| .data.token' | base64 -d )
KUBERNETES_SERVICE_PROTOCOL=$(kubectl get all -o json | jq -r .items[0].spec.ports[0].name)
KUBERNETES_SERVICE_HOST=$(kubectl get all -o json | jq -r .items[0].spec.clusterIP)
KUBERNETES_SERVICE_PORT=$(kubectl get all -o json | jq -r .items[0].spec.ports[0].port)
cat > /etc/cni/net.d/multus.d/multus.kubeconfig <<EOF
# Kubeconfig file for Multus CNI plugin.
apiVersion: v1
kind: Config
clusters:
- name: local
  cluster:
    server: ${KUBERNETES_SERVICE_PROTOCOL:-https}://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}
    certificate-authority-data: ${SERVICEACCOUNT_CA}
users:
- name: multus
  user:
    token: "${SERVICEACCOUNT_TOKEN}"
contexts:
- name: multus-context
  context:
    cluster: local
    user: multus
current-context: multus-context
EOF

To use secondary networks in a Pod, use the k8s.v1.cni.cncf.io/networks annotation. Multus supports several formats:

1. Text Annotation (Comma-separated list)

Use this for simple attachment to multiple networks or specifying a specific interface name using @.

• Multiple networks: macvlan-conf-1,macvlan-conf-2
• With interface name: macvlan-conf-1@macvlan1
• Cross-namespace: testns1/macvlan-conf-3

2. JSON Annotation

Use this for more complex requirements, such as specifying namespaces or interfaces explicitly.

• Multiple networks: '[{"name": "macvlan-conf-1"}, {"name": "macvlan-conf-2"}]'
• With namespace: '[{"name": "macvlan-conf-1", "namespace": "testns1"}]'
• With interface: '[{"name": "macvlan-conf-1", "interface": "macvlan1"}]'

# Example: Launching a pod with multiple networks via text annotation
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: pod-case-01
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf-1, macvlan-conf-2
spec:
  containers:
  - name: pod-case-01
    image: docker.io/centos/tools:latest
    command:
    - /sbin/init
EOF