DOCSEARCHbyAlgolia
Submit Docs

nvkind

repository·main·Indexed 42 days ago

https://github.com/nvidia/nvkind

A tool to simplify the creation and management of Kubernetes 'kind' clusters with direct access to host GPUs. It automates the injection of GPU support and the isolation of specific GPUs to worker nodes, supporting various distribution patterns and integration with the NVIDIA GPU Operator and k8s-device-plugin.

Tokens
4.6K
Snippets
18
Records
23
Agent score
53%

What's inside nvkind

  1. Install k8s-device-plugin on an nvkind cluster

    main

    Once your GPU-enabled cluster is running, you must install a device plugin (like k8s-device-plugin or the NVIDIA GPU Operator) to make the GPUs allocatable by Kubernetes pods.

    To install the k8s-device-plugin using Helm:

    1. Add the nvdp Helm repository.
    2. Set the KIND_CLUSTER_NAME environment variable to target your cluster.
    3. Run helm upgrade -i targeting the kind-<cluster-name> context.
    # Add repo
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update

# Target cluster
export KIND_CLUSTER_NAME=evenly-distributed-2-by-4

# Install plugin
helm upgrade -i \
    --kube-context=kind-${KIND_CLUSTER_NAME} \
    --namespace nvidia \
    --create-namespace \
    nvidia-device-plugin nvdp/nvidia-device-plugin

  2. Delete all kind clusters

    main

    To clean up your environment, you can use a shell loop to iterate through all existing kind clusters and delete them one by one.

    for cluster in $(kind get clusters); do kind delete cluster --name=${cluster}; done

  3. Verify NVIDIA GPU Operator installation

    main

    After running the Helm installation, use the following commands to verify that the operator pods are coming online and that the cluster policy is active. Use the --context=kind-${KIND_CLUSTER_NAME} flag to ensure you are targeting the correct nvkind cluster.

    To verify a workload can actually run on the GPUs, apply a test pod (e.g., from examples/gpu-operator/gpu-pod.yml) and check its logs.

    # Verify cluster policy and pods
kubectl --context=kind-${KIND_CLUSTER_NAME} get clusterpolicy -A
kubectl --context=kind-${KIND_CLUSTER_NAME} get pods -n gpu-operator
kubectl --context=kind-${KIND_CLUSTER_NAME} get ds -n gpu-operator

# Verify workload deployment
kubectl --context=kind-${KIND_CLUSTER_NAME} apply -f examples/gpu-operator/gpu-pod.yml
kubectl --context=kind-${KIND_CLUSTER_NAME} logs gpu-pod

  4. Install nvkind

    main

    You can install nvkind using go install or by building it inside a Go Docker container if you do not have Go installed locally.

    # Using go install
go install github.com/NVIDIA/nvkind/cmd/nvkind@latest

# Using Docker to build
docker run --rm -v $PWD/bin/:/go/bin/ golang:1.23 go install github.com/NVIDIA/nvkind/cmd/nvkind@latest

  5. Check the health of an existing demo cluster

    main

    To verify that an existing nvkind demo cluster is healthy, use the status mode. This command validates:

    • Cluster existence and node readiness.
    • NVIDIA runtime classes.
    • GPU Operator pod and daemonset health.
    • ClusterPolicy readiness.
    • End-to-end execution of the gpu-pod workload.

    A successful status check will output: [demo] Status: kind demo is healthy.

    ./gpu-operator-kind-demo.sh status

  6. Run the GPU Operator Kind Demo

    main

    The GPU Operator Kind Demo uses a shell script to automate the creation of an nvkind cluster, the installation of the NVIDIA GPU Operator (with driver.enabled=false), and the verification of GPU access via a sample workload.

    To run the full automated sequence from the examples/gpu-operator/ directory, use the all mode.

    ./gpu-operator-kind-demo.sh all

  7. Configure host NVIDIA drivers

    main

    The demo host must have an NVIDIA GPU visible and a working host NVIDIA driver. You can verify the driver installation using nvidia-smi -L.

    If nvidia-smi is missing, install the drivers on Ubuntu/Debian using the following commands:

    sudo apt-get update
sudo apt-get install -y ubuntu-drivers-common
sudo ubuntu-drivers devices
sudo ubuntu-drivers autoinstall
sudo reboot

  8. Create GPU-enabled kind clusters with nvkind

    main

    After building nvkind with make, you can create various cluster configurations using the cluster create command. nvkind supports templated configurations that allow for complex GPU distributions across nodes.

    Common Cluster Patterns:

    • Default: One worker node with access to all GPUs on the machine.
    • One node per GPU: Uses a config template to create one worker for every available GPU.
    • Evenly distributed: Distributes a specific number of GPUs across a set number of workers.
    • Explicit configuration: Assigns specific GPU device IDs to specific workers.
    # Build the tool
make

# Default: One node with all GPUs
./nvkind cluster create

# One node per GPU
./nvkind cluster create --config-template=examples/one-worker-per-gpu.yaml

# Four nodes with two GPUs each (assuming 8 GPUs total)
./nvkind cluster create \
  --name=evenly-distributed-2-by-4 \
  --config-template=examples/equally-distributed-gpus.yaml \
  --config-values=-\ <<EOF
numWorkers: 4
EOF

# Mixed configuration (Explicitly assigning devices)
./nvkind cluster create \
  --name=explicit-gpus \
  --config-template=examples/explicit-gpus-per-worker.yaml \
  --config-values=-\ <<EOF
workers:
- devices: 0
- devices: [1, 2, 3]
EOF

  9. Configure nvidia-container-toolkit for kind

    main

    Before creating clusters, you must configure the nvidia-container-toolkit to allow kind to inject GPU support into worker nodes. This involves setting the nvidia runtime as the default for Docker and enabling the accept-nvidia-visible-devices-as-volume-mounts feature flag.

    Run the following commands:

    1. Configure the runtime and enable CDI.
    2. Enable the specific feature flag for volume mounts.
    3. Restart the Docker service.
    sudo nvidia-ctk runtime configure --runtime=docker --set-as-default --cdi.enabled
sudo nvidia-ctk config --set accept-nvidia-visible-devices-as-volume-mounts=true --in-place
systemctl restart docker

  10. Install the NVIDIA GPU Operator in an nvkind cluster

    main

    To install the NVIDIA GPU Operator in a Docker-based nvkind environment, you must first add the NVIDIA Helm repository and then configure the cluster namespace with the appropriate pod security labels.

    Note that for nvkind environments, specific Helm flags are used to disable the driver and toolkit installation (as these are typically handled by the underlying host/container setup) and to enable CDI (Container Device Interface).

    Prerequisites:

    • A kind cluster must already be created using nvkind.
    • helm and kubectl must be installed.
    • The KIND_CLUSTER_NAME environment variable must be set to your target cluster name.
    # 1. Add the helm repo
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# 2. Set the target cluster
export KIND_CLUSTER_NAME=your-cluster-name

# 3. Create namespace and apply pod security label
kubectl --context=kind-${KIND_CLUSTER_NAME} create namespace gpu-operator --dry-run=client -o yaml | kubectl apply -f -
kubectl --context=kind-${KIND_CLUSTER_NAME} label namespace gpu-operator pod-security.kubernetes.io/enforce=privileged --overwrite

# 4. Install the gpu-operator
helm upgrade -i \
    --kube-context=kind-${KIND_CLUSTER_NAME} \
    --namespace gpu-operator \
    --create-namespace \
    --wait \
    nvidia-gpu-operator nvidia/gpu-operator \
    --set cdi.enabled=true \
    --set driver.enabled=false \
    --set toolkit.enabled=false \
    --set operator.runtimeClass=nvidia

  11. Prerequisites for running nvkind

    main

    To use nvkind, you must have the following tools installed on your host:

    • go
    • make
    • docker
    • kind
    • kubectl
    • helm

    Additionally, your host must have a working NVIDIA driver and the nvidia-container-toolkit configured for use with docker.

    You can verify your setup by running nvidia-smi -L on the host and ensuring that a Docker container can also access the GPUs using the --runtime=nvidia flag.

    # Verify host driver
$ nvidia-smi -L

# Verify docker runtime
$ docker run --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all ubuntu:20.04 nvidia-smi -L

  12. Troubleshoot GPU Operator Kind Demo issues

    main

    Docker Group Membership

    If the script adds your user to the docker group and exits, refresh your shell membership and rerun:

    newgrp docker
./gpu-operator-kind-demo.sh all

    Host Driver Validation

    Verify host and container-level driver access with:

    nvidia-smi -L
docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all ubuntu:22.04 nvidia-smi -L

    Inotify and File Limit Errors

    If cluster creation fails with inotify_init: too many open files, increase host limits:

    sudo sysctl -w fs.inotify.max_user_instances=1024
sudo sysctl -w fs.inotify.max_user_watches=524288
sudo sysctl -w fs.file-max=2097152
ulimit -n 1048576

    To make Docker's file descriptor limits persistent, create a systemd override:

    sudo mkdir -p /etc/systemd/system/docker.service.d
sudo tee /etc/systemd/system/docker.service.d/limits.conf <<'EOF2'
[Service]
LimitNOFILE=1048576
EOF2
sudo systemctl daemon-reload
sudo systemctl restart docker

    Inspecting Worker Nodes

    If the worker fails to join the cluster, inspect the logs directly via Docker:

    docker exec nvkind-gpu-operator-demo-worker systemctl status kubelet --no-pager -l
docker exec nvkind-gpu-operator-demo-worker journalctl -u kubelet --no-pager -n 200
docker exec nvkind-gpu-operator-demo-worker journalctl -u containerd --no-pager -n 200