nvbandwidth

To provide realistic memory latency measurements that prevent prefetching optimizations, nvbandwidth uses a pointer chasing methodology rather than sequential access:

1. Setup: Memory is organized as a linked list where each node contains a pointer to the next.
2. Pattern: The chain follows a strided pattern: Node[i] -> Node[(i + stride) % total_nodes].
3. Execution: A kernel follows this chain for a specified number of accesses.
4. Measurement: Total time is divided by the number of accesses to find latency per access.

Key characteristics of this measurement:

• TLB costs are excluded: The same buffer is reused to keep TLB entries warm, ensuring address translation overhead does not inflate the reported latency.
• Data cache hits are prevented: The random pointer chasing pattern is designed to bypass data cache benefits.

nvbandwidth supports two methods for performing data copies:

1. Copy Engine (CE) copies: These use standard memcpy APIs.
2. Streaming Multiprocessor (SM) copies: These use custom CUDA kernels.

Note on SM copy sizing: To ensure uniform reporting, SM copies truncate the requested copy size to fit the target device's architecture. The actual byte size used is calculated as: (threadsPerBlock * deviceSMCount) * floor(copySize / (threadsPerBlock * deviceSMCount)) where threadsPerBlock is fixed at 512.

To use the --pair option, you must provide the exact GPU UUIDs in the format GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

Single-node

Use nvidia-smi -L to list all GPUs and their UUIDs:

nvidia-smi -L

Multi-node

Use mpirun with your hostfile to query all nodes. To ensure you see all GPUs on every node, match the number of processes (-n) to the total number of GPUs in the cluster and use --map-by ppr:<gpus_per_node>:node:

# Example: 4 GPUs/node, 2 nodes = 8 total processes
mpirun --hostfile /etc/nvidia-imex/nodes_config.cfg \
    --map-by ppr:4:node -n 8 \
    nvidia-smi -L

The --pair option restricts device-to-device tests to specific GPUs, which is useful for isolating a single link.

Single-node Pairwise

Pass both UUIDs directly to the command:

./nvbandwidth -t <testcase> --pair <UUID-0> <UUID-1>

Multinode Pairwise

Run with exactly 2 MPI ranks (one per GPU). Each rank must be assigned the UUID of the GPU it will use:

mpirun -n 2 ./nvbandwidth -t <multinode_testcase> --pair <UUID-0> <UUID-1>

Supported testcase families for --pair:

• device_to_device_memcpy_*
• device_to_device_bidirectional_memcpy_*
• device_to_device_latency_sm / device_to_device_latency_tma
• multinode_device_to_device_memcpy_*
• multinode_device_to_device_bidirectional_memcpy_*

./nvbandwidth -t device_to_device_memcpy_read_sm --pair <GPU-UUID-0> <GPU-UUID-1>

If ldd nvbandwidth reports libmpi_cxx.so.X => not found, you need to install the OpenMPI development packages.

Ubuntu/Debian:

sudo apt-get install openmpi-bin libopenmpi-dev

Red Hat/CentOS/Fedora:

# RHEL/CentOS 7/8
sudo yum install openmpi-devel

# Fedora
sudo dnf install openmpi-devel

# Ubuntu/Debian
sudo apt-get install openmpi-bin libopenmpi-dev

# RHEL/CentOS
sudo yum install openmpi-devel

# Fedora
sudo dnf install openmpi-devel

Requirements

• Linux (primary platform).
• CUDA Toolkit: version 11.x or above. (Multinode requires 12.3+ and 550+ driver).
• C++17 compiler: GCC 7.x+ (Linux) or MSVC 2019+ (Windows).
• CMake: version 3.20 or above (3.24+ recommended).
• MPI: Required for multinode builds.

On Linux, ensure nvcc is in your $PATH.

Build Single-node

cmake .
make

On Ubuntu/Debian, use the provided utility script to automate installation and building:

sudo ./debian_install.sh

Build Multinode

To enable multinode support, pass the MULTINODE=1 flag to CMake:

cmake -DMULTINODE=1 .
make

NVBandwidth requirements depend on whether you are performing single-node or multi-node testing:

Multi-node requirements:

• CUDA Toolkit 12.3 & driver and above
• IMEX setup
• Proper MPI configuration

Single-node requirements:

• CUDA Toolkit 11.x or above

To enable multi-node support during the build process, ensure you are using CUDA 12.3+ and pass the MULTINODE=1 flag to CMake.

# Ensure CUDA 12.3+ is in use
$ nvcc --version

# Build with multi-node support
$ cmake -DMULTINODE=1

This error occurs when the NVIDIA driver is missing, GPUs are not detected, or there is a CUDA driver/toolkit mismatch.

Verification Steps:

1. Check GPU detection: nvidia-smi
2. Verify driver installation:
   • Check version: cat /proc/driver/nvidia/version
   • Check loaded modules: lsmod | grep nvidia
   • Check package: dpkg -l | grep nvidia-driver
3. Check CUDA setup:
   • Driver version: nvidia-smi | grep "CUDA Version"
   • Runtime version: nvcc --version

Solution: Install or reinstall the NVIDIA driver and CUDA toolkit. For detailed diagnostics, run nvidia-bug-report.sh.

$ nvidia-smi
$ cat /proc/driver/nvidia/version
$ lsmod | grep nvidia
$ dpkg -l | grep nvidia-driver
$ nvidia-smi | grep "CUDA Version"
$ nvcc --version
$ nvidia-bug-report.sh

If you encounter nvcc fatal : Unsupported gpu architecture, it is likely because you are targeting an older architecture with a newer CUDA version.

Solution:

• For older architectures (Volta, Maxwell, Pascal), use CUDA < 13.0.
• For Hopper architecture, specify the architecture explicitly in CMake: cmake -DCMAKE_CUDA_ARCHITECTURES=sm_90