linux-rdma/perftest

The --data_validation feature provides real-time verification of RDMA data transfers to detect corruption, DMA races, or stale data. It works for ib_write_bw and ib_read_bw using RC connection types.

Supported Backends:

• Host Memory: Uses SIMD-accelerated validators (AVX-512, AVX2, SSE4.2, or ARM NEON).
• GPU Memory: Uses a CUDA kernel plugin (libperftest_kernels.so) if --use_cuda is enabled.

Requirements:

• Connection type must be RC.
• Tests must be bandwidth tests (ib_write_bw, ib_read_bw).
• tx_depth (-t) must be $\ge 32$ and identical on both sides.
• Hardware must support SIMD (AVX2/NEON/SSE42).

Incompatible Options: Cannot be used with -a, --post_list > 1, --run_infinitely, --mr_per_qp, --gpu_touch, --use-null-mr, or --payload_file_path.

Interpreting Results:

• errors: Real corruption detected. The transfer failed.
• races: Temporary mismatch because new data arrived during verification. Normal.
• skips: Transfer rate exceeded verification rate. Normal.
• retries: Inconsistency at the tail of a chunk due to PCIe ordering. Normal.

# Host memory WRITE validation
./ib_write_bw -d mlx5_0 --data_validation -t 128

# GPU memory WRITE validation
./ib_write_bw -d mlx5_0 --use_cuda=0 --data_validation -t 128

The --data_validation flag enables real-time verification that received data matches expected byte patterns. This works with both host memory (using SIMD acceleration like AVX-512/AVX2/SSE4.2/NEON) and GPU memory (via CUDA kernel plugin).

Requirements:

• Must use RC (Reliable Connection) connection type.
• Must use WRITE or READ verbs (e.g., ib_write_bw, ib_read_bw).
• Must be a Bandwidth test (not latency).
• tx_depth (-t) must be the same on both server and client (minimum 32).
• Requires host memory or --use_cuda.

Incompatible with:

• -a, --post_list > 1, --run_infinitely, --mr_per_qp, --gpu_touch, --use-null-mr, --payload_file_path.

Note: Some bandwidth degradation is expected, especially in READ mode. Using 2MB or 1GB huge pages is recommended to improve performance.

# Example usage (conceptual)
./ib_write_bw -s 8388608 -t 32 --data_validation

Perftest automatically attempts to bind benchmark threads and memory to the NUMA node associated with the IB device to minimize cross-node latency.

Affinity Options:

• Automatic (Default): No flags needed. Perftest reads the NIC's NUMA node from sysfs.
• --pin_cores=<cores>:** Pin to specific CPU core(s) (e.g., 5 or 0-3,8). Mutually exclusive with --numa_node.
• --numa_node=<node>:** Bind to a specific NUMA node. Mutually exclusive with --pin_cores.
• --disable_numa:** Disable automatic NUMA binding entirely.

Note: These options are not supported for raw Ethernet benchmarks. Perftest respects external affinity settings (like numactl) and will skip automatic binding if they are detected.

Requirements: --numa_node and automatic detection require libnuma.

# Pin to a specific CPU core
./ib_write_bw --pin_cores=5 -d mlx5_0

# Bind to NUMA node 1
./ib_write_bw --numa_node=1 -d mlx5_0

# Explicitly disable NUMA binding
./ib_write_bw --disable_numa -d mlx5_0

Perftest benchmarks follow a client-server model. To run a test, you must execute the same test binary on both the server and the client machines, ensuring that all mode-specific options are identical.

Server: Run the test binary with your desired options.

./<test_name> <options>

Client: Run the test binary with the same options, followed by the server's IP address.

./<test_name> <options> <server_ip_address>

Important Notes:

• Options specific to a mode must be the same on both server and client.
• You may need to run these applications with sudo if you are not the root user.
• Applications are typically installed in /usr/bin/.
• Errors reported to stderr are sourced from rdma-core.

# Example 1: Bidirectional bandwidth test using Write verb
# Server
./ib_write_bw -s 8388608 -b -D 5 -q 3
# Client
./ib_write_bw -s 8388608 -b -D 5 -q 3 1.1.1.2

# Example 2: Latency test using Read verb
# Server
./ib_read_lat -s 32 -n 5000
# Client
./ib_read_lat -s 32 -n 5000 192.168.0.1

Use these flags when running bandwidth benchmarks (e.g., ib_send_bw, ib_write_bw).

These options are available across most benchmark tools in the collection.