DOCSEARCHbyAlgolia
Submit Docs

NCCL Tests

repository·master·Indexed 23 days ago

https://github.com/nvidia/nccl-tests

A suite of performance benchmarks for measuring the throughput and latency of NVIDIA Collective Communications Library (NCCL) operations across multiple GPUs and nodes. It includes tools for testing various collective operations, support for MPI multi-node execution, and detailed metrics for analyzing algorithm and bus bandwidth.

Tokens
1.7K
Snippets
2
Records
7
Agent score
32%

What's inside nccl-tests

  1. Understand NCCL tests performance metrics

    master

    NCCL tests report performance using three primary metrics:

    1. Time: Measured in milliseconds (ms). This is most useful for small data sizes to measure constant overhead (latency). For large sizes, time becomes linear with size and is less useful than bandwidth.
    2. Algorithm Bandwidth (algbw): Calculated as size / time. It represents how much data is processed per second and is useful for estimating how long a specific operation size will take.
    3. Bus Bandwidth (busbw): A metric designed to reflect how optimally the hardware is being used, independent of the number of ranks. It applies a correction factor to the algorithm bandwidth to account for the specific communication pattern of the collective operation. This value can be directly compared to hardware peak bandwidths (e.g., NVLink, PCI, or network speeds).

  2. How to run multiple operations in parallel using NCCL_TESTS_SPLIT

    master

    You can partition GPUs into smaller sets to execute the same operation in parallel by using the NCCL_TESTS_SPLIT environment variable. NCCL computes a "color" for each rank based on this variable; ranks with the same color form a group.

    Syntax: <operation><value>

    • Operations: AND, OR, MOD, DIV (or symbols &, |, %, /).
    • Values: decimal, hexadecimal (0x), or binary (0b).
    • NCCL_TESTS_SPLIT_MASK="<value>" is equivalent to NCCL_TESTS_SPLIT="&<value>".

    Examples (assuming 8 GPUs):

    • NCCL_TESTS_SPLIT="AND 0x7" or MOD 8: 8 parallel operations, 1 GPU per node (inter-node communication).
    • NCCL_TESTS_SPLIT="OR 0x7" or DIV 8: 1 operation per node (intra-node communication).
    • NCCL_TESTS_SPLIT="AND 0x1" or MOD 2: 2 operations, each using every other rank.

    Note: Reported bandwidth is per group. Multiply by the number of groups to get total bandwidth.

  3. Build NCCL Tests

    master

    To build the tests, use make or make -j.

    If CUDA or NCCL are not in their default locations (/usr/local/cuda and /usr respectively), you must specify CUDA_HOME and NCCL_HOME.

    To enable MPI support for multi-node testing, set MPI=1 and provide the MPI_HOME path. You can also use NAME_SUFFIX to append a string to the generated binary names (e.g., _mpi).

  4. Run NCCL Tests with MPI across multiple nodes

    master

    To run tests across multiple nodes using MPI, you must have compiled the tests with MPI=1. Use mpirun to launch the processes. The total number of ranks (CUDA devices) is calculated as (number of processes) * (number of threads) * (number of GPUs per thread).

    Example: Run 64 MPI processes on nodes with 8 GPUs each (total 64 GPUs across 8 nodes), scanning from 8 Bytes to 8GiB, doubling between each test.

    mpirun -np 64 -N 8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

  5. Run NCCL Tests on a single node

    master

    To run a test on a single node with a specific number of GPUs, use the -g flag to specify the number of GPUs. You can also define the size range for the test using -b (minimum bytes), -e (maximum bytes), and -f (step factor).

    Example: Run on a single node with 8 GPUs, scanning from 8 Bytes to 128MiB, doubling the size between each test.

    ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8

  6. Calculate Bus Bandwidth for collective operations

    master

    To compare test results against hardware peak bandwidth, use the Bus Bandwidth metric. The bus bandwidth is derived from the algorithm bandwidth (algbw) using a correction factor that depends on the number of ranks (n) and the specific collective operation being performed.

    OperationBus Bandwidth Formula (Correction Factor)
    AllReducealgbw * (2 * (n - 1) / n)
    ReduceScatteralgbw * (n - 1) / n
    AllGatheralgbw * (n - 1) / n
    AlltoAllalgbw * (n - 1) / n
    Broadcastalgbw * 1
    Reducealgbw * 1

    Note: For ReduceScatter and AllGather, the total size S used in calculations is the total size in bytes across all ranks (e.g., recvcount * sizeof(datatype) * n).

  7. Reference: NCCL Test CLI Arguments

    master

    All NCCL tests support the following command-line arguments:

    Number of GPUs

    • -t,--nthreads <num threads>: number of threads per process. Default: 1.
    • -g,--ngpus <GPUs per thread>: number of gpus per thread. Default: 1.

    Sizes to scan

    • -b,--minbytes <min size in bytes>: minimum size to start with. Default: 32M.
    • -e,--maxbytes <max size in bytes>: maximum size to end at. Default: 32M.
    • -i,--stepbytes <increment size>: fixed increment between sizes. Default: 1M.
    • -f,--stepfactor <increment factor>: multiplication factor between sizes. Default: disabled.

    NCCL operations arguments

    • -o,--op <sum/prod/min/max/avg/all>: reduction operation (for Allreduce, Reduce, ReduceScatter). Default: Sum.
    • -d,--datatype <nccltype/all>: datatype to use. Default: Float.
    • -r,--root <root/all>: root rank (for broadcast or reduce). Default: 0.

    Performance

    • -n,--iters <iteration count>: number of iterations. Default: 20.
    • -w,--warmup_iters <warmup iteration count>: warmup iterations (not timed). Default: 1.
    • -m,--agg_iters <aggregation count>: operations to aggregate per iteration. Default: 1.
    • -N,--run_cycles <cycle count>: run & print each cycle. Default: 1; 0=infinite.
    • -a,--average <0/1/2/3>: report performance as average across ranks (MPI=1 only). <0=Rank0, 1=Avg, 2=Min, 3=Max>. Default: 1.
    • -I,--per_iter_timing <0/1>: collect per-iteration CUDA event timings. Default: 0.
    • -K,--per_iter_skip <count>: exclude leading samples from -I summary. Default: 0.

    Test operation

    • -p,--parallel_init <0/1>: use threads to initialize NCCL in parallel. Default: 0.
    • -c,--check <check iteration count>: perform correctness checks on each iteration. Default: 1.
    • -z,--blocking <0/1/2/3>: collective blocking mode. 0: non-blocking; 1: wait/barrier after each -m; 2: wait after each -m, no barrier; 3: wait/barrier after each -n. Default: 0.
    • -G,--cudagraph <num graph launches>: capture and replay iterations as CUDA graphs. Default: 0.
    • -C,--report_cputime <0/1>: report CPU time instead of latency. Default: 0.
    • -R,--local_register <0/1/2>: enable local (1) or symmetric (2) buffer registration. Default: 0.
    • -D,--device_implementation <implementation number>: use custom device API. Default: 0.
    • -V,--device_cta_count <number>: number of CTAs for device API. Default: 16.
    • -S,--report_timestamps <0/1>: add timestamps to reports. Default: 0.
    • -J,--output_file <file>: write JSON output to filepath.
    • -T,--timeout <time in seconds>: timeout test after specified seconds. Default: disabled.
    • -M,--memory <0/1>: enable memory usage report. Default: 0.
    • -u,--unalign <index of first element>: misalign source/destination buffers. Default: 0.