NVIDIA Collective Communications Library (NCCL)

NCCL (NVIDIA Collective Communications Library) provides topology-aware inter-GPU communication primitives designed to accelerate communication in multi-GPU and multi-node environments. It implements both collective communication and point-to-point primitives.

Supported collective communication primitives include:

• AllReduce
• Broadcast
• Reduce
• AllGather
• ReduceScatter
• AlltoAll
• Gather
• Scatter

Additionally, NCCL supports point-to-point send/receive communication, which can be used to implement scatter, gather, or all-to-all operations.

The NCCL Device API enables communication to be initiated and performed directly from device (GPU) code. It is categorized into four primary functional areas:

• Host-Side Setup: Functions for creating and configuring device communicators, querying device properties, and managing host-accessible device pointers.
• Memory and LSA: Provides Load/Store Accessible (LSA) memory management, barriers, pointer accessors, and multimem capabilities.
• GIN (GPU-Initiated Networking): Supports one-sided transfers, signals, counters, and network barriers.
• Reduce, Broadcast, and Fused Building Blocks: Provides fundamental building blocks for computation-fused kernels, including reduce, copy (broadcast), and reduce-then-copy. These are used to implement collective algorithms like AllReduce, AllGather, and ReduceScatter.

The nccl.core.Communicator class is the primary interface for performing collective and point-to-point communication in nccl4py. Its functionality is categorized into the following operational areas:

• Class Identity: Constructor and properties for identity and device-API capability.
• Lifecycle Management: Creating, splitting, growing, and tearing down communicators.
• Collective Operations: Methods for collective communication such as allreduce, broadcast, and gather.
• Point-to-Point (P2P) & Signaling: Methods for send, recv, signal, wait_signal, and put_signal.
• Buffer & Window Registration: Registration for zero-copy and Remote Memory Access (RMA).
• Device Setup: Host-side bootstrapping of device communicators.
• Status & Cleanup: Error queries and resource management.

Starting with version 2.28, NCCL provides a device-side communication API for use directly within user CUDA kernels. The API is organized into several specialized modules:

• LSA (Load/Store Accessible): Enables communication between devices via memory load/store operations using CUDA P2P (e.g., NVLink or P2P-capable PCIe).
• Multimem: Uses hardware multicast via NVLink SHARP (available on certain Hopper-generation datacenter GPUs).
• GIN (GPU-Initiated Networking): Enables communication over the network (available since version 2.28.7).
• Reduce, Broadcast, and Fused Building Blocks: Optimized building blocks for computation-fused kernels (e.g., reduce, copy, and reduce-then-copy).

NCCL (pronounced "Nickel") is a library of optimized communication primitives for GPUs. It implements standard routines including all-reduce, all-gather, reduce, broadcast, and reduce-scatter, as well as arbitrary send/receive patterns.

It is optimized for high bandwidth across:

• PCIe
• NVLink
• NVSwitch
• Networking via InfiniBand Verbs or TCP/IP sockets.

NCCL supports an arbitrary number of GPUs in single-node or multi-node configurations and is compatible with single-process or multi-process (e.g., MPI) applications.

The NCCL Checkpoint Shim enables collective checkpointing of all processes sharing NCCL communicators across multiple hosts. It intercepts NCCL calls to capture resource initialization, allowing applications to resume execution without reconfiguring communicators.

To use the shim, you must:

1. Launch your application with LD_PRELOAD pointing to libnccl-checkpoint-shim.so.
2. Set NCCL_CHECKPOINT_KVS_PATH to a file containing the <address>:<port> of a Redis server used for process rendezvous during restore.
3. Call the prepare/restore APIs at the appropriate lifecycle stages in your code.

NCCL uses a symmetric memory design to achieve barrier-free, low-latency asynchronous All-to-All (A2A) communication. It relies on 16-byte atomic writes (uint4) and a dual-epoch validation scheme to ensure data integrity without explicit synchronization.

Key characteristics:

• Atomic Writes: Data is written in 16 B chunks to prevent tearing.
• Dual-Epoch Validation: Uses epoch tags to validate that data is fresh and not stale.
• Performance Trade-off: This design provides extremely low latency for small messages but incurs a 2× bandwidth cost.

NCCL core events are organized in a hierarchical structure. This allows profilers to understand the relationship between high-level API calls and low-level execution details.

Hierarchy Structure:

• Group API event
  • Collective API event
    • Collective event
      • ProxyOp event
        • ProxyStep event
          • NetPlugin event
      • KernelCh event
  • Point-to-point API event
    • Point-to-point event
      • ProxyOp event
        • ProxyStep event
          • NetPlugin event
      • KernelCh event
  • Kernel Launch event
• ProxyCtrl event

NCCL GIN (GPU-Initiated Networking) and Symmetric Memory are used to optimize communication in distributed systems, particularly for fusing computation and communication.

• Symmetric Memory: Provides a unified abstraction that maps peer-memory access across a distributed system into a single, flat virtual-address space. This simplifies how parallel-computing kernels manage addresses.
• GIN (GPU-Initiated Networking): Required for RDMA scale-out networks. While NVLink or PCIe peer-to-peer can use plain memory semantics, RDMA requires GIN to issue message-semantic network transfers directly from the device side.

GIN uses a hierarchy of connections and contexts to increase network parallelism:

• Connection: A (src, dst) network device pair. A collComm manages connections to all peers. The number of connections is typically low (max ~4), usually corresponding to the number of network devices a rank has affinity to.
• Context: A sub-resource of a connection. Multiple contexts allow parallel communication channels that are independent in terms of ordering. GIN ordering guarantees apply per-context (e.g., a signal completion guarantees the completion of all previous puts on that same context).