DOCSEARCHbyAlgolia
Submit Docs

Gloo Collective Communications Library

repository·main·Indexed 23 days ago

https://github.com/pytorch/gloo

A collective communications library optimized for multi-machine machine learning training workloads. Gloo provides primitives such as barrier, broadcast, and allreduce, supporting both system memory and NVIDIA GPU memory buffers. It features CUDA-aware algorithms, GPUDirect acceleration for InfiniBand/RoCE, and a stateless design via gloo::Context to support high levels of parallelism. The library is built for Linux using CMake and supports rendezvous coordination through Redis or MPI.

Tokens
4.5K
Snippets
8
Records
28
Agent score
81%

What's inside Gloo

  1. What is Gloo and its core capabilities

    main

    Gloo is a collective communications library designed for multi-machine training in machine learning applications. It provides several collective algorithms, including:

    • barrier
    • broadcast
    • allreduce

    Data transport is abstracted to support IP (TCP) at all times, or InfiniBand/RoCE when available. When using InfiniBand, it can leverage GPUDirect to accelerate cross-machine GPU-to-GPU memory transfers. Gloo supports both system memory buffers and NVIDIA GPU memory buffers; for GPU buffers, the algorithm implementations handle memory transfers without requiring manual host-to-device copies.

  2. Overview of Gloo collective algorithms

    main

    Gloo provides collective algorithms designed to run in parallel across multiple processes or machines. To use Gloo, your application must follow a specific lifecycle:

    1. Rendezvous: Before executing algorithms, participating machines must find each other. This process is called rendezvous and is the prerequisite for all Gloo operations.
    2. Connectivity: After rendezvous, machines establish communication channels. The topology (e.g., a full mesh where every machine connects to every other machine, or a subset like a ring) depends on the specific algorithm being used.
    3. Context Management: Each participating process tracks its own rank (a 0-based index) and the total number of processes. This state, along with persistent communication channels, is managed via a gloo::Context object.

    Gloo is designed to be stateless at the global or thread-local level, allowing you to instantiate multiple gloo::Context objects to support high levels of parallelism within your application.

  3. Use NVIDIA GPU support in Gloo

    main
    Gloo provides collective algorithm implementations that operate directly on NVIDIA GPU buffers. These implementations are designed to overlap host and GPU operations to reduce latency. To use these features, you must have CUDA 7 or newer installed.

  4. What is Rendezvous in Gloo

    main
    Rendezvous is a process that must occur exactly once per Gloo context. It allows participating Gloo processes to exchange the necessary details to set up communication channels. For example, when using the TCP transport, processes use rendezvous to exchange IP addresses and port numbers for their listening sockets.

  5. Understand Barrier synchronization algorithms

    main

    A Barrier acts as a synchronization point between processes. Gloo provides two main implementations:

    barrier_all_to_all

    • Communication steps: 1
    • Bytes on the wire: $P$
    • Mechanism: Every process sends a notification to every other process and waits for a notification from every other process.

    barrier_all_to_one

    • Communication steps: 2
    • Bytes on the wire: 1 for non-root, $P$ for root
    • Mechanism:
      • Non-root processes: Send a notification to the root and wait for a notification from the root.
      • Root process: Waits for notifications from $P-1$ processes, then sends notifications to $P-1$ processes.

  6. Compare Allreduce algorithm implementations

    main

    Gloo provides several Allreduce implementations optimized for different network characteristics. Use the following metrics to choose an algorithm:

    • Communication steps: The number of communication steps. Fewer steps are better for high-latency transports.
    • Bytes on the wire: Total bytes transmitted per process. Lower values are better for bandwidth-constrained networks.

    Available Allreduce Algorithms

    AlgorithmCommunication StepsBytes on the Wire
    allreduce_ring$P-1$$P \times S$
    allreduce_ring_chunked$4 \times P$$2 \times S$
    allreduce_halving_doubling$2 \times \lg(P)$$2 \times S$
    allreducube_bcube$2 \times \log_B(P)$$2 \times \sum_{s=0}^{\log_B(P)-1} S/B^s$

    Note: $P$ is the number of processes, $S$ is the buffer size, and $B$ is the base (max peers per step).

  7. Understand Allreduce algorithms and semantics

    main

    An Allreduce operation computes a user-specified reduction (e.g., sum) of $N$ arrays per process across $P$ processes. The computation is performed in place, meaning all input arrays will contain the resulting reduction once the algorithm completes.

    Every Allreduce implementation follows three phases:

    1. Local reduction: Reducing the $N$ buffers locally on each process.
    2. Allreduce between processes: Communicating data across the $P$ processes.
    3. Broadcast: Distributing the final result back to the $N$ buffers.

  8. Understand Broadcast algorithms

    main

    A Broadcast operation sends the contents of a buffer from one (root) process to the other $P-1$ processes.

    broadcast_one_to_all

    • Communication steps: 1
    • Bytes on the wire: $P \times S$
    • Mechanism:
      • Root process: Sends the buffer to all $P-1$ processes.
      • Non-root processes: Receive the buffer from the root.

  9. Understand the difference between recoverable errors and assertions

    main

    Gloo categorizes errors into two distinct groups:

    1. Recoverable errors: These are thrown as exceptions (extending ::gloo::Exception) for situations where the caller might be able to recover or reconfigure the system (e.g., IO errors).
    2. Assertions: These are used for unexpected errors or logical invariants that should not be handled by the caller. These are triggered via GLOO_ENFORCE macros.

  10. How to manage state and parallelism with gloo::Context

    main

    In Gloo, all state required for collective operations—including the process rank, the total number of participating processes, and the persistent communication channels—is encapsulated within a gloo::Context instance.

    Because Gloo does not rely on global or thread-local state, you can create multiple independent gloo::Context objects. This allows your application to run multiple independent collective operations in parallel by assigning them to different contexts.

  11. Use CUDA-aware Allreduce algorithms

    main

    For GPU-based workloads, Gloo provides CUDA-aware implementations of Allreduce algorithms. These implementations manage the movement of data between GPU and CPU memory to facilitate communication.

    • cuda_allreduce_ring: Copies GPU buffers to system memory in parallel for CPU-based local reduction, then copies the result back to the GPU.
    • cuda_allreduce_ring_chunked: Uses NCCL to reduce GPU buffers into a single GPU buffer, then copies the result to system memory asynchronously. It uses pipelining for local reduction and broadcasting.
    • cuda_allreduce_halving_doubling: A standard CUDA-aware implementation without pipelining.
    • cuda_allreduce_halving_doubling_pipelined: An optimized version that pipelines local reduction/broadcast steps with communication steps.

  12. Understand Reduce-Scatter algorithms and semantics

    main

    A Reduce-Scatter operation computes a user-specified reduction (e.g., sum) of $N$ arrays per process across $P$ processes. The computation is performed in place, and the result is scattered to all processes according to user specifications.

    Every Reduce-Scatter implementation follows three phases:

    1. Local reduction: Reducing the $N$ buffers locally.
    2. Reduce-Scatter between processes: Communicating and scattering the reduced data.
    3. Broadcast: Distributing the scattered result back to the $N$ buffers.

    reduce_scatter_halving_doubling implementation

    • Communication steps: $\lg(P)$
    • Bytes on the wire: $S$ (for scattering result evenly among $P$ processes)

    This algorithm uses a recursive vector-halving, distance-doubling approach. For non-power-of-two process counts, it uses binary blocks and handles load imbalance by communicating interblock after the intrablock reduce-scatter.