DOCSEARCHbyAlgolia
Submit Docs

MSCCL++ Documentation

repository·main·Indexed 19 days ago

https://github.com/microsoft/mscclpp

A high-performance, GPU-driven communication stack designed to optimize inter-GPU data movement for scalable AI applications. It provides a Python API and C++ interfaces for 1-sided 0-copy communication, offering hardware-agnostic abstractions across NVLink/xGMI and InfiniBand. MSCCL++ includes a Domain-Specific Language (DSL) for defining communication collectives and specialized channel types (PortChannel and MemoryChannel) to improve LLM inference performance, particularly for latency-sensitive AllReduce operations during token sampling.

Tokens
33.7K
Snippets
78
Records
119
Agent score
69%

What's inside MSCCL++

  1. Overview of MSCCL++

    main

    MSCCL++ is a GPU-driven communication stack designed for scalable AI applications. It provides a highly efficient and customizable interface for inter-GPU communication, specifically optimized for the diverse performance requirements of state-of-the-art AI workloads (such as LLM inference with tensor parallelism).

    Key features include:

    • Multi-layer Abstractions: Offers lightweight abstractions ranging from low-level hardware-proximate logic (for implementing data movement inside GPU kernels) to high-level Python-based building blocks.
    • 1-sided 0-copy Communication: Provides fine-grained synchronous and asynchronous primitives (put(), get(), signal(), flush(), and wait()) that allow for 0-copy data transfers. This enables overlapping communication with computation and implementing custom collective algorithms without intermediate buffers or deadlocks.
    • Hardware Agnostic: Provides unified abstractions that work consistently across different interconnection hardware (NVLink/xGMI or InfiniBand) and different GPU locations (local node or remote node).

  2. Understand the MSCCL++ C++ API structure

    main

    The MSCCL++ C++ API is divided into two primary categories based on where the code executes:

    1. Host-Side Interfaces: Used in CPU code for setup, memory management, connection coordination, and process synchronization. This includes classes for Bootstrap, Connection, Communicator, Semaphore, Channel, and Executor.
    2. Device-Side Interfaces: Designed for use directly within GPU kernels (CUDA/HIP). These provide handles for communication primitives such as MemoryChannelDeviceHandle, PortChannelDeviceHandle, FifoDeviceHandle, and various semaphore and atomic operations.

  3. Explore MSCCL++ Documentation

    main

    The MSCCL++ documentation is organized into several key sections to help you get started and advance your usage:

    • Overview: High-level features and capabilities of MSCCL++.
    • Quick Start: Instructions for building, installing, and running the project.
    • MSCCL++ DSL: Guidance on using the MSCCL++ Domain Specific Language.
    • Tutorials: Step-by-step guides for implementing GPU communication.
    • Programming Guide: Advanced topics and best practices.
    • C++ API Reference: Detailed technical documentation for the C++ interface.
    • Python API Reference: Detailed technical documentation for the Python interface.

  4. Use Device-Side Interfaces in GPU kernels

    main

    To perform communication within a GPU kernel, you must use the device-side handles. These handles are passed from the host to the device. Key components include:

    • Channel Device Handles: Use mscclpp::MemoryChannelDeviceHandle, mscclpp::PortChannelDeviceHandle, or mscclpp::SwitchChannelDeviceHandle for data movement.
    • Semaphore Device Handles: Use mscclpp::Host2DeviceSemaphoreDeviceHandle or mscclpp::MemoryDevice2DeviceSemaphoreDeviceHandle for synchronization.
    • FIFO Device Handles: Use mscclpp::FifoDeviceHandle for queue-based communication.
    • Atomics and Utilities: MSCCL++ provides device-side atomic operations (mscclpp::atomicLoad, mscclpp::atomicStore, etc.) and vector data types (e.g., mscclpp::f32x4, mscclpp::i32x2) optimized for communication workloads.

  5. How MSCCL++ improves LLM inference performance

    main

    MSCCL++ is particularly effective for scaling LLM inference using tensor parallelism. LLM workloads typically involve two phases with vastly different communication requirements:

    1. Prompt Processing: Uses large batch sizes (often equal to context length), resulting in large AllReduce sizes (e.g., ~48MB for GPT-3 size).
    2. Token Sampling: Uses smaller batch sizes (corresponding to concurrent users), resulting in much smaller AllReduce sizes (e.g., ~384KB for 16 concurrent users).

    MSCCL++ provides significant speedups over NCCL, especially for these smaller, latency-sensitive AllReduce operations used during token sampling, which are critical for efficient large-scale LLM serving.

  6. How MemoryChannel and RegisteredMemory work together

    main

    MemoryChannel provides direct access to remote GPU memory for communication. To establish a MemoryChannel, processes must exchange RegisteredMemory objects representing their local memory regions.

    Workflow

    1. Process A creates a RegisteredMemory object (A).
    2. Process B creates a RegisteredMemory object (B).
    3. Process A sends its RegisteredMemory A to Process B.
    4. Process B creates a MemoryChannel using its own RegisteredMemory B and the received RegisteredMemory A.
    5. The process is repeated in reverse so both sides have a MemoryChannel capable of bidirectional communication.

    This requires a pre-built Semaphore to coordinate the exchange, as described in the Basic Concepts tutorial.

  7. Use MemoryChannel for direct remote GPU memory access

    main

    A mscclpp::MemoryChannel allows direct access to remote GPU memory regions. To use it, you must first exchange mscclpp::RegisteredMemory objects between processes using the Communicator's sendMemory() and recvMemory() methods. Once exchanged, you can construct a MemoryChannel by providing a semaphore for synchronization, the remote destination memory, and the local source memory.

    Workflow:

    1. Serialize/Exchange Metadata: Use comm.sendMemory(localRegMem, remoteRank) and comm.recvMemory(remoteRank) to share memory region metadata.
    2. Construct Channel: Initialize mscclpp::MemoryChannel(sema, remoteRegMem, localRegMem).
    3. Perform Transfers: Use put() to write to remote memory or get() to read from remote memory via a MemoryChannelDeviceHandle in a GPU kernel.
    // 1. Exchange RegisteredMemory metadata
comm.sendMemory(localRegMem, remoteRank);
auto remoteRegMemFuture = comm.recvMemory(remoteRank);
mscclpp::RegisteredMemory remoteRegMem = remoteRegMemFuture.get();

// 2. Construct the MemoryChannel
mscclpp::MemoryChannel memChan(sema, remoteRegMem, localRegMem);

  8. Use Thread Block Groups to manage thread block allocation

    main

    The ThreadBlockGroup feature (currently a prototype) allows you to define a specific set of thread blocks to be used for operations. This enables non-uniform allocation, where different operations can be assigned different numbers or specific sets of thread blocks as needed.

    To use it, instantiate a ThreadBlockGroup with a list of thread block IDs and pass it to the tb_group parameter of a rank operation (e.g., rank.copy).

    # Create a Thread Block Group with 4 thread blocks
tbg = ThreadBlockGroup(tb_list=[0, 1, 2, 3])
# Use the Thread Block Group to perform the copy operation
rank.copy(output_buffer[0:1], input_buffer[0:1], tb_group=tbg)

  9. Understand MSCCL++ automatic resource management lifecycle

    main

    MSCCL++ uses a hierarchical ownership model where the lifetime of a parent object is tied to its children. In most cases, you do not need to explicitly destroy objects. The following lifecycle dependencies apply:

    • Context: Remains alive as long as the Context itself or any Connection objects created by it are alive.
    • Connection: Remains alive as long as the Connection itself or any SemaphoreStub objects created from it are alive.
    • SemaphoreStub: Remains alive as long as the SemaphoreStub itself or any Semaphore objects created from it are alive.
    • Semaphore: Remains alive as long as the Semaphore itself or any Channels created from it are alive.

  10. Understand the MSCCL++ abstraction layers

    main

    MSCCL++ provides three distinct levels of abstraction depending on your performance and development needs:

    1. Primitive API (Lowest Level): A boilerplate-free C++ API used for writing highly flexible, custom GPU communication kernels from scratch.
    2. DSL API (Middle Level): A Python-based Domain-Specific Language (DSL) designed for quickly developing and scaling large-scale collective communication algorithms.
    3. NCCL API (Highest Level): A reimplementation of the NCCL API. This allows you to swap NCCL for MSCCL++ in existing distributed GPU applications without modifying your application code.

  11. How PortChannel and ProxyService work together

    main

    A PortChannel enables data transfer between GPUs by offloading tasks to I/O ports (like GPU Copy Engines, InfiniBand QPs, or TCP sockets) instead of using GPU threads. This reduces interference with other GPU operations but may introduce higher latency.

    To use a PortChannel, you must use a ProxyService to manage the communication. The workflow is:

    1. Instantiate a mscclpp::ProxyService.
    2. Add Semaphore and RegisteredMemory objects to the ProxyService to obtain SemaphoreId and MemoryIds.
    3. Create the PortChannel using the proxyService.portChannel() method with these IDs.
    4. Call proxyService.startProxy() to start the background host thread that processes communication requests.
    5. Execute GPU kernels using the PortChannel device handle.
    6. Call proxyService.stopProxy() once all GPU operations are complete.
    mscclpp::ProxyService proxyService;
// Add objects to get IDs
mscclpp::SemaphoreId semaId = proxyService.addSemaphore(sema);
mscclpp::MemoryId localMemId = proxyService.addMemory(localRegMem);
mscclpp::MemoryId remoteMemId = proxyService.addMemory(remoteRegMem);

// Create the channel
mscclpp::PortChannel portChan = proxyService.portChannel(semaId, remoteMemId, localMemId);

// Start the host-side service
proxyService.startProxy();

// ... Run GPU kernels using portChan ...

// Stop the service
proxyService.stopProxy();

  12. How MSCCL++ DSL primitives work

    main

    Collective Definition

    AllGather(num_gpus, chunk_factor, inplace) defines the operation. inplace=True means the input buffer is a slice of the output buffer.

    Ranks and Buffers

    • Rank(id): Represents a specific GPU rank.
    • rank.get_output_buffer(): Returns the buffer where data is stored/received.

    Channels

    • MemoryChannel(dst_rank, src_rank): Used for fast, direct intra-node memory access.
    • PortChannel: Used for inter-node communication.

    Synchronization and Data Transfer

    • channel.signal(tb, relaxed, data_sync): Notifies a remote GPU of a state change. tb is the thread block ID. relaxed=True uses relaxed memory ordering.
    • channel.wait(tb, data_sync, relaxed): Waits for a remote GPU to reach a specific SyncType (e.g., SyncType.after or SyncType.before).
    • channel.put(dst_chunk, src_chunk, tb): Performs the actual data write from local to remote memory.