DOCSEARCHbyAlgolia
Submit Docs

DeepEP (DeepEveryParallel) Documentation

repository·main·Indexed 27 days ago

https://github.com/deepseek-ai/deepep

A high-performance communication library optimized for Mixture-of-Experts (MoE) training and inference. DeepEP provides high-throughput, low-latency all-to-all GPU kernels for dispatch and combine operations, featuring FP8 support, minimal SM overhead, and JIT compilation. It supports Expert Parallelism (EP) and experimental primitives for Pipeline Parallelism (PP), Context Parallelism (CP), and Remote Memory Access (Engram), utilizing a V2 backend based on the NCCL Gin backend.

Tokens
4.4K
Snippets
14
Records
28
Agent score
95%

What's inside DeepEP

  1. Overview of DeepEP (DeepEveryParallel)

    main

    DeepEP is a high-performance communication library designed for modern machine learning training and inference. It primarily focuses on Expert Parallelism (EP), providing high-throughput and low-latency all-to-all GPU kernels for MoE (Mixture-of-Experts) dispatch and combine operations.

    Key features include:

    • Low-precision support: Includes FP8 support.
    • Minimal SM occupation: Designed for zero or minimal Streaming Multiprocessor (SM) usage.
    • JIT Compilation: All kernels are compiled at runtime via a lightweight Just-In-Time (JIT) module, meaning no CUDA compilation is required during installation.
    • Experimental Primitives: Supports Pipeline Parallelism (PP), Context Parallelism (CP), and Remote Memory Access (Engram).
    • V2 Backend: Uses the lightweight NCCL Gin backend, which is header-only and can reuse existing NCCL communicators.

  3. Perform MoE Dispatch and Combine for Inference Decoding

    main

    For inference decoding, use the ElasticBuffer interface with a handle-caching pattern. If the cached_handle (an EPHandle from a previous iteration) is provided to dispatch, the library reuses the routing metadata and skips CPU synchronization, which is more efficient when gating decisions remain unchanged.

    # Dispatch with cached handle to avoid CPU sync
recv_x, _, _, handle, event = _buffer.dispatch(
    x, 
    handle=cached_handle, 
    num_sms=_num_comm_sms, 
    async_with_compute_stream=True
)

# Combine
combined_x, _, event = _buffer.combine(
    x, 
    handle=handle, 
    num_sms=_num_comm_sms, 
    async_with_compute_stream=True
)

  4. Develop and test DeepEP V1

    main

    To build DeepEP V1 for development and run test cases, follow these steps:

    1. Build and link: Build the project and create symbolic links for the .so files. Note that .so names may vary by platform.
    2. Run tests: Execute the provided test scripts. You may need to modify the init_dist function in tests/utils.py to match your cluster settings and launch across multiple nodes.
    # Build and make symbolic links for SO files
NVSHMEM_DIR=/path/to/installed/nvshmem python setup.py build
# You may modify the specific SO names according to your own platform
ln -s build/lib.linux-x86_64-cpython-38/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so

# Run test cases
# NOTES: you may modify the `init_dist` function in `tests/utils.py
# according to your own cluster settings, and launch into multiple nodes
python tests/test_intranode.py
python tests/test_internode.py
python tests/test_low_latency.py

  6. Configure Network Traffic Isolation for DeepEP V1

    main
    To prevent interference between different workloads (normal kernels, low-latency kernels, and other traffic), you can use InfiniBand Virtual Lanes (VL). In DeepEP V1, control the virtual lane assignment by setting the NVSHMEM_IB_SL environment variable.

  7. Optimize performance via auto-tuning

    main
    The default configurations in DeepEP are optimized for DeepSeek's internal cluster. To achieve optimal performance on your own cluster, you should run all the provided tests to generate auto-tuned configurations.

  8. Optimize RDMA Atomic Performance via PCI Atomic Mode

    main

    If your hardware supports it, you can improve RDMA atomic operation performance by setting the NIC's PCI_ATOMIC_MODE to 4. Use the mlxconfig utility to apply this setting.

    sudo mlxconfig -y -d mlx5_$i set PCI_ATOMIC_MODE=4

  10. Perform MoE Dispatch and Combine for Training or Prefilling

    main

    Use the ElasticBuffer.dispatch and ElasticBuffer.combine methods to route tokens to experts and reduce expert outputs back to original ranks.

    • Dispatch: Routes tokens to experts. Supports BF16 and FP8 (where x is a tuple of [data, scale_factors]). Returns recv_x, recv_topk_idx, recv_topk_weights, an EPHandle (containing routing metadata), and an EventOverlap object.
    • Combine: Reduces expert outputs.
    • Backward Pass: In V2, the backward pass of a dispatch is implemented as a combine call, and the backward pass of a combine is implemented as a dispatch call.

    To overlap communication with computation, use event.current_stream_wait() to synchronize the compute stream with the communication stream.

    # Dispatch (Forward)
recv_x, recv_topk_idx, recv_topk_weights, handle, event = _buffer.dispatch(
    x, 
    topk_idx=topk_idx, 
    topk_weights=topk_weights, 
    num_experts=num_experts, 
    num_max_tokens_per_rank=num_max_tokens_per_rank, 
    expert_alignment=expert_alignment, 
    num_sms=_num_comm_sms, 
    async_with_compute_stream=True
)

# Overlap computation
# ... do some independent computation here ...

# Synchronize before using results
event.current_stream_wait()

# Combine (Forward)
combined_x, _, event = _buffer.combine(
    x, 
    handle=handle, 
    num_sms=_num_comm_sms, 
    async_with_compute_stream=True
)

  11. Enable NVSHMEM IBGDA support via GDRCopy

    main

    If you cannot modify the NVIDIA driver registry keys, you can enable IBGDA through asynchronous post-send operations assisted by the CPU. This method involves installing GDRCopy and loading the gdrdrv kernel module. This approach may incur a small performance penalty.

    1. Download GDRCopy (prebuilt deb/rpm or source from GitHub).
    2. Follow the installation instructions in the GDRCopy GitHub repository.

  12. Develop and build DeepEP from source

    main

    If you are developing with DeepEP, you can build the project and create symbolic links for the shared object (.so) files.

    Note: You may need to modify the specific .so names according to your own platform/environment.

    # Build and make symbolic links for SO files
python setup.py build
# You may modify the specific SO names according to your own platform
ln -s build/lib.linux-x86_64-cpython-38/deep_ep_cpp.cpython-38-x86_64-linux-gnu.so