DeepSeek Open Infra Index

The open-infra-index repository serves as a central hub for DeepSeek's open-source infrastructure components, including optimized kernels, communication libraries, and file systems used in the DeepSeek-V3 and R1 inference and training systems.

Key components include:

• FlashMLA: Efficient MLA decoding kernels for Hopper GPUs.
• DeepEP: An EP communication library for MoE models.
• DeepGEMM: An FP8 GEMM library for dense and MoE workloads.
• DualPipe: Bidirectional pipeline parallelism for computation-communication overlap.
• EPLB: Expert-parallel load balancer.
• 3FS: A high-performance parallel file system.
• Smallpond: A data processing framework built on 3FS.

DeepSeek is not open-sourcing its full internal inference engine as a single monolithic repository due to codebase divergence from vLLM, tight coupling with internal infrastructure, and maintenance constraints. Instead, the strategy is to contribute to the existing open-source ecosystem by:

1. Extracting Standalone Features: Modularizing and contributing reusable components as independent libraries.
2. Sharing Optimizations: Contributing design improvements and implementation details directly to existing projects like vLLM.

For future model releases (like DeepSeek-V3 or DeepSeek-R1), DeepSeek commits to synchronizing inference-related engineering efforts prior to launches to enable Day-0 SOTA support across diverse hardware platforms.

To mitigate cross-node communication overhead introduced by large-scale EP, DeepSeek employs specific overlapping strategies:

• Prefilling Phase: Uses a dual-batch overlap strategy. A batch of requests is split into two microbatches that execute alternately, allowing the communication cost of one microbatch to be hidden behind the computation of the other.
• Decoding Phase: Uses a 5-stage pipeline. Because execution durations are unbalanced, the attention layer is subdivided into two steps to achieve seamless communication-computation overlapping.

Detailed profiling data for these mechanisms can be found at https://github.com/deepseek-ai/profile-data.

The DeepSeek-V3/R1 inference system achieves optimized throughput and latency through cross-node EP-powered batch scaling, computation-communication overlap, and load balancing.

Production Performance (per H800 node):

• Input tokens: 73.7k tokens/s
• Output tokens: 14.8k tokens/s

For a deep dive, see the Inference System Overview.

DeepSeek utilizes a high-performance storage stack for training and inference:

• 3FS (Fire-Flyer File System): A parallel file system utilizing modern SSDs and RDMA networks. It features a disaggregated architecture with strong consistency semantics, supporting training data preprocessing, dataset loading, checkpointing, and KVCache lookups.
  • GitHub Repo
• Smallpond: A data processing framework built on top of 3FS.
  • GitHub Repo

FlashMLA is an optimized MLA (Multi-head Latent Attention) decoding kernel designed for Hopper GPUs. It is optimized for variable-length sequences and is battle-tested in production.

Key Features:

• BF16 support
• Paged KV cache (block size 64)
• Performance: 3000 GB/s memory-bound | BF16 580 TFLOPS compute-bound on H800

For more details, visit the FlashMLA GitHub Repo.

DeepSeek provides specialized tools for optimizing parallelism in V3/R1 training:

• DualPipe: A bidirectional pipeline parallelism algorithm designed for computation-communication overlap.
  • GitHub Repo
• EPLB: An expert-parallel load balancer for V3/R1.
  • GitHub Repo
• Profile Data: Tools to analyze computation-communication overlap in V3/R1.
  • GitHub Repo

DeepSeek-V3/R1 uses a prefill-decode disaggregation architecture with different parallelism degrees for each phase to optimize throughput and latency via cross-node Expert Parallelism (EP).

• Prefilling Phase: Uses [Routed Expert EP32, MLA/Shared Expert DP32]. A deployment unit spans 4 nodes with 32 redundant routed experts. Each GPU handles 9 routed experts and 1 shared expert.
• Decoding Phase: Uses [Routed Expert EP144, MLA/Shared Expert DP144]. A deployment unit spans 18 nodes with 32 redundant routed experts. Each GPU manages 2 routed experts and 1 shared expert.

DeepGEMM is an FP8 GEMM library that supports both dense and MoE GEMMs, used to power V3/R1 training and inference.

Key Features:

• High Performance: Up to 1350+ FP8 TFLOPS on Hopper GPUs.
• Lightweight: No heavy dependencies; fully Just-In-Time (JIT) compiled.
• Layout Support: Supports dense layout and two MoE layouts.

For more details, visit the DeepGEMM GitHub Repo.

DeepSeek-V3/R1 inference services are served on H800 GPUs using the following precision formats to ensure optimal performance:

• Matrix Multiplications: FP8
• Dispatch Transmissions: FP8
• Core MLA Computations: BF16
• Combine Transmissions: BF16

DeepEP is an open-source EP (Expert Parallel) communication library specifically designed for MoE (Mixture-of-Experts) model training and inference.

Key Features:

• All-to-all communication: Efficient and optimized.
• Multi-tier support: Both intranode and internode support using NVLink and RDMA.
• Kernel types: High-throughput kernels for prefilling and low-latency kernels for inference decoding.
• FP8 support: Native FP8 dispatch support.
• Resource control: Flexible GPU resource control for computation-communication overlapping.

For more details, visit the DeepEP GitHub Repo.