DOCSEARCHbyAlgolia
Submit Docs

DeepSeek-V3

repository·main·Indexed 13 days ago

https://github.com/deepseek-ai/deepseek-v3

A high-performance Mixture-of-Experts (MoE) language model with 671B total parameters and 37B activated parameters. It features Multi-head Latent Attention (MLA), Multi-Token Prediction (MTP), and FP8 mixed precision training. The model supports a 128K context window and is available in Base and Chat versions, with support for deployment via SGLang, LMDeploy, vLLM, and other high-performance frameworks.

Tokens
2.8K
Snippets
5
Records
15
Agent score
50%

What's inside DeepSeek-V3

  1. Overview of DeepSeek-V3

    main

    DeepSeek-V3 is a high-performance Mixture-of-Experts (MoE) language model featuring 671B total parameters, with 37B parameters activated per token. It utilizes Multi-head Latent Attention (MLA) and the DeepSeekMoE architecture to achieve efficient inference and cost-effective training.

    Key technical features include:

    • Auxiliary-loss-free load balancing: Minimizes performance degradation during training.
    • Multi-Token Prediction (MTP): A training objective that improves model performance and can be used for speculative decoding to accelerate inference.
    • FP8 Mixed Precision Training: Validated for large-scale model training.
    • Reasoning Distillation: Capabilities are distilled from the DeepSeek-R1 series to improve reasoning while maintaining control over output style and length.

  2. Supported inference frameworks for DeepSeek-V3

    main

    DeepSeek-V3 can be deployed using several high-performance open-source frameworks. Depending on your hardware (NVIDIA, AMD, or Huawei Ascend) and requirements (latency, throughput, or parallelism), choose from the following:

    FrameworkSupported ModesNotes
    SGLangBF16, FP8Recommended. Supports NVIDIA/AMD, MLA optimizations, and multi-node tensor parallelism.
    LMDeployBF16, FP8Recommended. Supports offline pipeline and online deployment.
    vLLMBF16, FP8Recommended. Supports tensor and pipeline parallelism on NVIDIA/AMD.
    TensorRT-LLMBF16, INT4/8Supports BF16 and quantization; FP8 support coming soon.
    LightLLMBF16, FP8Supports single/multi-node tensor parallel deployment.
    DeepSeek-Infer DemoBF16, FP8Lightweight demo (Linux/Python 3.10 only).
    MindIEBF16, INT8For Huawei Ascend NPU devices.

  3. Dequantize FP8 weights using weight_scale_inv

    main

    To perform runtime operations with FP8 weights, use the weight_scale_inv field provided in the weight file.

    • Storage: weight_scale_inv is stored as a float32 Tensor alongside the weight data.
    • Dequantization Formula: (128x128 weight block) * weight_scale_inv.
    • Padding Note: If a weight block is not aligned to 128, it is zero-padded to 128 for scale calculation, and the padded portion is removed after quantization.

    This method enables online quantization at a granularity of per-token-per-128-channel.

  4. Understand the DeepSeek-V3 weight structure

    main

    The DeepSeek-V3 weight file is composed of two primary components: the Main Model Weights and the Multi-Token Prediction (MTP) Modules.

    1. Main Model Weights

    • Composition: Includes input/output embedding layers and 61 Transformer hidden layers.
    • Parameter Count: 671B total parameters; 36.7B activation parameters.
    • Key Tensors:
      • Embedding: model.embed_tokens.weight
      • Hidden Layers: model.layers.0 through model.layers.60
      • Output: model.norm.weight and lm_head.weight

    2. Multi-Token Prediction (MTP) Modules

    • Composition: Additional modules defined by num_nextn_predict_layers (set to 1 in the open-sourced weights).
    • Parameter Count: 11.5B unique parameters; 2.4B activation parameters.
    • Key Tensors:
      • embed_tokens: Shares parameters with the Main Model's embedding layer.
      • enorm & hnorm: RMSNorm parameters for speculative decoding.
      • eh_proj: Dimensionality reduction projection parameters.
      • model.layers.61.self_attn & mlp: An additional Transformer layer (structure identical to main layers).
      • shared_head: Shares parameters with the Main Model's output head.

  5. Set up DeepSeek-Infer Demo for local inference

    main

    The DeepSeek-Infer Demo is a lightweight implementation for FP8 and BF16 inference.

    System Requirements:

    • Linux with Python 3.10 only (Mac and Windows are not supported).

    Dependencies:

    • torch==2.4.1
    • triton==3.0.0
    • transformers==4.46.3
    • safetensors==0.4.5

    Setup Steps:

    1. Clone the repository: git clone https://github.com/deepseek-ai/DeepSeek-V3.git
    2. Install dependencies: Navigate to the inference folder and run pip install -r requirements.txt.
    3. Download model weights from Hugging Face and place them in your target directory.
    4. Convert Hugging Face weights to the demo format using convert.py before running.
    git clone https://github.com/deepseek-ai/DeepSeek-V3.git
cd DeepSeek-V3/inference
pip install -r requirements.txt

  7. Convert FP8 weights to BF16

    main

    Since the framework natively adopts FP8, only FP8 weights are provided by default. If you require BF16 weights for experimentation, use the fp8_cast_bf16.py script located in the inference directory.

    Usage: Pass the path to your FP8 Hugging Face weights as --input-fp8-hf-path and specify the destination for the BF16 weights using --output-bf16-hf-path.

    cd inference
python fp8_cast_bf16.py --input-fp8-hf-path /path/to/fp8_weights --output-bf16-hf-path /path/to/bf16_weights

  8. Loading rules for Main Model and MTP weights

    main

    Weights must be loaded according to specific layer ID rules:

    1. Main Model Weights: Loaded using the num_hidden_layers parameter from config.json.
    2. MTP Modules: Loaded using the num_nextn_predict_layers parameter. Layer IDs for MTP modules are appended immediately after the Main Model hidden layers.

    Example: If num_hidden_layers = 61 and num_nextn_predict_layers = 1, the MTP Module's layer ID is 61.

  9. Download DeepSeek-V3 models

    main

    DeepSeek-V3 models are available on Hugging Face. There are two primary versions: the Base model and the Chat model.

    ModelTotal ParamsActivated ParamsContext LengthDownload
    DeepSeek-V3-Base671B37B128KHugging Face
    DeepSeek-V3671B37B128KHugging Face

    Note on Model Size: The total size of the models on Hugging Face is approximately 685B. This includes 671B for the Main Model weights and 14B for the Multi-Token Prediction (MTP) Module weights. For detailed information on weight structures, refer to README_WEIGHTS.md.

  10. Configure FP8 quantization for DeepSeek-V3

    main

    DeepSeek-V3 supports native FP8 weight format using 128x128 block scaling and dynamic activation quantization. The quantization_config field in config.json defines these settings.

    Configuration Schema Example:

    "quantization_config": {
  "activation_scheme": "dynamic",
  "fmt": "e4m3",
  "quant_method": "fp8",
  "weight_block_size": [128, 128]
}
    • fmt: Use e4m3 (corresponding to torch.float8_e4m3fn).
    • weight_block_size: Set to [128, 128].
    • activation_scheme: Set to dynamic.

  11. Run interactive or batch inference with DeepSeek-Infer Demo

    main

    After converting the weights, you can run inference using torchrun. This requires a multi-node setup (e.g., 2 nodes with 8 processes per node).

    Interactive Mode: Use the --interactive flag to chat with the model.

    Batch Mode: Provide an --input-file to perform batch inference on a file.

    Required Arguments:

    • --ckpt-path: Path to the converted demo weights.
    • --config: Path to the model configuration JSON (e.g., configs/config_671B.json).
    • --nnodes: Number of nodes.
    • --nproc-per-node: Number of processes per node.
    • --node-rank: The rank of the current node.
    • --master-addr: The address of the master node.
    # Interactive chat
torchrun --nnodes 2 --nproc-per-node 8 --node-rank $RANK --master-addr $ADDR generate.py --ckpt-path /path/to/DeepSeek-V3-Demo --config configs/config_671B.json --interactive --temperature 0.7 --max-new-tokens 200

# Batch inference
torchrun --nnodes 2 --nproc-per-node 8 --node-rank $RANK --master-addr $ADDR generate.py --ckpt-path /path/to/DeepSeek-V3-Demo --config configs/config_671B.json --input-file $FILE

  12. Convert Hugging Face weights for DeepSeek-Infer Demo

    main

    To use the DeepSeek-Infer Demo, you must first convert the standard Hugging Face checkpoint into the specific format required by the demo using convert.py.

    Arguments:

    • --hf-ckpt-path: Path to the original Hugging Face checkpoint.
    • --save-path: Path where the converted weights will be saved.
    • --n-experts: Number of experts (e.g., 256).
    • --model-parallel: Model parallelism degree (e.g., 16).
    python convert.py --hf-ckpt-path /path/to/DeepSeek-V3 --save-path /path/to/DeepSeek-V3-Demo --n-experts 256 --model-parallel 16