DeepSeek-V3.2-Exp Documentation

DeepSeek-V3.2-Exp is an experimental model that introduces DeepSeek Sparse Attention (DSA). DSA provides fine-grained sparse attention to improve training and inference efficiency in long-context scenarios while maintaining performance levels comparable to the V3.1-Terminus model.

Key architectural note: When implementing custom inference, note that the input tensor to RoPE in the indexer module requires a non-interleaved layout, whereas RoPE in the MLA module expects an interleaved layout.

To run the model locally using the provided inference demo, you must first convert the HuggingFace model weights into the format required by the demo. You will need to specify the number of experts (e.g., 256) and set the MP (Model Parallel) variable to match your available GPU count.

After conversion, you can launch an interactive chat interface using torchrun with the appropriate configuration file.

cd inference
export EXPERTS=256
python convert.py --hf-ckpt-path ${HF_CKPT_PATH} --save-path ${SAVE_PATH} --n-experts ${EXPERTS} --model-parallel ${MP}

export CONFIG=config_671B_v3.2.json
torchrun --nproc-per-node ${MP} generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --interactive

To launch an interactive chat session, use torchrun to distribute the process across your GPUs. You must provide the path to the converted checkpoint, the specific model configuration JSON file, and the model parallel degree (MP).

Ensure you have exported the CONFIG environment variable pointing to your configuration file (e.g., config_671B_v3.2.json) before running the command.

export CONFIG=config_671B_v3.2.json
torchrun --nproc-per-node ${MP} generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --interactive

Before running the inference demo, you must convert HuggingFace model weights into the format required by the DeepSeek inference engine. You need to specify the number of experts, the model parallel degree (MP), the source HuggingFace checkpoint path, and the destination save path.

Set the EXPERTS environment variable (e.g., 256) to match the model's expert count and MP to match your available GPU count.

cd inference
export EXPERTS=256
python convert.py --hf-ckpt-path ${HF_CKPT_PATH} --save-path ${SAVE_PATH} --n-experts ${EXPERTS} --model-parallel ${MP}

SGLang supports DeepSeek-V3.2-Exp via Docker containers optimized for different hardware architectures.

1. Install via Docker

Select the image corresponding to your hardware:

• H200: lmsysorg/sglang:dsv32
• MI350: lmsysorg/sglang:dsv32-rocm
• NPUs: lmsysorg/sglang:dsv32-a2 or lmsysorg/sglang:dsv32-a3

2. Launch the Server

Use the following command to launch the server with Tensor Parallelism (--tp) and Data Parallelism (--dp) enabled, along with --enable-dp-attention for optimized performance.

# Example Docker Pulls
docker pull lmsysorg/sglang:dsv32

# Launch Command
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 --enable-dp-attention

Depending on your use case, you can access different kernel implementations:

• Research/Readability: Use TileLang for kernels designed for better readability.
• High-Performance CUDA:
  • Indexer logit kernels (including paged versions) are available in DeepGEMM.
  • Sparse attention kernels are released in FlashMLA.