AIConfigurator Documentation

The Kimi-K3 (KimiK3ForConditionalGeneration) bring-up ledger tracks the integration of performance data into the database across multiple hardware systems and backends. As of the final-state record (2026-08-04), the following lanes are complete across eight systems (b200_sxm, b300_sxm, gb200, gb300, h100_sxm, h200_sxm, l40s, rtx_pro_6000_server):

• kda: Supported via sglang 0.5.16 (kimi-k3 branch) and vllm 0.1.dev19262 (kimi-k3 preview).
• moe (K3 shape, 3584/3072, 896x16): Supported via sglang 0.5.14 (Blackwell precision lanes, Hopper/rtx Marlin, l40s Triton) and vllm 0.24.0 (situ-as-silu Marlin lane).
• mla_bmm 96-family (96/48/24/12 heads): Supported via sglang b200 and new vllm mla_bmm tables on b200/b300/gb200/gb300/h100/h200.
• MegaMoE module (K3 shape): Supported via b200 sglang 0.5.16 (64 rows).

Note: The support matrix contains 228 exact K3 rows at the merged head.

The AIConfigurator generator transforms unified YAML inputs and overrides into backend-specific configurations, bash run scripts, and Kubernetes manifests. It supports multiple backends (vLLM, SGLang, TRT-LLM) and orchestration platforms (Dynamo, llm-d, FPM).

End-to-End Flow

1. Inputs: Unified YAML + inline overrides.
2. Schema & Defaults: Uses deployment_config.yaml to define deployment-facing inputs (ports, images, cache settings, SLA knobs).
3. Parameter Aggregation: Combines service, K8s, and role-specific parameters.
4. Rule Plugins: Applies DSL rules (e.g., calculating GPUs per worker) to influence outputs.
5. Parameter Mapping: Harmonizes backend-specific flags under unified field names via backend_config_mapping.yaml.
6. Template Rendering: Uses Jinja templates to produce final artifacts.
7. Generated Artifacts: Produces k8s_deploy.yaml, run_*.sh, and engine configs/CLI args.

The SGLang operator performance collection tools support two distinct execution modes:

1. Direct Diagnostic Execution (Mode 1): Used for single GPU diagnostics by running specific scripts (collect_mla_module.py or collect_deepep_moe.py) directly with command-line arguments. This is ideal for isolated testing of a single module.

2. Framework Execution (Mode 2): Uses the collect.py framework to run integrated collections. This allows you to run multiple operators sequentially or specify specific modules (like dsa_context_module or wideep_moe). In this mode, cases within a single operator are distributed across the available GPU pool. Module-level operators use subprocess-based GPU isolation via CUDA_VISIBLE_DEVICES to prevent NCCL/distributed initialization conflicts.

# Example of Framework Execution (Mode 2)
python collect.py --backend sglang --ops mla_bmm_gen_pre dsa_context_module

To estimate the total prefill latency, assemble the components (DSA, MoE, and Communication) into a serial layer and multiply by the number of layers.

Assembly Pattern:

1. DSA: dsa = ContextDSAModule.get_cp_dsa(b, isl, cp, db, dims)
2. MoE: moe = MoEOp.get(num_tokens=isl, moe_tp=cp, moe_ep=ep, distribution=dist) (Note: MoE uses the full isl, not per-card).
3. Comm: comm = db.ag_latency(isl*hidden*2, cp) + db.rs_latency(isl*hidden*2, cp)
4. Layer: layer = dsa + moe + comm (These are treated as serial operations).
5. Total Prefill: prefill = layer * num_layers (Note: The first k dense layers may not have MoE/DSA-MoE communication).

Critical Invariants:

• Base Row: Must be bf16-KV (even if KV is stored in fp8, the FMHA kernel is bf16).
• MoE Parallelism: Communication primitives must follow the MoE parallel mode (e.g., TP+SP $\rightarrow$ AG_hidden+RS; EP $\rightarrow$ all-to-all).
• RS (Reduce-Scatter): There is one RS per layer after MoE, performing TP-reduce+SP-scatter. Exclude cross_device_reduce (allreduce) as it is for decode/post-processing.

dsa  = ContextDSAModule.get_cp_dsa(b, isl, cp, db, dims)
moe  = MoEOp.get(num_tokens=isl, moe_tp=cp, moe_ep=ep, distribution=dist)   # full tokens
comm = db.ag_latency(isl*hidden*2, cp) + db.rs_latency(isl*hidden*2, cp)    # AG_hidden + RS
layer = dsa + moe + comm        # serial
prefill = layer * num_layers    # (first_k_dense_replace layers have no MoE/DSA-MoE comm)

When a GEMM quantization mode has no collected data for a specific query, the system can estimate its utility in HYBRID or EMPIRICAL modes by borrowing a utility curve from a collected reference quant. This is managed via util_empirical.quant_transfer_grid.

Estimation is based on a correction ratio derived from the relationship between the query and the reference. The system uses three confidence labels to determine how to borrow data:

1. xshape: Same quantization, different slice. The correction is 1 (kernel identity).
2. xquant: Same profile, different quantization. The correction is 1 (structural identity based on SOL-coefficient).
3. xprofile: Different profile. Uses a level ratio (calibrated per-profile utility level) as a last resort.

Reference selection follows a lexicographic preference: relation rank, then profile distance, then slice-feature distance.

latency(query) = SOL_query / (util_ref × e(profile_query)/e(profile_ref))