Loading...
Loading...
The official release of DeepSeek V4 Pro, a 1.7T MoE superseding the April preview with greatly enhanced agentic capabilities, 1M context, and optional reasoning effort
[interactive architecture ↓]DeepSeek is a Chinese AI lab founded in 2023 in Hangzhou by Liang Wenfeng. It is funded solely by High-Flyer Quant, the quantitative hedge fund Liang co-founded in 2015, and grew out of GPU capacity originally stockpiled for trading research. The lab publishes frontier open-weights models under permissive licenses.
more about DeepSeek →Take DeepSeek V4 Pro 0813 apart, layer by layer. The 3D teardown below shows its 61 transformer layers (0 dense + 61 MoE), MLA attention, and expert routing, with values from the public model config.
7168-dim vectors over a 129,280-token vocabulary, the same width as V3 and unchanged from the 0423 preview.
Pre-norm blocks with Manifold-Constrained Hyper-Connections (mHC) replacing plain residual adds. The first 3 layers use static Hash-MoE routing, the remaining 58 use learned MoE routing. A DSpark speculative decoding module is attached at layers 58-60 (not drawn).
Compressed Sparse Attention and Heavily Compressed Attention with a 128-token local sliding window: 128 query heads onto a single 512-dim compressed KV head with a 64-dim RoPE part. The sparse indexer (64 heads, top-k 1024) makes the 1M context tractable. (Rendered here as the MLA block.)
384 routed experts, 6 active per token, plus 1 shared expert, each 3072 wide. The router scores with sqrt(softplus) affinity. Pro activates 49B parameters per token; the 1.65T checkpoint is the 1.6T backbone plus the DSpark draft module.
Per-head RMSNorm then a projection to 129,280 logits. DSpark drafts several tokens semi-autoregressively and the main model verifies them; 0813 pairs that with the agentic post-training that separates it from the preview.