Loading...
Loading...
Flagship 1.6T MoE (49B active) with hybrid attention, 1M context, and high/xhigh reasoning for full-codebase analysis and long-horizon agents
[interactive architecture ↓]DeepSeek is a Chinese AI lab founded in 2023 in Hangzhou by Liang Wenfeng. It is funded solely by High-Flyer Quant, the quantitative hedge fund Liang co-founded in 2015, and grew out of GPU capacity originally stockpiled for trading research. The lab publishes frontier open-weights models under permissive licenses.
more about DeepSeek →Take DeepSeek V4 Pro apart, layer by layer. The 3D teardown below shows its 61 transformer layers (0 dense + 61 MoE), MLA attention, and expert routing, with values from the public model config.
4096-dim vectors over a 129,280-token vocabulary. The hidden dim shrank from V3's 7168 because V4 spends its parameter budget on far more expert capacity.
Pre-norm blocks with Manifold-Constrained Hyper-Connections (mHC) replacing plain residual adds. The first 3 layers use static Hash-MoE routing (content-independent), the remaining 58 use learned MoE routing.
Replaces MLA with a hybrid of Compressed Sparse Attention and Heavily Compressed Attention, plus a 128-token local sliding window and 64 heads. This is what makes the 1-million-token context tractable. (Rendered here as the MLA block; the real mechanism is CSA+HCA.)
256 routed experts, 6 active per token, plus 1 shared expert. The router scores with sqrt(softplus) affinity. V4 Pro activates 49B of its 1.6T parameters per token.
Per-head RMSNorm then a projection to 129,280 logits. V4 Pro targets agentic and long-document workloads that genuinely use the full 1M context.