Loading...
Loading...
Multimodal 552B MoE (8B active prefill / 16B decode) with 1M context and optional reasoning effort, the V4.1 refresh of the fast, high-throughput Flash line for coding and agent workflows
[interactive architecture ↓]DeepSeek is a Chinese AI lab founded in 2023 in Hangzhou by Liang Wenfeng. It is funded solely by High-Flyer Quant, the quantitative hedge fund Liang co-founded in 2015, and grew out of GPU capacity originally stockpiled for trading research. The lab publishes frontier open-weights models under permissive licenses.
more about DeepSeek →Take DeepSeek V4.1 Flash apart, layer by layer. The 3D teardown below shows its 40 transformer layers (0 dense + 40 MoE), MLA attention, and expert routing, with values from the public model config.
5120-dim vectors over a 129,280-token vocabulary. Image tokens from a 32-layer DeepSeek-ViT encoder join the same stream (not drawn). Encoder blocks 2 and 15 also look up Engram conditional memory: 196B parameters of token n-gram tables (n-grams up to 4, 8 heads of 256 dims) read sparsely per token, outside the 552B backbone.
40 layers in two halves. The decoder's global KV cache is projected from the final encoder hidden states instead of each decoder layer's own states, which is why prefill activates only 8B parameters and decode 16B. Single-Pass mHC carries 4 parallel residual streams around every attention and MoE sublayer. The scene draws one 40-layer stack; the encoder / decoder split and the CED bridge are not shown.
64 query heads attend to a single 512-dim compressed KV head with a 64-dim RoPE part. The first two layers are SWA-only; the other 38 pair a 128-token sliding window with Compressed Sparse Attention 2, whose Full / Reindex / Reuse modes share main KV across layers (sources at layers 2, 8, 14, 20) and reuse top-k indices (32 indexer heads, top-k 512). Global KV is 890 bytes per token. (Rendered as the MLA block.)
384 routed experts plus 1 shared, 6 routed active per token, expert dim 2304, sqrt(softplus) routing. Of the 552B backbone, 8B is active during prefill and 16B during decode.
The 4 mHC residual streams are mixed back to one, then RMSNorm and a projection to 129,280 logits. Three MTP layers and DSpark speculative decoding (target layers 37-39 with 128 draft experts, 3 active) speed up generation; neither is drawn.