Loading...
Loading...
The official release of DeepSeek V4 Flash, a 304B MoE superseding the April preview with substantially enhanced agentic capabilities and a 1.3M context, at the lowest price in the family
[interactive architecture ↓]DeepSeek is a Chinese AI lab founded in 2023 in Hangzhou by Liang Wenfeng. It is funded solely by High-Flyer Quant, the quantitative hedge fund Liang co-founded in 2015, and grew out of GPU capacity originally stockpiled for trading research. The lab publishes frontier open-weights models under permissive licenses.
more about DeepSeek →Take DeepSeek V4 Flash 0731 apart, layer by layer. The 3D teardown below shows its 43 transformer layers (0 dense + 43 MoE), MLA attention, and expert routing, with values from the public model config.
4096-dim vectors over a 129,280-token vocabulary, unchanged from the 0423 preview.
The same 43-layer backbone as the preview, with mHC residuals. The first 3 layers use static Hash-MoE routing; the rest use learned MoE routing. A DSpark speculative decoding module is attached at layers 40-42 (not drawn).
Compressed Sparse + Heavily Compressed hybrid with a 128-token sliding window, 64 query heads and a single 512-dim compressed KV head. The sparse indexer (64 heads, 512 top-k) keeps the 1M context cheap. (Rendered here as the MLA block.)
256 routed experts, 6 active per token, 1 shared. Flash activates 13B parameters per token; the 304B checkpoint is the 284B backbone plus the DSpark draft module.
Per-head RMSNorm then projection to 129,280 logits. DSpark drafts several tokens semi-autoregressively and the main model verifies them, the speed-up that distinguishes 0731 from the preview at inference time.