Loading...
Loading...
Open-source reasoning model rivalling o1, 671B params (37B active), transparent thinking tokens
[interactive architecture ↓]DeepSeek is a Chinese AI lab founded in 2023 in Hangzhou by Liang Wenfeng. It is funded solely by High-Flyer Quant, the quantitative hedge fund Liang co-founded in 2015, and grew out of GPU capacity originally stockpiled for trading research. The lab publishes frontier open-weights models under permissive licenses.
more about DeepSeek →Take DeepSeek R1 (0528) apart, layer by layer. The 3D teardown below shows its 61 transformer layers (3 dense + 58 MoE), MLA attention, and expert routing, with values from the public model config.
Each input token id maps to a 7168-dim vector from a 129,280-token vocabulary. Identical base to V3; R1 inherits the same tokenizer and embedding table.
R1 shares V3's pre-norm block exactly: RMSNorm, attention, residual add, RMSNorm, FFN, residual add. The first 3 layers are dense; the remaining 58 are MoE. The reasoning behaviour comes entirely from RL post-training, not architecture changes.
128 query heads share a compressed 512-dim KV latent, the same MLA as V3. This keeps the KV cache small for long reasoning chains that span many thousands of tokens.
A router scores 256 experts per token and activates 8 plus 1 shared. R1 spends 37B of its 671B parameters per token, the same sparse routing as V3.
After a final RMSNorm, the hidden state projects to 129,280 logits. R1 was trained with GRPO reinforcement learning to emit <think> reasoning traces before the final answer, improving math, code, and logic benchmarks without changing the architecture.