DeepSeek V3 (0324)

DeepSeek[text][open-weights]

Budget-friendly 685B MoE model, high-volume capable with 2M+ daily requests

[interactive architecture ↓]

Input

$0.20

per 1M tokens

Output

$0.77

per 1M tokens

Context

164K tokens

Max Output

66K tokens

Released

2025-03-24

Features

[tool-calling][long-context][cost-effective]

Technical Specifications

LicenseMIT

Benchmarks

AIME 202459.4%

GPQA Diamond68.4%

LiveCodeBench49.2%

MMLU-Pro81.2%

The Lab

location: Hangzhou, Chinafounded: 2023backed by: High-Flyer Quant

DeepSeek is a Chinese AI lab founded in 2023 in Hangzhou by Liang Wenfeng. It is funded solely by High-Flyer Quant, the quantitative hedge fund Liang co-founded in 2015, and grew out of GPU capacity originally stockpiled for trading research. The lab publishes frontier open-weights models under permissive licenses.

more about DeepSeek →

Interactive Architecture

Take DeepSeek V3 (0324) apart, layer by layer. The 3D teardown below shows its 61 transformer layers (3 dense + 58 MoE), MLA attention, and expert routing, with values from the public model config.

61 layers (3 dense → 58 MoE)

MLA attention · 128Q heads

8+1 of 256 experts active

671B params · 37B active

Token embedding

Each input token id is mapped to a 7168-dim vector drawn from a 129,280-token vocabulary.

61 transformer layers

Every layer runs the same pre-norm block: RMSNorm, attention, a residual add, RMSNorm again, then the FFN with one more residual add. The first 3 layers use a dense FFN so low-level features stabilize early; the remaining 58 use the MoE FFN in the exploded view.

Multi-head Latent Attention

Instead of caching 128 full KV heads, MLA compresses keys and values into a shared 512-dim latent vector, keeping the KV cache small enough for a 128K+ context. RoPE adds position information.

Mixture of Experts FFN

A learned router scores all 256 routed experts for every token, then sends it to just 8 of them plus 1 always-on shared expert. The model stores 671B parameters yet spends only 37B per token.

Output head

After a final RMSNorm, the hidden state is projected back to 129,280 vocabulary logits to predict the next token.

What vocabulary and embedding size does DeepSeek V3 (0324) use?+

Each input token id is mapped to a 7168-dim vector drawn from a 129,280-token vocabulary.

How many transformer layers does DeepSeek V3 (0324) have?+

What is Multi-head Latent Attention (MLA) in DeepSeek V3 (0324)?+

Instead of caching 128 full KV heads, MLA compresses keys and values into a shared 512-dim latent vector, keeping the KV cache small enough for a 128K+ context. RoPE adds position information.

How does the Mixture of Experts in DeepSeek V3 (0324) work?+

A learned router scores all 256 routed experts for every token, then sends it to just 8 of them plus 1 always-on shared expert. The model stores 671B parameters yet spends only 37B per token.

How does DeepSeek V3 (0324) produce output tokens?+

After a final RMSNorm, the hidden state is projected back to 129,280 vocabulary logits to predict the next token.

scene values sourced from:HuggingFace config paper

Family · DeepSeek V3 series

2024-01

DeepSeek V1 67B

dense, first release

→

2024-05

DeepSeek V2 236B

first MoE + MLA

→

2024-12

DeepSeek V3 671B

FP8 training, MTP

→

2025-03

DeepSeek V3 (0324)

code/math bump, MIT license

→

→

→

1.6T MoE, hybrid attention

Training Run

Tokens

14.8T

Hardware

2048 × NVIDIA H800

GPU Hours

2.788M H800 GPU-hours

Est. Cost

~$5.576M (final run, at $2/GPU-hour)

Duration

< 2 months pre-training

Precision

FP8 mixed precision (first validated at this scale)

[*] 180K H800 GPU-hours per trillion tokens
[*] No unrecoverable loss spikes or rollbacks during the entire run
[*] DualPipe and cross-node MoE overlap hide nearly all expert-routing communication behind compute

Key Innovations

Auxiliary-loss-free load balancing

Expert load is balanced by adjusting per-expert bias terms instead of an auxiliary loss, avoiding the quality hit that loss usually causes.

Multi-Token Prediction

Trained to predict several future tokens at once. The extra prediction heads double as a speculative decoder at inference time.

FP8 mixed-precision training

First public validation that FP8 training works at frontier scale, cutting memory bandwidth and interconnect pressure.

DualPipe + MoE comm overlap

Custom pipeline parallelism that hides nearly all cross-node expert-routing communication behind compute.

Why It Matters

[*] DeepSeek R1 (January 2025) is a reinforcement-learning fine-tune of this base model. Its launch was called "AI's Sputnik moment" and knocked roughly $600B off Nvidia's market cap in one session, a record single-day loss for a US company.

[*] V3 showed a frontier-class model could be trained for about $5.6M on export-restricted H800 chips, far below assumed frontier training costs, and reset expectations across the industry.

[*] The 0324 revision (March 2025) moved the weights to the MIT license.

sources:tech report HuggingFace theverge.com techjacksolutions.com

Model ID: deepseek-v3-0324

The Lab

location: Hangzhou, Chinafounded: 2023backed by: High-Flyer Quant

more about DeepSeek →

Interactive Architecture

Take DeepSeek V3 (0324) apart, layer by layer. The 3D teardown below shows its 61 transformer layers (3 dense + 58 MoE), MLA attention, and expert routing, with values from the public model config.

61 layers (3 dense → 58 MoE)

MLA attention · 128Q heads

8+1 of 256 experts active

671B params · 37B active

Token embedding

Each input token id is mapped to a 7168-dim vector drawn from a 129,280-token vocabulary.

61 transformer layers

Multi-head Latent Attention

Instead of caching 128 full KV heads, MLA compresses keys and values into a shared 512-dim latent vector, keeping the KV cache small enough for a 128K+ context. RoPE adds position information.

Mixture of Experts FFN

A learned router scores all 256 routed experts for every token, then sends it to just 8 of them plus 1 always-on shared expert. The model stores 671B parameters yet spends only 37B per token.

Output head

After a final RMSNorm, the hidden state is projected back to 129,280 vocabulary logits to predict the next token.

What vocabulary and embedding size does DeepSeek V3 (0324) use?+

Each input token id is mapped to a 7168-dim vector drawn from a 129,280-token vocabulary.

How many transformer layers does DeepSeek V3 (0324) have?+

What is Multi-head Latent Attention (MLA) in DeepSeek V3 (0324)?+

Instead of caching 128 full KV heads, MLA compresses keys and values into a shared 512-dim latent vector, keeping the KV cache small enough for a 128K+ context. RoPE adds position information.

How does the Mixture of Experts in DeepSeek V3 (0324) work?+

A learned router scores all 256 routed experts for every token, then sends it to just 8 of them plus 1 always-on shared expert. The model stores 671B parameters yet spends only 37B per token.

How does DeepSeek V3 (0324) produce output tokens?+

After a final RMSNorm, the hidden state is projected back to 129,280 vocabulary logits to predict the next token.

Training Run

Tokens

14.8T

Hardware

2048 × NVIDIA H800

GPU Hours

2.788M H800 GPU-hours

Est. Cost

~$5.576M (final run, at $2/GPU-hour)

Duration

< 2 months pre-training

Precision

FP8 mixed precision (first validated at this scale)

[*] 180K H800 GPU-hours per trillion tokens

[*] No unrecoverable loss spikes or rollbacks during the entire run

[*] DualPipe and cross-node MoE overlap hide nearly all expert-routing communication behind compute

Key Innovations

Auxiliary-loss-free load balancing

Expert load is balanced by adjusting per-expert bias terms instead of an auxiliary loss, avoiding the quality hit that loss usually causes.

Multi-Token Prediction

Trained to predict several future tokens at once. The extra prediction heads double as a speculative decoder at inference time.

FP8 mixed-precision training

First public validation that FP8 training works at frontier scale, cutting memory bandwidth and interconnect pressure.

DualPipe + MoE comm overlap

Custom pipeline parallelism that hides nearly all cross-node expert-routing communication behind compute.

Why It Matters

[*] V3 showed a frontier-class model could be trained for about $5.6M on export-restricted H800 chips, far below assumed frontier training costs, and reset expectations across the industry.

[*] The 0324 revision (March 2025) moved the weights to the MIT license.

sources:tech report HuggingFace theverge.com techjacksolutions.com