Loading...
Loading...
Free 550B (55B active MoE) flagship reasoning model with large context
[interactive architecture ↓]The AI research division of NVIDIA, the Santa Clara-based computing giant founded in 1993. NVIDIA AI Research develops foundation models, simulation platforms, and synthetic data generation tools that run on NVIDIA hardware, spanning LLMs, robotics world models, and drug discovery. The Nemotron family targets enterprise language tasks, while Cosmos builds generative world models for robotics and autonomous systems.
more about NVIDIA AI Research →Take Nemotron 3 Ultra apart, layer by layer. The 3D teardown below shows its 108 transformer layers (0 dense + 108 MoE), GQA attention, and expert routing, with values from the public model config.
8192-dim vectors over a 131,072-token vocabulary. The wide hidden dim supports the large expert capacity below.
A deep stack that interleaves Mamba2 SSM blocks with attention + MoE blocks. The SSM layers give linear-time state mixing over the 1M context; the attention layers do precise retrieval. (The scene renders the full stack as transformer slabs.)
64 query heads share just 2 KV heads via GQA, an aggressive KV compression that complements the Mamba layers. Used on the attention subset of the hybrid stack.
A router scores 512 experts per token and activates 22 plus a shared expert, spending 55B of 550B per token. The very high expert count (and 108 layers) is what reaches the 550B total.
Final norm then projection to 131,072 logits. Nemotron 3 Ultra is tuned for agentic reasoning over million-token contexts.