Loading...
Loading...
Multimodal MoE (12B of 120B parameters active) that reads text, images, documents, GUI screenshots and video over a 1M context
Take Nemotron 3.5 Super VL apart, layer by layer. The 3D teardown below shows its 88 blocks (40 Mamba-2, 40 MoE, 8 GQA), GQA attention, and expert routing, with values from the public model config.
NVIDIA Research is the research arm of NVIDIA Corporation, the Santa Clara-based computing company founded in 1993; the research team itself was founded in 2006 by chief scientist David Kirk and now spans more than 200 scientists working on AI, graphics, high-performance computing and autonomous vehicles. Its Nemotron line of open models restarted its numbering in 2025: Nemotron-4 (2024) was followed by Nemotron Nano 2 (2025) and then the Nemotron 3 family of Nano, Super and Ultra models, all released with open weights.
more about NVIDIA Research →4096-dim vectors over a 131,072-token vocabulary. A C-RADIO vision encoder turns images, documents, GUI screenshots and video frames into tokens that join this stream (not drawn).
The Nemotron 3 Super backbone: 40 Mamba-2 blocks and 40 LatentMoE blocks interleaved, with just 8 attention blocks. Mamba-2 keeps a fixed-size state, so most of the stack adds nothing to the cache as context grows to 1M tokens. (The scene renders every block as a transformer slab.)
32 query heads share 2 KV heads in each of the 8 attention blocks, caching about 8 KiB per token at bf16 across the whole model.
Each token is projected down into a 1024-dim latent, where a router picks 22 of 512 experts. One shared expert runs at the full 4096-dim width alongside them. About 12B of 120B parameters fire per token.
Final RMSNorm then 131,072 logits. A shared-weight multi-token predictor (one attention block plus one MoE block) drafts extra tokens; it is not drawn.