Loading...
Loading...
Small open MoE (3B of 30B parameters active) for high-throughput agent workloads and specialised tasks
Take Nemotron 3.5 Lightning apart, layer by layer. The 3D teardown below shows its 52 blocks (23 Mamba-2, 23 MoE, 6 GQA), GQA attention, and expert routing, with values from the public model config.
NVIDIA Research is the research arm of NVIDIA Corporation, the Santa Clara-based computing company founded in 1993; the research team itself was founded in 2006 by chief scientist David Kirk and now spans more than 200 scientists working on AI, graphics, high-performance computing and autonomous vehicles. Its Nemotron line of open models restarted its numbering in 2025: Nemotron-4 (2024) was followed by Nemotron Nano 2 (2025) and then the Nemotron 3 family of Nano, Super and Ultra models, all released with open weights.
more about NVIDIA Research →2688-dim vectors over a 131,072-token vocabulary, the same tokenizer as the rest of the Nemotron 3 line.
The Nemotron 3 Nano backbone: 23 Mamba-2 blocks and 23 MoE blocks interleaved, with only 6 attention blocks. Mamba-2 keeps a fixed-size state instead of a growing cache, which is how the context reaches 1M tokens. (The scene renders every block as a transformer slab.)
32 query heads share 2 KV heads in each of the 6 attention blocks. They handle precise lookups while Mamba-2 does the bulk of the sequence mixing; together they cache about 6 KiB per token at bf16.
A router picks 6 of 128 experts per token, plus 1 shared expert, each with a 1856-dim hidden layer and Squared ReLU. About 3B of the 30B parameters are active per token.
Final RMSNorm then 131,072 logits. A shared multi-token predictor (one attention block plus one MoE block) drafts extra tokens for faster decoding; it is not drawn.