Loading...
Loading...
Native multimodal Flash model for efficient coding and agent work; hybrid sparse and linear attention keeps long-context recall accurate at low cost
Take GLM-5.3 Flash apart, layer by layer. The 3D teardown below shows its 45 layers (3 dense + 42 MoE), MLA attention, and expert routing, with values from the public model config.
Founded in 2019 as a Tsinghua University spinout, the company was known internationally as Zhipu AI until it rebranded as Z.ai in July 2025; it is headquartered in Beijing and has been listed on the Hong Kong Stock Exchange since January 2026. The lab develops the GLM (General Language Model) family, released under the MIT licence since GLM-4.5 in July 2025 and focused on coding and agentic tasks, alongside the GLM-V vision line and the AutoGLM phone agent.
more about Z.ai (Zhipu AI) →4096-dim vectors over a 154,880-token vocabulary shared with GLM-5. The model is natively multimodal: image and video tokens join this stream.
Three Kimi Delta Attention layers for every MLA layer (34 + 11). Residuals run through four parallel mHC streams that are mixed at each sublayer instead of one running sum. The first 3 layers use a dense FFN; the other 42 are MoE.
KDA is linear attention with a fixed-size state and a short convolution, so most layers cache nothing per token. Every fourth layer is MLA with a 512-dim KV latent and no rotary part, and a DeepSeek Sparse Attention indexer (32 heads, top-2048) picks which earlier tokens it reads. Only those 11 layers cache, about 11 KiB per token at bf16.
A sigmoid router picks 8 of 288 experts per token, plus 1 shared expert, each 2048 dims wide. About 18B of the 320B parameters fire per token.
The four mHC streams are mixed back to one, then RMSNorm and an untied projection to 154,880 logits. One MTP layer drafts an extra token; it is not drawn.