Loading...
Loading...
Preview of Tencent's next MoE (49B of 770B parameters active), built for coding agents, complex tool use and productivity work
Take Hunyuan Hy4 (Preview) apart, layer by layer. The 3D teardown below shows its 78 layers (1 dense + 77 MoE), MLA attention, and expert routing, with values from the public model config.
Hunyuan is the foundation-model brand of Tencent Holdings, the Shenzhen company founded in 1998; the Hunyuan large language model was launched on 7 September 2023 at Tencent's Global Digital Ecosystem Summit and is served through Tencent Cloud and products such as WeChat. Most recent releases are open weights: Hunyuan-Large (November 2024), HunyuanVideo (December 2024), the Hunyuan3D 2.x series (2025), HunyuanImage 2.1 and 3.0 (September 2025) and the 295B Hy3 language model (July 2026, Apache 2.0). Hunyuan 3D 3.0, announced in September 2025, is served through Tencent's Hunyuan 3D platform and API.
more about Tencent Hunyuan →6144-dim vectors over the same 120,832-token vocabulary as Hy3.
Every layer is gated MLA with sparse attention. Identity Hyper-Connections keep four parallel residual streams and mix them around each sublayer. Only layer 0 uses a dense FFN; the other 77 are MoE.
Keys and values are compressed into a 512-dim latent, and an elementwise gate plus a learnable sink shape each head’s output. A 32-head indexer picks the top 2,048 earlier tokens to attend to; it runs in full on 21 layers and its picks are reused by the next layers in the other 57, which makes the 1M context cheaper. The latent cache is about 87.8 KiB per token at bf16.
A router picks 8 of 256 experts per token, plus 1 shared expert, each 2048 dims wide. About 49B of the 770B backbone fires per token.
The four residual streams are mixed back to one, then RMSNorm and an untied projection to 120,832 logits. A 10B multi-token-prediction layer drafts extra tokens; it is not drawn.