Tencent Releases Hy3: An Open 295B Mixture-of-Experts (MoE) Model with 21B Active Parameters and 256K Context
Tencent’s Hy team released Hy3 . Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model. It activates only 21B parameters per token. The weights ship under the Apache License 2.0 . Hy3 is aimed at reasoning, agentic workflows, and long-context tasks. What is Hy3? Hy3’s architecture contains a sparse MoE with 192 experts and top-8 routing. Only 8 experts fire per token, so compute stays low. The model also uses a Multi-Token Prediction (MTP) layer. MTP predicts several tokens at once for faster decoding. Both vLLM and SGLang enable it through speculative decoding. Property Value Architecture Mixture-of-Experts (MoE) Total parameters 295B Activated parameters 21B MTP layer parameters 3.8B Layers (excluding MTP) 80 MTP layers 1 Attention heads 64 (GQA, 8 KV heads, head dim 128) Hidden size 4096 Intermediate size 13312 Context length 256K Vocabulary size 120832 Experts 192 experts, top-8 activated Supported precisions BF16 A separate Hy3-FP8 checkpoint is ...
