NVIDIA Releases Nemotron-Labs-3-Puzzle-75B-A9B: A Compressed Hybrid MoE LLM Delivering 2.03x Server Throughput at Matched User Throughput
Large hybrid MoE models like Nemotron-3-Super are accurate but expensive to serve. Their active parameters, KV cache, and Mamba state cap how many users a node can hold at a given per-user token rate. NVIDIA AI team has released Nemotron-Labs-3-Puzzle-75B-A9B , a compressed variant of Nemotron-3-Super. The parent model has 120.7B total and 12.8B active parameters. The compressed model has 75.3B total and 9.3B active parameters. The deployment target was fixed before the architecture search began. Target one was 2x server throughput at 100 tokens per second per user. Target two was 8 concurrent 1M-token requests on a single H100. Three checkpoints on Hugging Face: BF16, FP8, and NVFP4. TL;DR 120.7B/12.8B active compresses to 75.3B/9.3B active, with the 88-block hybrid layout preserved. 8xB200 total throughput rises 1.60x to 2.14x over Super at matched NVFP4 and matched user throughput. Single-H100 1M-token concurrency goes 1 to 8, driven by a 70 GB to 44.5 GB weigh...
