World Wire

Open Mixture-of-Experts (MoE) models share one idea but disagree on almost every routing detail. Expert counts range from 8 to 384. Top-k ranges from 1 to 8. Some models keep an always-on shared expert; others drop it. TL;DR Size: 10 models, from Qwen3.6-35B-A3B (35B total, 3B active) to DeepSeek-V4-Pro (1.6T total, 49B active). Runs on: gpt-oss-120b fits one 80 GB GPU with MXFP4; DeepSeek-V4-Pro and Kimi K2 need multi-GPU expert parallelism. Performance: Routed sparsity spans 4x in Mixtral (8 experts, top-2) to 128x in Llama 4 Maverick (128, top-1). Best: Kimi K2 found sparsity 48 needed 1.69x fewer FLOPs than sparsity 8 at equal validation loss. Worst: Llama 4 Maverick and gpt-oss do not publish their gating or balancing method in config.json. Bottom line: Best thing: configs make routing fully auditable. Worst thing: names like “17B active” hide very different routers. How do open MoE models route tokens? Every model below uses...

Every Mixture-of-Experts (MoE) model repeats one decision for every token: which experts should process it. That decision belongs to the router. MoE routing strategies shape quality, load balance and serving cost in every major open model. This guide explains the 7 routing strategies used in production, and the models that use each one. TL;DR Size: Routers choose from 8 experts (Mixtral) to 384 (DeepSeek-V4-Pro), activating 1 to 8 per token. Runs on: Routing cost shows up in multi-GPU expert parallelism, where each choice becomes all-to-all network traffic. Performance: Sigmoid scoring with aux-loss-free bias balancing is the dominant 2026 recipe for frontier open MoE models. Best: Loss-free balancing cut MaxVio from 0.72 to 0.04 at 1B, with lower perplexity. Worst: Expert choice leaks future tokens in causal models; one test showed a roughly 10% abnormal loss drop. Bottom line: Best thing: bias-based balancing adds no interfering gradients. Worst th...

Nous Research has released Arete-27B , a supervised fine-tune of Alibaba’s Qwen3.8-27B . Arete-27B is a 27B open-weight model that strips Chinese state framing from Qwen’s answers. On Nous’s own 803-prompt eval, answers containing propaganda fell from 22% to 0.9%. Developers get Qwen-class agentic skill without state-aligned answers. The weights, eval and training data are all public. TL;DR Size: 27B dense parameters (all active). Trained at 64k context. Base Qwen3.8-27B supports 262,144 tokens natively. Runs on: BF16 weights are 53.81 GB. A community Q4_K_M quant is 17.20 GB, sized for 24 GB consumer GPUs. Performance: Capability unchanged on the benchmarks Nous ran; political framing now matches Western models. Best: Propaganda rate 0.9%, down from 22% for base Qwen3.8-27B. Worst: Fact score 2.98 of 5; 216 of 803 answers still score 0. What is Arete-27B? Arete-27B is Qwen3.8-27B after supervised fine-tuning only . Nous used no...

In this tutorial, we work with Dory 1.0 4B , a small reasoning model from NVIDIA’s Nemotron Labs whose middle layers form a shared block that can be applied once, twice, three, or four times for every token. Each extra pass, or loop, spends more computation per token without adding a single parameter, so one checkpoint can run at four different compute budgets with no retraining. We build the llama.cpp branch that NVIDIA’s model card pins, download the 3.7 GB GGUF, and read the architecture straight from the file to predict what each loop should cost in parameters and cache memory. We then check those predictions against the running server, measure decode speed at every loop count, test whether the cheaper last-loop cache changes the model’s predictions, and run a paired GSM8K evaluation to see how accuracy, answer length, and time per solved question move as the loops increase. Copy Code Copied Use a different Browser import os import re import sys import json im...