HyperRouter: Efficient Sparse Mixture of Experts via a Frozen HyperNetwork
A sparse mixture of experts routes each token to a few of N experts, but a freely trainable router, optimized jointly with the experts, tends to collapse: it over-uses a handful of experts, the load imbalances, and the experts drift toward each other (representation collapse), so practitioners bolt on a load-balancing auxiliary loss as a crutch. HyperRouter instead generates the router weights from a frozen random hypernetwork applied to a tiny trainable embedding e: a 2D code drives the whole gate through a fixed nonlinear map that keeps experts diverse and the load balanced without any auxiliary loss. Train both routers side by side and watch the standard one collapse.
Tokens are 2D points drawn from a few Gaussian clusters; the toy self-supervised task is to reconstruct each token as a gated blend of its top-k experts (here the experts are learnable 2D vectors, so this is a soft vector quantization). Both panels share the same tokens, the same target and the same initial experts; only the router differs. Left (standard): the router weight matrix Wr is a free trainable [N×2] matrix plus bias. Under joint SGD without a balancing loss, a rich-get-richer feedback makes a few experts win most tokens: the decision regions collapse (a couple of colours swallow the plane), the load piles onto 1–2 experts, and the surplus experts stay dead at their initialization, so the collapse meter (the load concentration, a Herfindahl index over the experts, where 1/N is perfectly balanced) stays high. Turn on the load-balancing aux and the standard side rebalances, the crutch it needs. Right (HyperRouter): the router is R(h) = TopK(softmax(H(e) h), k), where H is a frozen, randomly initialized 2-layer MLP (R2 → ReLU → RN·2+N) and only the embedding e ∈ R2 is trained; each step Wfull = H(e) is reshaped to [N, d+1] to give the router rows and biases. Because the fixed random map keeps the N rows spread apart, every expert keeps receiving tokens, so the load stays balanced and no expert goes dead (the collapse meter stays low) with no auxiliary loss. Drag e₁, e₂ to see a 2D code reshape the entire gate, and lower top-k for the inference-efficiency regime. (This is a 2D didactic version of the exact mechanism: a frozen hypernetwork generates the router weights from a trainable embedding. The real paper trains Transformer-XL language models on enwik8, text8 and WikiText-103 with 16 experts, reporting large gains especially at small k, much lower bits-per-character and perplexity than SMoE and SMoE-Dropout, and matching baselines with roughly half the experts at inference. In this low-dimensional reduction a 2-parameter frozen router is far less expressive than a free one, so the fixed H is drawn to be well-conditioned and the standard router often reaches a lower training loss precisely by collapsing onto a few experts, the very pathology the frozen router avoids. Published at EMNLP 2023, with TrungTin Nguyen among the co-authors.)
Run the experiments
Every animation runs live in your browser. Click a button to play that experiment on the demo (it scrolls up and starts); drag any control to take over. Nothing is downloaded.
Train both routers (SGD)
Run the live SGD: the standard router collapses onto a few experts while the frozen-hypernetwork router stays balanced.
Trainable embedding e₁
Drag the first embedding coordinate and watch the frozen hypernetwork reshape the whole gate.
Trainable embedding e₂
Drag the second embedding coordinate; a 2D code drives the entire router.
Top-k experts
Route each token to more or fewer experts.
Number of experts N
More experts leaves more room for the standard router to collapse.
The idea in three steps
A sparse mixture of experts is only as good as its router. Left free, the router quietly collapses; a frozen random map turns that instability into a structured regularizer.
Routers collapse
A freely trainable router over-commits to a few experts. The load imbalances, unused experts go dead, and expert representations drift together, so training leans on a load-balancing auxiliary loss.
Freeze a random hypernetwork
Re-parameterize the router as Wr = H(e): a frozen random network H applied to a tiny trainable embedding e. Only e (and the experts) train; H never moves. The fixed map keeps the expert routing directions diverse.
Balanced, cheaper, no crutch
Experts stay diverse and the load stays balanced with no auxiliary loss, and the model matches strong baselines at much smaller k, the inference-efficiency win.
For the full method, the Transformer-XL experiments on enwik8 / text8 / WikiText-103, and the comparisons against SMoE and SMoE-Dropout, see HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts via HyperNetwork (EMNLP 2023) and the code on GitHub.