CompeteSMoE: Effective Training of Sparse Mixture of Experts via Competition
A sparse mixture of experts routes each token to a few of N experts through a router, a cheap gate scored on a detached embedding dot-product x⊤Wr that has no idea what the experts will actually do. That indirection makes routers slow to learn and prone to collapse. CompeteSMoE removes the middle-man during training: every expert computes its output, each expert's own neural response is scored, and tokens go to the strongest responders (winner-take-all), exactly like neurons competing. A cheap linear router is then distilled to predict the competition winner, so at inference only the router runs, same cost as vanilla SMoE. Train a standard router and CompeteSMoE side by side and watch the distilled router learn the competition winner while the standard one collapses.
Tokens are 2D points drawn from N Gaussian clusters on a ring; the toy self-supervised task is to reconstruct each token as a gated blend of its top-k experts (each expert is a tiny 2-layer FFN gi(x) = Vi·leakyrelu(Kix+bi)), a 2D proxy for the language-model loss LNLL. Both panels share the same tokens, the same target, and the same initial experts and router; only the training rule differs, isolating "competition as the routing target" as the single variable. Left (standard SMoE): the router scores experts by x⊤Wr, takes a softmax top-k, and trains the router and experts together through the task loss only. Under joint SGD a rich-get-richer feedback lets a few experts win most tokens: the decision regions collapse, the load piles onto 1–2 experts (the load-concentration Herfindahl index stays high, 1/N is balanced), and surplus experts go dead. Right (CompeteSMoE): every expert computes its output, its response si = mean softplus(gi(x)) is scored, and the top-k strongest responders win (weights are the L1-normalized responses, not a softmax). The same linear router is trained by MSE(softmax(gate), competition‑softmax.detach()) with an extra α term on the winners, so it is supervised to predict the competition outcome rather than left to discover routing through the task gradient alone. The agreement curve (right, top) is the fraction of tokens where the router's argmax equals the competition winner: CompeteSMoE's climbs (the lower expert-change-rate / faster router convergence story), the standard router lags. A diversity loss keeps the winners' outputs apart, and an adaptive balance bias keeps every expert fed. Drag ω down to the paper's ~0.07 regime (competition on only a few steps) or turn Competition off to collapse the right panel back to plain SMoE (ablation). (This is a 2D didactic version of the exact mechanism, response = mean-softplus of each expert's output, winner-take-all top-k, and MSE distillation of a linear router onto the detached competition softmax, with the same load-balance and diversity regularizers. The real paper sparse-upcycles a Phi-3.5-mini + SigLIP vision-language model to 5.1B params, 4 experts top-2, and a 151M decoder-only LM, 64 experts top-8, and reports the best average across 11 VLM and 7 language benchmarks (VLM avg 50.16, LM PPL 13.66) with only ~3% training overhead and identical inference cost to SMoE, since at inference only the distilled router runs. Theoretically, in an over-specified Gaussian MoE competition reaches an exactly-fit expert in O(ε-4) samples versus O(ε-12) for a softmax-gated MoE, the sample-efficiency claim this demo can only illustrate qualitatively, through the climbing agreement / balanced load, not measure. All N experts always compute here, so the training-cost scheduler is illustrative rather than a performance necessity. Preprint arXiv 2402.02526, with TrungTin Nguyen among the co-authors.)
Run the experiments
Every animation runs live in your browser. Click a button to play that experiment on the demo (it scrolls up and starts); drag any control to take over. Nothing is downloaded.
Train (competition routing)
Run the live SGD: the distilled router learns the competition winner while the standard router collapses.
Competition schedule ω
Sweep how often competition is scheduled; the paper's best is a small fraction of steps.
Distillation winner-weight α
Sweep how much the router distillation up-weights the winning experts.
Top-k experts
Route each token to more or fewer winners.
Number of experts N
More experts, more room to specialise.
The idea in three steps
A sparse mixture of experts is only as good as its router. Scoring experts by a detached embedding is indirect and unstable; letting the experts compete, then distilling the winner, is both more stable and provably more sample-efficient.
The router is a guess
Standard SMoE routes on x⊤Wr, an embedding detached from what the experts actually compute. It learns slowly through the task gradient of the few selected experts, over-commits to a handful, and collapses, so training leans on a load-balancing crutch.
Let experts compete, then distill
Score every expert by its own neural response mean softplus(gi(x)) and route to the strongest responders. A cheap linear router is regressed (MSE) onto that competition outcome, so it learns to predict the winner. At inference only the router runs, same cost as vanilla SMoE.
Balanced, faster, guaranteed
The router converges faster (lower expert-change-rate), the load stays balanced, and, in an over-specified Gaussian MoE, competition needs O(ε-4) samples where softmax gating needs O(ε-12). CompeteSMoE tops 11 VLM and 7 LM benchmarks at ~3% training overhead and identical inference cost.
For the full method, the statistical theory, the sparse-upcycled Phi-3.5 + SigLIP VLM and 151M SlimPajama LM experiments, and the comparisons against SMoE, XMoE, PCosine and shared-expert baselines, see CompeteSMoE: Effective Training of Sparse Mixture of Experts via Competition and the code on GitHub.