Convergence Rates for Parameter Estimation in Mixtures of Experts
How fast do the estimated experts approach the true ones as data accumulate? If you fit the right number of experts, the parameters converge at the fast parametric rate √N. But you rarely know the true number, and the moment you fit too many experts the rate collapses to something far slower, set by the interaction between the gating and the experts. Run a real simulation and read the rate off a log–log plot.
Each of these mixture-of-experts models has a known true number of experts K0. Data are simulated and the model is fit many times at growing sample sizes; the curve is the distance from the estimated mixing measure (the experts and their weights) to the truth in Wasserstein distance, the quantity the theory controls. Fitting the exact number of experts gives the fast rate √N (rate ≈ K−1/2, teal). Fitting too many experts, the realistic case since K0 is unknown, makes the parameter error decay far more slowly (amber): the surplus experts cannot vanish, they split their mass and hover near a true expert (see the extra amber stem on the left), and how slowly they converge is governed by an algebraic interaction between the gating and the expert functions, a system of polynomial / differential equations whose solvability sets the exponent. Add a second surplus expert (K0+2) and the rate slows again. Switch between the three models from the papers, softmax-gated Gaussian (Demystifying Softmax Gating, NeurIPS 2023), Gaussian-gated (Towards Convergence Rates, AISTATS 2024) and softmax-gated multinomial logistic (A General Theory, ICML 2024), plus the plain location mixture: the over-specification collapse is shared by all, while the exact exponent is model-specific. The key subtlety the theory resolves: even with badly estimated parameters, the fitted density still converges at √N, so the model fits the data well while recovering the wrong experts, which is why identifiability and rate analysis matter. (The curves are real Monte-Carlo results, 60 replicates per point, precomputed with each paper's model and estimator; empirical slopes approach the theoretical exponents as N and the replicate count grow. Repositories: CRPE-SGaME, CRPE-GMoE, Extension-SGMLMoE.)
Run the experiments
Every animation runs live in your browser. Click a button to play that comparison on the demo (it scrolls up and starts). Nothing is downloaded.
The four MoE models
Tour the softmax-gated Gaussian, Gaussian-gated, softmax-gated logistic and location-mixture models; each contrasts the exact-fit sqrt(n) rate against the collapsed over-specified rate.
Over-specification K₀+1 vs K₀+2
Flip between one and two surplus experts and watch the estimation rate degrade further.
The idea in three steps
A mixture of experts can fit almost anything, but that flexibility hides a statistical cost: recovering which experts generated the data is much harder than fitting the data.
The fast rate
With the right number of experts, the estimates are √N-consistent: the parameter error halves for roughly every quadrupling of the sample size.
The rate collapses
Fit even one extra expert and the surplus components cannot disappear. Their interaction with the gating drives a much slower, gating-dependent rate.
Well fit, wrongly identified
The predictive density still converges fast, so a good fit is no guarantee the experts are right, the reason these rates are worth knowing.
For the exact exponents, the Voronoi-loss analysis and the role of the gating function, see the convergence-rate papers on softmax-gated and Gaussian-gated mixtures of experts.