Model selection for mixtures of experts via the slope heuristic
How many experts does the data support? Fit a Gaussian-gated mixture of experts for every candidate number of experts K, then let the slope heuristic (Birge-Massart) choose. The method knows the penalty shape (proportional to the model dimension) but not its multiplicative constant κ. Slide κ and watch the selected model walk down a staircase: over-fit (large K) → correct (K₀) → under-fit (small K). The dimension jump reads the right constant straight off the data, and the fitted conditional-mean curve on the left changes in lockstep.
The covariate x lives in an interval and the response follows a Gaussian-gated mixture of experts: for each expert k a Gaussian gate localizes it near a center γk, and inside its region the response is linear, y = akx + bk + noise. For each candidate K = 1..8 we fit this model by EM (a few random restarts) and record its maximized log-likelihood and its dimension D(K) = 5K free parameters. The paper proves a non-asymptotic oracle inequality for the penalized maximum-likelihood estimator under the Jensen-Kullback-Leibler loss: as long as the penalty is at least a known shape times a large-enough constant, the selected model performs as well as the best (oracle) model. The penalty shape is known only up to a multiplicative constant κ, so we select K by minimizing −logLik(K) + κ·D(K)/n. The slope heuristic reads κ off the data: as κ grows the selected dimension collapses in a staircase, and the location κ̂ of its biggest drop (the dimension jump) is the minimal penalty. The theory then prescribes κ* = 2κ̂, which here lands near the true K₀. AIC (κ = 1) and BIC (κ = ½ log n) are marked as fixed ticks. (This is a 1D toy: the real paper handles high-dimensional block-diagonal-covariance localized experts with covariates and responses growing with n, and its guarantee is the asymptotic-free oracle inequality, not the finite staircase you drag here. In-browser EM uses few restarts, so a candidate fit can occasionally miss its global optimum.)
Run the experiments
Every animation runs live in your browser. Click a button to run that experiment on the demo (it scrolls up and starts); drag any control to take over. Nothing is downloaded, it is generated on the fly.
Drag the penalty constant
Sweep the penalty constant κ and watch the selected model collapse down a staircase: over-fit (large K) to correct (K0) to under-fit, with the fitted conditional-mean curve on the left changing in lockstep. The amber dimension jump and the teal prescribed 2·κ-hat mark where the slope heuristic lands.
Vary the true number of experts
Change the true number of experts K0 in the well-specified generating model and refit. For K0 = 3, 4, 5 the prescribed penalty 2·κ-hat recovers K0 exactly; larger K0 needs more data.
Grow the sample size
Increase the sample size n and refit every candidate model by EM. More data sharpens the dimension jump and makes the recovered model track the true K0 more reliably.
Turn up the expert noise
Raise the expert noise standard deviation. As the experts blur together the log-likelihood gains from extra experts shrink, so the slope heuristic favors a more parsimonious model.
Well-specified vs misspecified
Toggle between data from a true K0-expert mixture (the slope heuristic recovers K0) and a smooth non-mixture curve with no true finite K (it selects the best parsimonious approximation, the oracle model).
Draw fresh samples
Redraw the sample from the same generating model and refit all candidate K. The dimension jump and selected model shift slightly with sampling noise but stay centered on the oracle.
The idea in three steps
Every candidate K
Fit a Gaussian-gated mixture of experts for K = 1..8 by penalized maximum likelihood (EM). Each fit gives a log-likelihood and a known dimension D(K), the two ingredients the penalty needs.
The dimension jump
The penalty shape is known, its constant κ is not. Sweep κ: the selected dimension falls in a staircase, and the κ̂ where it drops hardest is the minimal penalty the oracle inequality allows.
Twice the jump
The slope heuristic prescribes κ* = 2κ̂. The model chosen at κ* matches the oracle: it recovers K₀ when the family is well-specified, and the best parsimonious approximation when it is not.
For the weak oracle inequalities, the Jensen-Kullback-Leibler loss, the block-diagonal-covariance localized experts and the high-dimensional theory, see A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models (TrungTin Nguyen, Ho, Nguyen, Chamroukhi, Forbes, Electronic Journal of Statistics).