Mixture of Experts & Dendrogram Merging
One over-fitted mixture of experts: the gating clusters the inputs (point colour), each expert fits a local regression (mean curve + band), and the number of components is read off a dendrogram of mixing measures with no model sweeps. The data is generated from the selected gate's own mechanism (softmax-gated SGMoE, or Gaussian-gated), so each gate is fit well-specified; the sliders set the true number of experts K0, the sample size N, and the ε-contamination. The data starts ε-contaminated (Laplace outliers): the classical likelihood criteria AIC / BIC / ICL over-select, while the consistent dendrogram selection criterion (DSC, gold line) holds near the true count; drag ε to 0 for the clean case. You can also compare the paper's rate-weighted merge against average-linkage agglomerative clustering.
Fit the mixture of experts once at Kmax experts, then drag the cut down the dendrogram: redundant experts merge, the effective count K̂ falls, and the fit barely moves (consistency without model sweeps). The gold DSC line marks the paper's consistent selector: it maximizes height(κ) + εN ℓ̄(κ), combining the merge height with the model's average log-likelihood (shown here scale-normalized, since the in-browser dissimilarities are illustrative, not on the paper's loss scale); the draggable teal line is for manual exploration. Switch the gating (Gaussian-gated, or the softmax-gated SGMoE) and the merge method (the paper's rate-weighted dendrogram of mixing measures, or average-linkage agglomerative clustering). The demo ships ε-contaminated (a fraction of Laplace outliers). The classical criteria are the textbook ones: refit the K-component model for every K and score its maximized likelihood, so they are identical whichever dendrogram you build (the merge method only changes the tree, hence the DSC). AIC over-selects readily (its weak penalty rewards the extra experts that fit the outliers); BIC and ICL are more conservative but still exceed the truth; only the height-robust DSC holds K̂ near K0. Drag ε to 0 for the clean case. The curve is the mean E[y|x] of the current merged model (the experts are fit once and never refit; dragging the cut only aggregates them), so it barely moves as redundant experts merge and flattens only when you over-merge past K0; the band is ± 1.64 times the empirical residual spread; both gatings fit tightly (the softmax path uses the paper's IRLS-EM). Drag the K0 slider and the Gaussian DSC tracks it. On clean data (set ε to 0), turning Refit-DSC on (a few warm-started EM steps at each cut, giving the proper K-component likelihood the frozen barycenter lacks at small N) also lets the softmax selector recover K0 = 3, 4; with Refit-DSC off the paper's frozen-merge path collapses to 2. Note the trade-off: the refit leans on the likelihood, so under ε-contamination the height-robust (Refit-DSC off) default gives the cleaner AIC/BIC/ICL contrast. This needs enough data, so raise Samples N (the fit runs several EM restarts, so it is slower). It is the theory-consistent fix: the paper's consistency assumes the aggregated-measure likelihood has converged, which the warm-started refit supplies.
Run the experiments
Every animation here is generated live in your browser by the interactive demo above, there are no video files. Click a button to run that experiment on the demo (the page scrolls up and it starts playing); drag any control to take over manually. You can also use the ▸ button beside each slider, or Run merging, directly on the demo.
The merging procedure
Drag the cut down the dendrogram: redundant experts merge and the effective count K̂ falls from 10 experts down to 2.
ε-contamination robustness
As the outlier fraction grows, AIC / BIC / ICL over-select, while the consistent dendrogram criterion (DSC) holds K̂ near the true count.
True number of experts K₀
The data-generating number of experts changes; the fit, the clustering and the mixing-measure dendrogram track it.
Sample size N
More data sharpens both the regression fit and the model selection read off the dendrogram.
Over-fit K
More fitted experts (deliberate over-specification), yet the dendrogram still merges back toward the truth.
Noise level
Heavier observation noise widens the predictive bands and blurs the experts' separation.
The idea in three steps
Choosing the number of mixture-of-experts components usually means refitting the model for every candidate value and scoring each fit. My work reads the answer off a single over-fitted model instead.
Deliberately use too many experts
Fit a softmax/location-gated Gaussian mixture of experts with a large Kmax. Several experts end up explaining the same region, the fit is good but redundant.
Cluster the mixing measure
Each expert is an atom of the mixing measure. Agglomerating atoms by their distance in parameter space yields a dendrogram whose merge heights reveal which experts are really the same.
Read off the components
Cutting the dendrogram merges redundant atoms into one, giving a consistent estimate K̂ of the true number of components, without ever refitting the model.
The softmax gating runs a faithful in-browser port of the paper's own estimator: EM with an IRLS (Newton-Raphson) gating M-step and weighted-least-squares experts, then the exact rate-weighted dendrogram of mixing measures (divclus dissimilarity + softmax-weighted merge). For the theory, the full estimator and the experiments see Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts.