An overfitted Bayesian mixture of experts, summarised by merge-truncate-merge; the posterior over the number of experts is the Bayesian analogue of the dendrogram cut. Use the ▸ buttons beside each control (or the Run the experiments launchers) to auto-play; everything runs client-side.

Bayesian Nonparametric Mixture of Experts · Merge-Truncate-Merge

The Bayesian companion to the dendrogram demo above. A Gaussian-gated mixture of experts (GLLiM) is fit by variational Bayes with a Dirichlet-process prior at a large truncation, so it deliberately over-fits the mixing measure; the Merge-Truncate-Merge (MTM) post-processing collapses the redundant atoms, and repeating its randomization gives a full posterior over the number of experts. Drag the merge radius (the Bayesian analogue of cutting the dendrogram) and watch the posterior concentrate on the true count with calibrated uncertainty.

Bayesian nonparametricsMixture of expertsUncertainty quantificationInverse problems

Read the paper Journal (JNPS)

BNP mixture of experts (GLLiM) · MTM merging

π-shape inverse problem · faint rings = over-fitted atoms · colour = MTM-merged expert

Posterior over # expertsmode K̂ = 3

drag the merge radius ↔ below

Merge radius ω 0.85 True K₀ 3 Samples N 600 Truncation K 16

Each expert is a local linear map with a Gaussian gate on the input, the GLLiM parameterization of a mixture of experts for inverse problems; the π-shape scatter is K₀ overlapping linear branches. The model is fit by variational Bayes with a Dirichlet-process stick-breaking prior at a deliberately large truncation, so the mixing measure is over-fitted into many redundant atoms (a known effect: with N observations the variational posterior cannot prune non-empty components on its own, which is exactly what motivates a post-hoc merge). Merge-Truncate-Merge then operates on the joint-Gaussian atoms (μ, V) of the mixing measure: merge atoms within the radius ω, truncate the negligible-weight atoms, then merge again. Because the merge order is randomized, repeating MTM over resampled fits yields a posterior over the number of experts (right), whose mode is the estimate K̂ and whose spread is the uncertainty. The radius is anchored to the paper's contraction rate ω_N ∼ √(log log N / log N) and defaults to the value that recovers K₀; drag it (the Bayesian analogue of the dendrogram cut above) and the posterior shifts, raise N and it concentrates, switch to a Pitman-Yor prior for a heavier tail of small components. (An illustrative in-browser version: the engine is a 1-D DP-GLLiM with point-mass expert parameters and MTM per the paper's Algorithm 1; the published method adds full hierarchical priors and handles high-dimensional inverse problems. Values are illustrative.)

Run the experiments

Every animation runs live in your browser. Click a button to play that experiment on the demo (it scrolls up and starts); drag any control to take over. Nothing is downloaded, it is generated on the fly.

Merge-Truncate-Merge (the merging)

Sweep the merge radius ω: redundant atoms merge and the posterior number of experts falls, the Bayesian analogue of the dendrogram cut.

True number of experts K₀

Change the data-generating number of experts and watch the posterior over the count track it.

Sample size N

More data sharpens the posterior over the number of experts.

Truncation level K

Raise the truncation; the variational fit over-segments, which is exactly why MTM is needed.

How the EM algorithm evolves

Play the variational EM iterations on the panel below: the experts start broad and diffuse, then the Dirichlet-process prior concentrates the ones that carry data and shrinks the redundant weights.

Merge-Truncate-Merge trajectory

Play the merge: the effective number of experts K falls one at a time from the over-fitted count down to 1, the gold panel stepping through the trajectory grid.

How the EM algorithm evolves

Watch the variational EM converge. The atoms start as broad, diffuse experts; over the iterations the Dirichlet-process prior concentrates the ones that carry data and shrinks the redundant weights (thin outlines), the over-fitting you then prune with Merge-Truncate-Merge. Press Run the EM evolution below to replay iterations 0 to 60.

Merge-Truncate-Merge trajectory

The Bayesian analogue of cutting the dendrogram. Starting from the over-fitted variational atoms (the many faint experts the DP/PY prior keeps), the two closest atoms of the mixing measure merge at each step, so the effective number of experts K falls one at a time down to 1. Each panel is one merge level, with the induced clustering and its regression mean; the panel at the current posterior mode K̂ is outlined in gold. Drag the merge radius ω above (or run its sweep) and watch the gold panel move as the posterior mode shifts.

How the fit forms, the merge unfolds, and how it compares to a model sweep

How the points are coloured. Left: the complete data, each point at its true generating expert. Middle: the incomplete data actually observed (labels unknown). Right: the fitted responsibilities γ(z_nk), each point drawn with proportions of the experts' ink, so points in the overlaps blend colours. The Soft γ-colours toggle switches every panel between this soft view and hard argmax.

Fit the Gaussian-gated mixture of experts by variational Bayes under a Dirichlet process prior at truncation K = 16. It keeps 11 atoms with non-negligible weight for a truth of only K₀ = 3: the prior over-fits into redundant, near-duplicate atoms spread along each branch and does not prune them on its own (toggle Pitman-Yor above: the count barely changes). That is why the post-hoc Merge-Truncate-Merge is needed. Raise the truncation K and the atom count grows; only MTM collapses them back to K₀.

How the EM algorithm evolves. Ellipses are the one-standard-deviation contours of each expert's joint Gaussian; the density of red ink is proportional to the mixing weight E[π_k] (redundant low-weight atoms are thin outlines, so the over-fitting stays visible). Points are tinted by their responsibilities (with Soft γ-colours on). The orange lines are the per-expert local regressions; the white curve is the global mixture mean. Compare maximum-likelihood EM (no prior: all K components keep fighting for data) with variational EM (the Dirichlet-process prior shrinks the redundant weights). The number is the iteration.

Merge-Truncate-Merge trajectory. One realization of how the over-fitted mixing measure collapses to K̂: over-fit (all atoms) → merge atoms within the radius ω → truncate the light-weight atoms → final merge. Coloured lines are the surviving experts' local regressions, the white curve is the global mixture mean; the number in each panel is the surviving expert count.

Model selection: evidence sweep vs MTM. The variational lower bound L(K) peaks at the true K₀; each '+' is one random restart, and a few land on suboptimal local maxima (lower points). But reading K this way needs a full sweep over K with many restarts (6 × the restart count = dozens of fits). The MTM estimate (gold) reads the same answer off a single over-fitted model, with a posterior instead of a point.

The idea in three steps

The same over-fit-then-merge principle as the dendrogram demo, but Bayesian: the prior over-fits on purpose, and the merge is a randomized procedure whose repetitions quantify how sure we are of the component count.

1 · Over-fit

A nonparametric prior with too many experts

Fit the Gaussian-gated mixture of experts by variational Bayes under a Dirichlet-process prior at a large truncation. The mixing measure ends up with many redundant, near-duplicate atoms spread along each branch.

2 · Merge-Truncate-Merge

Collapse the mixing measure

On the joint-Gaussian atoms (μ, V): merge atoms within the radius ω, truncate the negligible-weight ones, then merge again. One consistent estimator of the component count, with no model sweep.

3 · Quantify

A posterior, not a point

Randomizing the merge over resampled fits turns the estimate into a posterior over the number of experts, with a credible interval, the Bayesian uncertainty a single dendrogram cut cannot give.

Where the dendrogram demo builds a full agglomerative hierarchy and reads K̂ off a cut, the Bayesian MTM makes a single randomized merge pass whose repetitions become the uncertainty. Both act on the same mixing-measure atoms with the same joint-Gaussian distance, so they are directly comparable. For the model, the consistency theory (MTM recovers K₀ as N grows) and the high-dimensional inverse-problem experiments, see Bayesian Nonparametric Mixture of Experts for Inverse Problems (Nguyen, Forbes, Arbel & Nguyen, J. Nonparametric Statistics 2024) and the published article.