Universal Approximation · Mixtures and Mixtures of Experts
Why these models can represent essentially anything. A finite location-scale mixture approximates any probability density, and a mixture of experts approximates any regression function (and conditional density), to arbitrary accuracy as the number of components K grows. Pick a hard target and a family, then drag K: the approximation hugs the target and the Lp error falls to zero, at a rate set by how smooth the target is.
These are the universal approximation theorems behind mixture models, made visual. In Mixture · density mode, a finite location-scale mixture (here Gaussian components at a shrinking scale) approximates the target density; my work shows such mixtures are dense in the Lebesgue spaces Lp and in the continuous densities that vanish at infinity, so the Lp error → 0 as K → ∞. In MoE · regression mode, a softmax-gated MoE (soft-partition gates with local linear experts) approximates any regression function E[y|x]. In MoE · conditional density mode the same MoE approximates a full conditional density p(y|x), drawn as a 2-D heatmap with the true conditional modes dashed on top: a mixture of experts can capture bimodal, bifurcating and heteroscedastic conditionals that a single regression mean cannot. (Approximating a 2-D conditional is genuinely harder than a 1-D density, so its error falls more slowly, the honest curse of dimensionality.) The shaded band is the pointwise gap between target and approximation, and its total area is the L1 error plotted on the right against K on log–log axes; the slope is the convergence rate. Try the smooth targets (Sine, Skewed) for a fast rate, then the hard ones (Comb, Doppler, Steps): the model still converges, just more slowly, because smoothness sets the rate. This illustrates three papers: continuous densities vanishing at infinity (Cogent Math. & Stat. 2020), location-scale mixtures in Lebesgue spaces (Comm. Stat. 2022), and conditional densities via mixtures of experts (JSDA 2021). (The approximations here are the constructive ones from the proofs, not fitted to data, so the curves show the theorem itself.)
Run the experiments
Every animation runs live in your browser. Click a button to play that experiment on the demo (it scrolls up and starts); drag the slider to take over. Nothing is downloaded, it is generated on the fly.
Increase the number of components K
Watch the approximation converge to the target as K grows, the universal-approximation theorem made visible.
Density, regression, conditional
Tour the three settings: approximating a density, a regression mean, and a full conditional density.
The idea in three steps
A universal approximation theorem says a model class can represent any target arbitrarily well. These demos turn that abstract denseness statement into something you can watch converge.
Denseness
Whatever the target, a mixture (for densities) or a mixture of experts (for regressions and conditional densities) can get within any tolerance, because these classes are dense in the Lebesgue space Lp.
Convergence
Increase K and the approximation error falls to zero. The shaded gap shrinks and the solid curve locks onto the dashed target.
Smoothness sets the speed
Smooth targets converge fast (a steep log–log slope); rough targets (jumps, high-frequency wiggles) converge more slowly. The convergence rate is the quantitative half of the theory.
For the theorems, the exact function classes, and the rates, see Approximation capabilities and convergence rates of the mixture of experts models and the companion papers on location-scale mixtures.