Software · Mixture of experts

Examples

A static gallery in the spirit of scikit-learn's example gallery (every number is produced by running the package), followed by a live in-browser demo that fits a softmax-gated mixture of experts by EM as you watch.

Gallery

Softmax-gated mixture of experts on a change-point regression

Softmax-gated mixture of experts: data coloured by expert, the two expert fits, the mixture prediction, and the softmax gating weights. — Two linear regimes joined at x = 0. Left: points coloured by their dominant expert, the two local linear experts (dashed), and the mixture prediction E[y|x] (black). Right: the softmax gate weights, which switch at x = 0.

Reproduced by running GaussianMixtureOfExperts(n_components=2, gate="softmax", n_init=5, random_state=0) on n = 600 synthetic points: converged in 56 EM iterations, mean log-likelihood -0.274, recovered expert slopes [-0.989, 2.025] and residual std [0.309, 0.317] (data-generating truth: slopes -1 and 2, noise std 0.30). Synthetic simulation.

from mixture import GaussianMixtureOfExperts
moe = GaussianMixtureOfExperts(n_components=2, gate="softmax", n_init=5, random_state=0).fit(x, y)
moe.predict(grid)        # mixture prediction
moe.predict_proba(grid)  # gating weights

Gaussian-gated mixture of experts (GLLiM) on clustered inputs

Gaussian-gated mixture of experts: data coloured by expert, expert fits, the mixture prediction, and the Gaussian gating weights. — Inputs drawn from two Gaussian clusters. The Gaussian gate learns where each expert applies from a Gaussian on the inputs (GLLiM).

Reproduced with gate="gaussian", covariance_type="full", n_init=5, random_state=0: converged in 4 EM iterations, mean log-likelihood -1.556, recovered gate means [-1.474, 1.521] and gate variances [0.231, 0.252] (truth: centres -1.5 and 1.5, variance 0.25). Synthetic simulation.

Run it in your browser

This demo implements the same softmax-gated EM in JavaScript (Bohning gate M-step, weighted-least-squares experts). It runs entirely client-side, no data leaves your browser. Press Run EM and watch the experts and gate settle.

Softmax-gated mixture of experts · live EM

Experts K 2 Noise 0.30

Points are coloured by their most likely expert; dashed lines are the experts, the bright curve is the mixture prediction E[y|x], and the band under the axis shows the gate weight of expert 1. Illustrative in-browser EM; the Python package is the reference implementation.