BSL with Mixtures of Experts: a surrogate likelihood that bends
Bayesian Synthetic Likelihood (BSL) replaces an intractable likelihood by a single Gaussian fitted to simulated summaries, one fit per MCMC step. When the summaries are crescent-shaped (the classic two-moons model), one Gaussian cannot bend to follow them. This paper swaps that Gaussian for a Mixture of Experts (a GLLiM / Gaussian-mixture surrogate estimated by EM), which hugs the crescent, and it is fitted once instead of every iteration. Drag the source, raise K, and watch the amber ellipse fail while the teal mixture wins, then read the amortization saving.
The unknown is a source θ = (θ₁, θ₂); the two-moons simulator maps it to a crescent-shaped cloud of summary statistics (blue dots), a likelihood that strongly departs from Gaussianity. Classic BSL (Price et al.) approximates that likelihood by a single Gaussian with the empirical mean and covariance of the summaries: the amber 2σ ellipse straddles the gap and cannot follow the arc. The paper's surrogate is a Mixture of Experts estimated by a Gaussian Locally Linear Mapping (GLLiM) model, here a K-component Gaussian mixture fitted by a few in-browser EM steps: its teal component ellipses hug the crescent, and at K = 1 it collapses back to classic BSL, so the K slider is a literal morph from one blob to a crescent-hugging mixture. The average log-likelihood readout quantifies the fit: the mixture scores higher as K grows. The right panel is the efficiency story. Classic BSL is pointwise in θ, so each of I MCMC iterations re-simulates m summaries, a total of I × m draws (the steep amber line). GLLiM-BSL learns the MoE once on a size-N sample and then takes m = 1: "no further simulations are required," a cost independent of I (the flat teal line), which is the amortization win reported in the paper (there N = 10⁵). The headline ratio is I×m / N. (Honest toy caveats: this is a 2D two-moons illustration, not the paper's full pipeline. The MoE here is an unconditional K-Gaussian mixture fitted directly to the summaries by a handful of EM iterations, a stand-in for the paper's conditional GLLiM inverse-regression estimator with BIC-selected K. The cost curves are an illustrative simulation-count model with an assumed fixed I and N, not measured wall-clock time. No MCMC is actually run.)
Run the experiments
Every animation runs live in your browser. Click a button to run that experiment on the demo (it scrolls up and starts); drag any control to take over. Nothing is downloaded, it is generated on the fly.
Morph BSL into a mixture of experts
Sweep the number of experts from K=1 (which is exactly classic single-Gaussian BSL) up to K=8, and watch the teal component ellipses bend around the two-moons crescent while the average log-likelihood climbs.
Move the source along theta1
Slide the first source coordinate to translate and bend the crescent across observation space; the single-Gaussian BSL ellipse keeps straddling the gap the mixture follows.
Move the source along theta2
Slide the second source coordinate to translate the two-moons crescent and see both surrogates refit to the shifted summary cloud.
Grow the per-iteration simulation budget
Increase the number of simulated summaries m. In classic BSL this whole batch is re-drawn every one of I MCMC iterations, so the amber I x m cost line steepens and the fewer-simulations ratio over amortized GLLiM-BSL grows.
Classic BSL vs GLLiM-MoE vs both
Cycle the overlay from the single-Gaussian BSL ellipse, to the crescent-hugging K-component mixture of experts, to both together, to see exactly what the extra components buy.
Resample the two-moons summaries
Draw fresh simulated summary clouds from the two-moons model at the current source and refit both surrogates, showing the fit is stable across simulation noise.
The idea in three steps
BSL needs a tractable stand-in for an intractable likelihood. The usual stand-in is a single Gaussian, refitted at every MCMC step. This paper makes the stand-in flexible and fits it only once.
Crescent summaries
For a source θ the two-moons model emits a curved, non-Gaussian cloud of summary statistics. A single Gaussian, the classic BSL surrogate, cannot represent that shape.
A mixture of experts
A GLLiM / Gaussian-mixture surrogate, estimated by EM, places K local Gaussians along the arc. Raising K lets it hug the crescent and match the true likelihood far better.
Fit once, not every step
The MoE is learned once on N draws, then reused with m = 1 across all MCMC iterations. Cost stops scaling with I × m, giving the paper's accuracy and efficiency gains.
For the approximation guarantees, the GLLiM estimator, BIC selection of K, and the full simulation study, see Bayesian Likelihood Free Inference using Mixtures of Experts (H. D. Nguyen, TrungTin Nguyen & Forbes, IJCNN 2024).