Model selection for the Gaussian-gated localized mixture of experts (GLoME)

The covariate x is drawn from a two-component Gaussian mixture (centers 0.2 and 0.8, variances 0.10 and 0.15), and the response follows a Gaussian-gated localized mixture of experts: the gate posterior τ_k(x) ∝ π_k N(x; c_k, Γ_k) picks an expert, and inside its region y = mean_k(x) + σ noise with σ = 0.30. In the well-specified case the two experts are linear (y = −5x + 2 and y = 0.1x); in the misspecified case they are parabolas (y = 1 − 6x + 3x² and y = −1.5x², an illustratively amplified curvature, see the note below). For each candidate K = 1..8 we fit the Gaussian-gated linear-expert family (GLLiM) by EM (a few random restarts), recording its conditional log-likelihood and its dimension D(K) = 6K − 1 (per expert: gate mean + gate variance + slope + intercept + expert variance = 5, plus K−1 mixing weights; K = 1 is a single linear regression with D(1) = 3). The paper proves a non-asymptotic oracle inequality for the penalized maximum-likelihood estimator under the Jensen-Kullback-Leibler loss: as long as the penalty is at least a known shape times a large-enough constant, the selected model performs as well as the oracle. Since the shape is known only up to a multiplicative constant κ, we select K by minimizing −logLik(K) + κ·D(K). The slope heuristic (capushe Djump) reads κ off the data: as κ grows the selected dimension collapses in a staircase, and the location κ̂ of its biggest drop (the dimension jump) is the minimal penalty. The theory prescribes κ* = 2κ̂, which recovers K = 2 when well-specified and a larger K (about 3-4) when the linear family must approximate the parabola. AIC (κ = 1) and BIC (κ = ½ log n) are marked as fixed ticks. (This is a faithful but 1D, small-n port of the author's R package NamsGLoME: the published experiments use n = 2000 and 10000 with GLLiM fitted by xLLiM and penalties calibrated by capushe over 100 trials, and the guarantee is the asymptotic-free oracle inequality, not the finite staircase you drag here. The well-specified experts are the author's exact linear coefficients; the misspecified parabola's curvature is amplified beyond the author's gentle x² / −0.4x² (which a faithful fit already approximates well with K = 2 at every sample size) so the linear-expert family visibly needs a larger order (K ≈ 3-4) here, reproducing the paper's well-vs-misspecified contrast at browser scale. In-browser EM uses few restarts, so a candidate fit can occasionally miss its global optimum and the recovered K can wobble, especially in the misspecified case at small n.)