The Asymptotic Law of the Minimum Empirical Risk

The unknown is a hypothesis x ∈ [0,1]. The population risk f(x) (teal) is a valley whose bottom you reshape: at w = 0 it is a sharp parabola with a single minimizer x₀ = ½; as w grows the bottom flattens into a plateau, so the population argmin set S = [x₀ − w/2, x₀ + w/2] becomes an interval (teal band). Each draw is a fresh empirical risk f̂ₙ(x) = f(x) + n^−1/2 G(x) (blue), where G is a smooth mean-zero Gaussian field (a random-Fourier surrogate for the empirical process, correlation length ℓ, marginal variance σ²). We locate its minimizer (amber), read off the minimum empirical risk, and drop the rescaled blow-up √n (MER − min f) = inf_x [√n f(x) + G(x)] into the histogram. Singleton (w = 0): the minimizer localizes at x₀ and the statistic converges to G(x₀) ~ N(0, σ²), symmetric and centered, and the teal Gaussian overlay fits, so the CLT and the bootstrap are valid. Plateau (w > 0): the statistic becomes inf_x∈S G(x), the infimum of a Gaussian process over S, which is negatively shifted and left-skewed, so the histogram detaches from the Gaussian: this is the over-specification regime where naive Gaussian inference and the standard bootstrap fail. Raising n tightens the singleton toward the Gaussian but leaves the plateau's non-Gaussian law unchanged. (Honest toy: this is the i.i.d., continuous, √n baseline with a random-Fourier Gaussian-field stand-in for the empirical process and a grid argmin. The paper proves the general limit for non-i.i.d. data, discontinuous losses and losses indexed by non-Euclidean spaces, then builds consistent bootstrap confidence sets, penalized model-selection tests, and neural-network applications: none of that heavier machinery is in this 2D cartoon.)

This is the paper's dependent-data application to mixture-of-experts model selection, reproducing NIIDMoE/BIC_NIID_SGaME.R (built on the meteorits R package). The true generator is a univariate softmax-gated Gaussian Mixture-of-Experts (SGaME) with K₀ experts (K₀ = 2 uses the script's Context 2: gating α = (−7, 15), experts β₁ = (0.6, 0.4), β₂ = (8, −15), σ = (0.4, 0.3)). Given inputs x, the response is y | x ~ ∑_k π_k(x;α) N(β_k^⊤(1,x), σ_k²) with softmax gates π_k. The inputs are dependent (non-i.i.d.): an m-dependent stationary sequence drives xₕ (i.i.d. ε ~ N(0,1); xₕ lands in a low or high band according to the sign of ∑_j=0..m ε_i+j), so neighbours share innovations and the effective sample size shrinks as m grows. Each trial fits K = 1..K max by softmax-gated Gaussian MoE EM (an emNMoE re-implementation: weighted least squares experts + IRLS/Newton for the softmax gating, one random start), with degrees of freedom df = (K−1)(q+1) + K(p+1) + K (p = q = 1, so df = 5K − 2). We score the three penalised maximum-log-likelihood criteria of the R script and select K by argmax: AIC = ℓ/n − df (heavy df penalty), SWIC = ℓ/n − df log n/√n (the dependence-corrected penalty whose 1/√n rate is exactly the MER blow-up rate) and BIC = ℓ/n − df log n/(2n) (the naive i.i.d. penalty, the weakest). The top-right plot shows each criterion versus K (normalised per criterion) with its selected K ringed; the bottom-right plot accumulates the histogram of selected K over trials with P(K̂ = K₀) read out. What the simulation robustly shows, matching the author's own committed run (n = 2000, 50 trials, K₀ = 2): AIC collapses to K = 1 (its df penalty is far too strong: 50/50 at K = 1), SWIC concentrates exactly on the true K₀ (50/50 at K = 2), and BIC finds K₀ at its mode but leaves an over-selection tail (36/50 at K = 2, with 14 spilling to K = 3, 4, 5). This is the paper's point: the √n MER rate makes SWIC the consistent, dependence-aware choice, whereas the naive log n/n BIC under-penalises and over-selects. Push True experts K₀ to 3 to see the tension sharpen: at this modest n the heavy SWIC penalty starts under-selecting (it needs larger n for larger K₀) while BIC's weak penalty visibly over-selects past K₀. (Honest scope: to run live, n is a few hundred to ~1200 (not the paper's full 1000–3000) and the EM uses one random start with a variance floor and capped iterations. That regularised single-start EM reproduces AIC's collapse and SWIC's exactness robustly but under-represents BIC's over-selection tail at K₀ = 2 (it converges to the clean 2-expert fit), which is why the BIC tail is easiest to see at K₀ = 3. The K₀ = 3 setting uses illustrative gating and expert parameters that are not in the R script, which defines only K₀ = 2 (Context 2); it is a pedagogical extension to show the penalty tradeoff, not a reproduction of a paper experiment. The full-scale meteorits EM tables and exact rate exponents are in the paper and the R scripts.)