Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps

Do Tien Hai; Trung Nguyen Mai; TrungTin Nguyen; Nhat Ho; Binh T. Nguyen; Christopher Drovandi

Abstract

We develop a unified statistical framework for softmax-gated Gaussian mixture of experts (SGMoE) that addresses three long-standing obstacles in parameter estimation and model selection: (i) non-identifiability of gating parameters up to common translations, (ii) intrinsic gate-expert interactions that induce coupled differential relations in the likelihood, and (iii) the tight numerator-denominator coupling in the softmax-induced conditional density. Our approach introduces Voronoi-type loss functions aligned with the gate-partition geometry and establishes finite-sample convergence rates for the maximum likelihood estimator (MLE). In over-specified models, we reveal a link between the MLE’s convergence rate and the solvability of an associated system of polynomial equations characterizing near-nonidentifiable directions. For model selection, we adapt dendrograms of mixing measures to SGMoE, yielding a consistent, sweep-free selector of the number of experts that attains pointwise-optimal parameter rates under overfitting while avoiding multi-size training. Simulations on synthetic data corroborate the theory, accurately recovering the expert count and achieving the predicted rates for parameter estimation while closely approximating the regression function. Under model misspecification (e.g., $ϵ$ -contamination), the dendrogram selection criterion is robust, recovering the true number of mixture components, while the Akaike information criterion, the Bayesian information criterion, and the integrated completed likelihood tend to overselect as sample size grows. On a maize proteomics dataset of drought-responsive traits, our dendrogram-guided SGMoE selects two experts, exposes a clear mixing-measure hierarchy, stabilizes the likelihood early, and yields interpretable genotype-phenotype maps, outperforming standard criteria without multi-size training. ^†^†^⋆Co-first author, ^†Corresponding author.

1 INTRODUCTION

Mixture of Experts: Scope and Appeal. Mixture of experts (MoE) were introduced as modular neural architectures in Jacobs et al. (1991); Jordan and Jacobs (1994), where a gating network dispatches inputs to specialized experts. Beyond their practical versatility in speech, language, and vision (Fedus et al., 2022; Pham et al., 2024; Do et al., 2023; Eigen et al., 2014; Bao et al., 2022; Dosovitskiy et al., 2021; Liang et al., 2022; You et al., 2021, 2022; Peng et al., 1996), MoE admit strong approximation guarantees and learning theory. Universal approximation results for conditional densities and regressors quantify how MoE improve upon unconditional mixtures by allowing both gates and experts to depend on covariates (Norets, 2010; Nguyen et al., 2016, 2019, 2021a). These developments complement classical approximation and risk bounds for unconditional mixtures (Genovese and Wasserman, 2000; Rakhlin et al., 2005; Nguyen et al., 2025b; Chong et al., 2024; Nguyen, 2013; Shen et al., 2013; Ho and Nguyen, 2016a, b; Nguyen et al., 2020, 2023b) and are surveyed in Yuksel et al. (2012); Nguyen and Chamroukhi (2018); Nguyen (2021); Chen et al. (2022).

Parameter Estimation: from Unconditional Mixtures to MoE. Over-specified finite mixtures can display slow, nonstandard parameter rates. In unconditional mixtures this is explained by singular Fisher information and merging components. Foundational results start with Chen (1995) for univariate mixtures, and extend via Wasserstein tools to multivariate models and weaker identifiability (Nguyen, 2013; Ho and Nguyen, 2016a), with minimax studies in Heinrich and Kahn (2018); Manole and Ho (2020). Algorithmic guarantees for Expectation-Maximization (EM) and Majorization-Minimization or Minimization-Maximization (MM) algorithms and moments have been analyzed under both exact-fit and over-fit regimes (Balakrishnan et al., 2017; Anandkumar et al., 2012; Hardt and Price, 2015; Dwivedi et al., 2020b, a; Wu and Yang, 2020; Doss et al., 2023; Wu and Zhou, 2021; Tran et al., 2026b). For MoE with covariate-free gates, parameter rates depend on algebraic independence of experts and PDE-type couplings (Ho et al., 2022; Do et al., 2025). In softmax-gated Gaussian mixture of experts (SGMoE), parameter estimation is harder due to translation invariance in softmax gates and intrinsic gate-expert couplings; recent progress includes identifiability, inverse bounds, and finite-sample guarantees for the maximum likelihood estimator (MLE) with unified exact- and over-fit treatments in Nguyen et al. (2023a, 2024a, 2024c).

Model Selection: Information Criteria, Penalties, and Bayes. Choosing the number of experts remains critical despite universal approximation theorems. Classical criteria balance fit and complexity, including AIC (Akaike, 1974; Frühwirth-Schnatter et al., 2018), BIC and its MoE adaptations (Schwarz, 1978; Khalili et al., 2024; Forbes et al., 2022a; Berrettini et al., 2024; Forbes et al., 2022b; Nguyen and Nguyen, 2025; Ho et al., 2025), ICL (Biernacki et al., 2000; Frühwirth-Schnatter et al., 2012), eBIC for structured settings (Foygel and Drton, 2010; Nguyen and Li, 2024), and SWIC for dependent data (Sin and White, 1996; Nguyen et al., 2025a; Westerhout et al., 2024). These methods are largely asymptotic and often require multi-size model sweeps. Non-asymptotic penalization brings risk guarantees via weak oracle bounds in high-dimensional MoE (Nguyen et al., 2021b, 2022a, 2022b, 2023c; Montuelle and Le Pennec, 2014; Nguyen et al., 2023d). Bayesian strategies avoid fixing the order but need careful marginal-likelihood evaluation or post-processing; the merge-truncate-merge approach ensures consistency in related mixture settings yet introduces sensitive tuning (Frühwirth-Schnatter, 2019; Zens, 2019; Guha et al., 2021; Nguyen et al., 2024d). A recent alternative leverages dendrograms of mixing measures for selection without exhaustive sweeps in (Do et al., 2024; Thai et al., 2025; Tran et al., 2026a).

Gaps Specific to SGMoE. Softmax gating creates three intertwined obstacles. First, gate parameters are identifiable only up to common translations, so parameter losses must factor out these symmetries. Second, the softmax numerator-denominator coupling and the expert structure induce exact PDE relations between derivatives, which collapse naive Taylor decompositions and require algebra-aware inverse bounds. Third, when models are over-specified, the first nonvanishing terms in the expansions are ruled by solvability of polynomial systems; the resulting exponents govern slow parameter rates and depend on how many fitted atoms approximate each truth (Ho et al., 2022; Nguyen et al., 2023a). Existing selection criteria do not exploit this rate geometry for the MLE, and sweep-based procedures are computationally heavy for SGMoE.

Contributions. We introduce a fast-rate-aware Voronoi distance for SGMoE that augments the unified exact- and over-fit loss with merged-moment couplings inside multi-covered Voronoi cells (eq. 6). This exposes slow directions created by redundant atoms, motivates a hierarchical merge operator, and yields an aggregation path (dendrogram) on mixing measures. Along this path we prove a monotone strengthening of the loss (Lemma 1), obtain near-parametric finite-sample rates for the aggregated estimators together with height and likelihood control (Theorems 1, 2 and 3 and Table 1), and derive a sweep-free dendrogram selection criterion (DSC) that is consistent and avoids multi- $K$ training (Theorems 4, 1 and 3). Empirically, DSC is less prone to overfitting than AIC/BIC/ICL under $ϵ$ -contamination due to its structural penalty on small heights (Figure 4), and it restores fast parameter rates after aggregation in over-specified SGMoE (Figure 2). To our knowledge this is the first method that couples finite-sample, fast-rate-aware merging with consistent model selection for SGMoE, avoiding multi-size training while preserving statistical efficiency.

Table 1: Summary of density and parameter rates for SGMoE. The Voronoi cells

𝔸_{j}

are defined in eq. 2. The function

\bar{r} ​ (\cdot)

is determined by solvability of the polynomial systems recalled in eq. 3 (e.g.,

\bar{r} ​ (2) = 4

,

\bar{r} ​ (3) = 6

). The merged row and the fast pathwise rates correspond to the aggregation path described in Section 3.

Setting	Loss	$p_{G_{0}} (y ∣ 𝒙)$	$\exp (ω_{0 k}^{0})$	$𝝎_{1 k}^{0}, b_{k}^{0}$	$𝒂_{k}^{0}, σ_{k}^{0}$
Exact-fit	$D_{E}$	$𝒪 ({(\log N / N)}^{1 / 2})$	$𝒪 ({(\log N / N)}^{1 / 2})$	$𝒪 ({(\log N / N)}^{1 / 2})$	$𝒪 ({(\log N / N)}^{1 / 2})$
Over-fit	$D_{O}$	$𝒪 ({(\log N / N)}^{1 / 2})$	$𝒪 ({(\log N / N)}^{1 / 2})$	$𝒪 ({(\log N / N)}^{1 / 2 \bar{r} (\| 𝔸_{k} \|)})$	$𝒪 ({(\log N / N)}^{1 / \bar{r} (\| 𝔸_{k} \|)})$
Merged	$D_{FRA}$	$𝒪 ({(\log N / N)}^{1 / 2})$	$𝒪 ({(\log N / N)}^{1 / 2})$	$𝒪 ({(\log N / N)}^{1 / 2})$	$𝒪 ({(\log N / N)}^{1 / 2})$

SGMoE Setting. Let ${(𝐱_{n}, y_{n})}_{n = 1}^{N}$ be i.i.d. samples with $𝐱_{n} \in ℝ^{D}$ and $y_{n} \in ℝ$ . Assume the data are generated by a SGMoE model of order $K_{0}$ , whose conditional density is

	$p_{G_{0}} (y ∣ 𝒙)$	$:= \sum_{k = 1}^{K_{0}} \frac{\exp ({(𝝎_{1 k}^{0})}^{⊤} 𝒙 + ω_{0 k}^{0})}{\sum_{j = 1}^{K_{0}} \exp ({(𝝎_{1 j}^{0})}^{⊤} 𝒙 + ω_{0 j}^{0})}$
		$\times 𝒩 (y \| 𝒂_{k}^{0 ⊤} 𝒙 + b_{k}^{0}, σ_{k}^{0}) .$		(1)

Each expert is Gaussian with mean $𝒂_{k}^{0 ⊤} 𝒙 + b_{k}^{0}$ and variance $σ_{k}^{0} > 0$ . We encode parameters via the (not-necessarily normalized) mixing measure

G_{0} \equiv G_{0} ​ (K_{0}) := \sum_{k = 1}^{K_{0}} \exp (ω_{0 ​ k}^{0}) ​ δ_{(𝝎_{1 ​ k}^{0}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})},

where $𝜼_{k}^{0} := (ω_{0 k}^{0}, 𝝎_{1 k}^{0}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}) \in 𝚯 \subset ℝ \times ℝ^{D} \times ℝ^{D} \times ℝ \times ℝ_{> 0}$ . Assume $𝚯$ is compact and $𝒳 \subset ℝ^{D}$ , the support of $𝐱$ , is bounded. Assume $𝐱$ has a continuous distribution so that the model is identifiable under this convention, a standard mild assumption; see Proposition 1 of Nguyen et al. (2023a).

Maximum Likelihood Over At Most $K$ Experts. When the true order $K_{0}$ is unknown, we estimate within $𝒪_{K} (𝚯) := {G = \sum_{k = 1}^{K^{'}} \exp (ω_{0 k}) δ_{(𝝎_{1 k}, 𝒂_{k}, b_{k}, σ_{k})} :$ $1 \leq K^{'} \leq K, (ω_{0 k}, 𝝎_{1 k}, 𝒂_{k}, b_{k}, σ_{k}) \in 𝚯} .$ We analyze the exactly specified case $K = K_{0}$ , the over-specified case $K > K_{0}$ , and the merging scheme using the following maximum likelihood estimator (MLE): ${\hat{G}}_{N} \in {\arg \max}_{G \in 𝒪_{K} (𝚯)} \frac{1}{N} \sum_{n = 1}^{N} \log (p_{G} (y_{n} ∣ 𝐱_{n})) .$

Practical Implication. Practitioners can fit a single over-specified SGMoE with moderate $K \geq K_{0}$ , compute its aggregation path, and select $\hat{K}$ via DSC. This single-fit workflow avoids grid sweeps over $K$ , merges near-duplicate atoms to collapse slow directions within Voronoi cells, accelerates parameter convergence, and often recovers the correct expert count even under mild contamination. Dendrogram heights provide a transparent structural summary.

Paper Organization. Section 2 states the unified parameter-rate result and the algebraic exponents $\bar{r} (\cdot)$ . Section 3 introduces the fast-rate-aware distance, merge operator, aggregation path, fast pathwise rates, and DSC. Section 4 illustrates parameter rates, path behaviour, model selection under clean and contaminated regimes, and a real-data application to maize drought-response traits in Section 5. Then, we offer concluding remarks, limitations, and future work in Section 6. Proof sketches appear at the end of Section 3, with full proofs deferred to the appendix. Additional biological background, preprocessing details for the maize dataset, and further geometric and technical discussion are provided in the supplementary material.

Notation. Throughout the paper, for any natural number $N \in ℕ$ we abbreviate ${1, 2, \dots, N}$ by $[N]$ . Given two sequences of positive real numbers ${a_{N}}_{N = 1}^{\infty}$ and ${b_{N}}_{N = 1}^{\infty}$ , we write $a_{N} = 𝒪 (b_{N})$ (equivalently, $a_{N} ≲ b_{N}$ ) to mean that there exists a constant $C > 0$ such that $a_{N} \leq C b_{N}$ for all $N \in ℕ$ . For a vector $𝒗 \in ℝ^{D}$ , set $| 𝒗 | := v_{1} + \dots + v_{D}$ , and let ${‖ 𝒗 ‖}_{p}$ denote its $p$ -norm; by default, $‖ 𝒗 ‖$ refers to the $2$ -norm unless otherwise stated. We also use $‖ 𝑨 ‖$ for the Frobenius norm of a matrix $𝑨 \in ℝ^{D \times D}$ . For any set $𝕊$ , $| 𝕊 |$ denotes its cardinality. Finally, for two probability density functions $p$ and $q$ with respect to the Lebesgue measure $μ$ , define $D_{TV} (p, q) := \frac{1}{2} \int | p - q | 𝑑 𝝁$ as their Total Variation distance, while $D_{h}^{2} (p, q) := \frac{1}{2} \int {(\sqrt{p} - \sqrt{q})}^{2} 𝑑 𝝁$ denotes the squared Hellinger distance between them. Let $𝚯$ be the parameter space. Write $ℰ_{K} (𝚯)$ for the collection of discrete probability measures on $𝚯$ with exactly $K$ atoms, and $𝒪_{K} (𝚯) := ⋃_{K^{'} \leq K} ℰ_{K^{'}} (𝚯)$ for those with at most $K$ atoms. For a mixing measure $G = \sum_{k = 1}^{K} π_{k} δ_{𝜽_{k}}$ , we (slightly abusively) refer to each component $π_{k} δ_{𝜽_{k}}$ as an “atom,” comprising both its weight $π_{k}$ and parameter $𝜽_{k}$ . When clear from context, we drop $𝚯$ and simply write $ℰ_{K}$ and $𝒪_{K}$ .

2 PRELIMINARIES

We present a unified result for the parameter estimation rate of the MLE in the SGMoE that simultaneously covers the exact-specified case ( $K = K_{0}$ ) and the over-specified case ( $K > K_{0}$ ), building on Nguyen et al. (2023a).

Voronoi Cells. For a candidate mixing measure $G = \sum_{k = 1}^{K} \exp (ω_{0 k}) δ_{(𝝎_{1 k}, 𝒂_{k}, b_{k}, σ_{k})}$ and the true $G_{0} = \sum_{k = 1}^{K_{0}} \exp (ω_{0 k}^{0}) δ_{(𝝎_{1 k}^{0}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})}$ , define for $k \in [K_{0}]$ :

𝔸_{k} ​ (G) := {ℓ \in [K] : ‖ 𝜽_{ℓ} - 𝜽_{k}^{0} ‖ \leq ‖ 𝜽_{ℓ} - 𝜽_{j}^{0} ‖, \forall j \neq k},

(2)

where we denote $𝜽_{ℓ} := (𝝎_{1 ℓ}, 𝒂_{ℓ}, b_{ℓ}, σ_{ℓ})$ . We use the softmax-translation $(t_{0}, 𝒕_{1})$ from identifiability (cf. Proposition 1 of Nguyen et al., 2023a) and the shorthand $Δ_{𝒕_{1}} 𝝎_{1 ℓ k} := 𝝎_{1 ℓ} - 𝝎_{1 k}^{0} - 𝒕_{1}$ , $Δ 𝒂_{ℓ k} := 𝒂_{ℓ} - 𝒂_{k}^{0}$ , $Δ b_{ℓ k} := b_{ℓ} - b_{k}^{0}$ , $Δ σ_{ℓ k} := σ_{ℓ} - σ_{k}^{0}$ . For notational simplicity, we write $𝔸_{k}$ instead of $𝔸_{k} (G)$ .

Algebraic Obstruction and Exponents. For $M \geq 2$ , let $\bar{r} (M)$ be the smallest integer $r$ determined by the polynomial system as follows: given $0 \leq | ℓ_{1} | \leq r, 0 \leq ℓ_{2} \leq r - | ℓ_{1} |, | ℓ_{1} | + ℓ_{2} \geq 1,$ the polynomial system

\sum_{j = 1}^{M} \sum_{(𝜶_{1}, 𝜶_{2}, α_{3}, α_{4}) \in 𝕀_{ℓ_{1}, ℓ_{2}}} \frac{p_{5 ​ j}^{2} ​ p_{1 ​ j}^{𝜶_{1}} ​ p_{2 ​ j}^{𝜶_{2}} ​ p_{3 ​ j}^{α_{3}} ​ p_{4 ​ j}^{α_{4}}}{𝜶_{1}! ​ 𝜶_{2}! ​ α_{3}! ​ α_{4}!} = 0,

(3)

admits no non-trivial solution (all $p_{5 j} \neq 0$ and at least one $p_{3 j} \neq 0$ ). The ranges of $𝜶_{1}, 𝜶_{2}, α_{3}, α_{4}$ in the above sum satisfy $𝕀_{ℓ_{1}, ℓ_{2}} = {𝜶 = (𝜶_{1}, 𝜶_{2}, α_{3}, α_{4}) \in ℕ^{D} \times ℕ^{D} \times ℕ \times ℕ : 𝜶_{1} + 𝜶_{2} = ℓ_{1}, | 𝜶_{2} | + α_{3} + 2 α_{4} = ℓ_{2}}$ . For general dimension $D$ and parameter $M \geq 2$ , finding the exact value of $\bar{r} (M)$ is a non-trivial central problem in algebraic geometry (Sturmfels, 2002). Known values:

Fact 1 (Nguyen et al., 2023a, Lemma 1).

For any $D \geq 1$ : $\bar{r} (2) = 4$ , $\bar{r} (3) = 6$ , and $\bar{r} (M) \geq 7$ for $M \geq 4$ .

Classical Overfit-Aware Voronoi Distance. Define a single loss that reduces to the exact-fit metric when each cell has one atom, and adds over-fit penalties otherwise:

	$D_{O} (G, G_{0}) := D_{E} (G, G_{0})$
	$+ inf_{t_{0}, 𝒕_{1}} \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) (∥ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}, Δ b_{ℓ k}) ∥^{\bar{r} (\| 𝔸_{k} \|)}$
	$+ ∥ (Δ 𝒂_{ℓ k}, Δ σ_{ℓ k}) ∥^{\bar{r} (\| 𝔸_{k} \|) / 2}),$		(4)
	$D_{E} (G, G_{0}) := inf_{t_{0}, 𝒕_{1}} \sum_{k = 1}^{K_{0}} \| \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) - \exp (ω_{0 k}^{0} + t_{0}) \|$
	$+ \sum_{k : \| 𝔸_{k} \| = 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) ‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}, Δ 𝒂_{ℓ k}, Δ b_{ℓ k}, Δ σ_{ℓ k}) ‖ .$

When $| 𝔸_{k} | = 1$ for all $k$ (i.e., $K = K_{0}$ ), eq. 4 equals the exact-fit metric $D_{E}$ ; if some $| 𝔸_{k} | > 1$ (i.e., $K > K_{0}$ ), eq. 4 adds the higher-order penalties determined by $\bar{r} (\cdot)$ .

Fact 2 (Nguyen et al., 2023a, Theorems 1 and 2).

There exist universal constants $C, c > 0$ (depending only on $G_{0}$ and $𝚯$ ) s.t. the MLE ${\hat{G}}_{N}$ of order $K \geq K_{0}$ satisfies

ℙ ​ (D_{O} ({\hat{G}}_{N}, G_{0}) > C ​ {(\log N / N)}^{1 / 2}) ≲ e^{- c ​ \log N} .

(5)

Remarks. (i) If $K = K_{0}$ (all $| 𝔸_{k} | = 1$ ), then $D_{O} = D_{E}$ and eq. 5 yields the exact-specified rate $ℙ (D_{E} ({\hat{G}}_{N}, G_{0}) > C {(\log N / N)}^{1 / 2}) ≲ e^{- c \log N},$ implying parametric ( $N^{- 1 / 2}$ up to logs) estimation of $\exp (ω_{0 k}^{0})$ , $𝝎_{1 k}^{0}$ (up to translation), $𝒂_{k}^{0}$ , $b_{k}^{0}$ , $σ_{k}^{0}$ for all $k \in [K_{0}]$ . (ii) If $K > K_{0}$ (some $| 𝔸_{k} | > 1$ ), the same bound holds for $D_{O}$ , while the exponents $\bar{r} (| 𝔸_{k} |)$ inside eq. 4 encode the slower algebraic behavior of over-covered parameters within each Voronoi cell.

3 FAST-RATE-AWARE EXPERT AGGREGATION IN SGMOE

3.1 Why Merge Experts? The Rate Gap

Building on Section 2, identifiability and the unified parameter-rate bound (Fact 2) imply that converting density accuracy into parameter accuracy hinges on a suitable inverse (loss) inequality. When the model is over-specified ( $K > K_{0}$ ), several fitted atoms may fall into the same Voronoi cell $𝔸_{k}$ (defined in eq. 2), which induces a rate gap: single-covered truths achieve (near) parametric rates, whereas multi-covered truths converge more slowly with exponents governed by $\bar{r} (| 𝔸_{k} |)$ from Section 2. To exploit this, we (i) refine the loss to expose mergeable structure, and (ii) aggregate (merge) near-duplicate atoms to recover fast rates and guide model order selection.

3.2 A Fast-Rate-Aware Voronoi Distance

Our Proposal. Let $D_{O} (G, G_{0})$ denote the over-fit Voronoi loss from eq. 4 and $𝔸_{k}$ be as in eq. 2. We augment it with first-order “merged-moment” couplings inside multi-covered cells to obtain

	$D_{FRA} (G, G_{0}) := D_{O} (G, G_{0})$
	$+ inf_{t_{0}, 𝒕_{1}} \sum_{k : \| 𝔸_{k} \| > 1} (∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) (Δ b_{ℓ k}) ∥$
	$+ ‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}) ‖$
	$+ ‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) [{(Δ b_{ℓ k})}^{2} + (Δ σ_{ℓ k})] ‖$
	$+ ‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) [(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}) (Δ b_{ℓ k}) + (Δ 𝒂_{ℓ k})] ‖$
	$+ ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}) {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k})}^{⊤} ∥) .$		(6)

Link to Section 2. The penalties inside eq. 6 are consistent with the exponents $\bar{r} (| 𝔸_{k} |)$ that appear in the unified loss eq. 4: when $| 𝔸_{k} | = 1$ , $D_{FRA}$ reduces to the exact-fit metric $D_{E}$ ; when $| 𝔸_{k} | > 1$ , the added block-sums control the slow directions and quantify how well the cell behaves as if merged.

Motivation for Merging. Because the slow rates originate from multiple atoms sharing a cell, replacing these atoms by their softmax-weighted aggregate collapses the problematic directions and restores first-order (parametric) behavior for the merged parameters. Thus, $D_{FRA}$ both (i) certifies where merging is beneficial (large intra-cell terms) and (ii) predicts the rate improvement obtained by aggregation, which we leverage next for hierarchical merging and model selection.

3.3 A Merge Operator Tailored to SGMoE

Connection to Section 2 and Novelty. The unified rate result in Section 2 shows that parameter convergence hinges on how fitted atoms distribute across Voronoi cells; multi-covered cells induce slower algebraic behavior governed by $\bar{r} (\cdot)$ . The merge operator below is the first ingredient of our contribution: it operationalizes that insight by collapsing near-duplicate atoms within a cell using softmax-weighted updates. This turns slow, multi-component directions into a single, first-order direction, setting up our fast pathwise rates (Theorem 1) and height/likelihood controls (Theorems 2 and 3).

Rate-Weighted Dissimilarity. For $G^{(K)} = \sum_{k = 1}^{K} \exp (ω_{0 k}) δ_{𝜽_{k}}$ with $𝜽_{k} = (𝝎_{1 k}, 𝒂_{k}, b_{k}, σ_{k})$ , define

	$𝖽 (\exp (ω_{0 ℓ_{1}}) δ_{𝜽_{ℓ_{1}}}, \exp (ω_{0 ℓ_{2}}) δ_{𝜽_{ℓ_{2}}})$
	$:= \frac{\exp (ω_{0 ℓ_{1}} + ω_{0 ℓ_{2}})}{\exp (ω_{0 ℓ_{1}}) + \exp (ω_{0 ℓ_{2}})} {‖ (𝝎_{1 ℓ_{1}}, b_{ℓ_{1}}) - (𝝎_{1 ℓ_{2}}, b_{ℓ_{2}}) ‖}^{2}$
	$+ \frac{\exp (ω_{0 ℓ_{1}} + ω_{0 ℓ_{2}})}{\exp (ω_{0 ℓ_{1}}) + \exp (ω_{0 ℓ_{2}})} ‖ (𝒂_{ℓ_{1}}, σ_{ℓ_{1}}) - (𝒂_{ℓ_{2}}, σ_{ℓ_{2}}) ‖ .$		(7)

Pick $(i, j) = {\arg \min}_{ℓ_{1} \neq ℓ_{2} \in [K]} 𝖽 (\cdot, \cdot)$ and replace the pair by the softmax-weighted aggregate

$ω_{0, *}$	$= \log (\exp ω_{0 i} + \exp ω_{0 j}),$
$𝝎_{1, *}$	$= \exp (ω_{0 i} - ω_{0, }) 𝝎_{1 i} + \exp (ω_{0 j} - ω_{0, }) 𝝎_{1 j},$
$b_{*}$	$= \exp (ω_{0 i} - ω_{0, }) b_{i} + \exp (ω_{0 j} - ω_{0, }) b_{j},$
$𝒂_{*}$	$= \frac{\exp (ω_{0 i})}{\exp (ω_{0, })} [(𝝎_{1 i} - 𝝎_{1, }) (b_{i} - b_{*}) + 𝒂_{i}]$
	$+ \frac{\exp (ω_{0 j})}{\exp (ω_{0, })} [(𝝎_{1 j} - 𝝎_{1, }) (b_{j} - b_{*}) + 𝒂_{j}],$
$σ_{*}$	$= \frac{\exp (ω_{0 i})}{\exp (ω_{0, })} [{(b_{i} - b_{})}^{2} + σ_{i}]$
	$+ \frac{\exp (ω_{0 j})}{\exp (ω_{0, })} [{(b_{j} - b_{})}^{2} + σ_{j}] .$	(8)

Then we define $G^{(K - 1)} = \exp (ω_{0, *}) δ_{(𝝎_{1, *}, 𝒂_{*}, b_{*}, σ_{*})} + \sum_{k \neq i, j} \exp (ω_{0 k}) δ_{(𝝎_{1 k}, 𝒂_{k}, b_{k}, σ_{k})}$ . A description of the whole procedure can be seen in Algorithm 1. The choice of merging atoms and deriving the new atom (eqs. 7 and 8) are in particular faithful to hierarchical clustering and $K$ -means algorithms.

3.4 The Hierarchical View of Aggregation Path

Transition and Main Idea. The merge step converts local redundancy into a single effective atom. Repeating it induces a global hierarchy, the aggregation path, along which our new analysis proves a monotone strengthening of the loss and, crucially, fast convergence at every level. This bridges Section 2 (unified loss but slow rates) with a constructive, data-driven path that achieves the same near-parametric behavior after aggregation.

Having presented the algorithm to choose and merge a mixing measure with $K$ atoms to $K - 1$ atoms, we now describe the dendrogram (hierarchical aggregation) of $G$ that emerges by repeatedly applying the merging procedure.

Dendrogram (Hierarchical Aggregation). Iterate the merge in eqs. 7 and 8 from $κ = K$ down to $2$ , generating ${G^{(κ)}}_{κ = 2}^{K}$ . Define the dendrogram $𝒯 (G) = (𝒱, ℰ, ℋ)$ with $𝒱$ containing $K$ levels, the $κ$ -th level holding the atoms of $G^{(κ)}$ , $ℰ$ storing the links between merged pairs across adjacent levels, and $ℋ = (0 p t^{(K)}, \dots, 0 p t^{(2)})$ with $0 p t^{(κ)} := \min {𝖽 (\cdot, \cdot) over pairs in G^{(κ)}} .$ The quantity $0 p t^{(κ)}$ is the height between levels $κ$ and $κ - 1$ .

When we represent $𝒯 (G)$ on a graph, $0 p t^{(κ)}$ is the height between $κ$ -th level and $(κ - 1)$ -th level. The procedure to construct the dendrogram of $G$ is given by Algorithm 2.

Algorithm 1 SGMoE Merge Step (Fast-Rate-Aware)

1:

G^{(κ)} = \sum_{k = 1}^{κ} \exp (ω_{0 ​ k}) ​ δ_{(𝝎_{1 ​ k}, 𝒂_{k}, b_{k}, σ_{k})}

2:

(i, j) \leftarrow \arg ​ \min {𝖽 ​ (\exp (ω_{0 ​ ℓ_{1}}) ​ δ_{𝜽_{ℓ_{1}}}, \exp (ω_{0 ​ ℓ_{2}}) ​ δ_{𝜽_{ℓ_{2}}}) : ℓ_{1} \neq ℓ_{2} \in [κ]}

3:Compute

(ω_{0, *}, 𝝎_{1, *}, 𝒂_{*}, b_{*}, σ_{*})

by eq. 8

4:return

G^{(κ - 1)} = \exp (ω_{0, *}) ​ δ_{(𝝎_{1, *}, 𝒂_{*}, b_{*}, σ_{*})} + \sum_{k \neq i, j} \exp (ω_{0 ​ k}) ​ δ_{𝜽_{k}}

Algorithm 2 SGMoE Hierarchical Aggregation Path

1:

G^{(K)} = \sum_{k = 1}^{K} \exp (ω_{0 ​ k}) ​ δ_{(𝝎_{1 ​ k}, 𝒂_{k}, b_{k}, σ_{k})}

2:Initialize

𝒯 ​ (G) = (𝒱, ℰ, ℋ)

with

𝒱_{K} = {atoms of ​ G^{(K)}}

,

ℰ = \emptyset

3:for

κ = K, \dots, 2

do

4:

G^{(κ - 1)} \leftarrow Algorithm 1 ​ (G^{(κ)})

5: Append atoms of

G^{(κ - 1)}

to level

𝒱_{κ - 1}

, link merged pair in

ℰ

6:

0 ​ p ​ t^{(κ)} \leftarrow \min 𝖽 ​ (\cdot, \cdot)

over pairs in

G^{(κ)}

; append to

ℋ

7:end for

8:return

𝒯 ​ (G) = (𝒱, ℰ, ℋ)

and

{G^{(κ)}}_{κ = 1}^{K}

Monotone Strengthening of the Loss (Bridge to Fast Rates). The following lemma formalizes that each merge step cannot increase our fast-rate-aware distance to $G_{0}$ , making the path progressively easier to estimate:

Lemma 1.

As $D_{FRA} (G^{(K)}, G_{0}) \to 0$ , $D_{FRA} (G^{(K)}, G_{0}) ≳ D_{FRA} (G^{(K - 1)}, G_{0}) ≳ \dots ≳ D_{FRA} (G^{(K_{0})}, G_{0}),$ with constants depending only on $G_{0}$ , $𝚯$ , and $K$ .

Behavior of the Path for the MLE (Main Fast-Rate Theorem). Leveraging the monotonicity above together with the unified inverse bound from Section 2, we obtain fast rates at every level of the path, including the exact-fit and under-fit levels where aggregation recovers optimal parametric rate behavior:

Theorem 1 (Fast convergence rates along the path).

There exist universal constants $C_{1}^{'}, c_{1}, C_{2}^{'}, c_{2} > 0$ such that for all $κ \in [K_{0} + 1, K]$ and $κ^{'} \in [K_{0}]$ , we have

	$ℙ (D_{FRA} ({\hat{G}}_{N}^{(κ)}, G_{0}) > C_{2}^{'} {(\log N / N)}^{1 / 2}) ≲ e^{- c_{2} \log N},$		(9)
	$ℙ (D_{E} ({\hat{G}}_{N}^{(κ^{'})}, G_{0}^{(κ^{'})}) > C_{1}^{'} {(\log N / N)}^{1 / 2}) ≲ e^{- c_{1} \log N} .$

3.5 Heights and Likelihood Along the Path

Transition from Structure to Statistics. Heights summarize structural redundancy; likelihood captures statistical fit. Our second set of novel guarantees shows (i) heights shrink at a rate dictated by $\bar{r} ({\hat{G}}_{N}) := \max_{k \in [K_{0}]} \bar{r} (| 𝔸_{k} ({\hat{G}}_{N}) |)$ , and (ii) the empirical likelihood concentrates to its population counterpart along the path.

Height Definitions. For all $κ \in [K_{0} + 1, K]$ and $κ^{'} \in [K_{0}]$ , let

	$0 p t_{N}^{(κ)} := \min$	${𝖽 (\exp ({\hat{ω}}_{0 k_{1}}) δ_{{\hat{𝜽}}_{k_{1}}}, \exp ({\hat{ω}}_{0 k_{2}}) δ_{{\hat{𝜽}}_{k_{2}}})$
		$: k_{1} \neq k_{2}, atoms of {\hat{G}}_{N}^{(κ)}},$		(10)

and let $0 p t_{0}^{(κ^{'})}$ be the analogous height on the true path. Then:

Theorem 2 (Height control).

For all $κ \in [K_{0} + 1, K]$ and $κ^{'} \in [K_{0}]$ , $0 p t_{N}^{(κ)} ≲ {(\log N / N)}^{1 / \bar{r} ({\hat{G}}_{N})},$ and

| 0 ​ p ​ t_{N}^{(κ^{'})} - 0 ​ p ​ t_{0}^{(κ^{'})} | ≲ {(\log N / N)}^{1 / 2},

with constants depending only on $G_{0}$ , $𝚯$ , and $κ$ .

Likelihood. We define empirical average log-likelihood and population average log-likelihood as follow: ${\bar{ℓ}}_{N} (p_{G}) := N^{- 1} \sum_{n = 1}^{N} \log p_{G} (y_{n} | 𝐱_{n})$ and $ℒ (p_{G}) := 𝔼_{(𝐱, y) \sim P_{G_{0}}} [\log p_{G} (y | 𝒙)]$ .

Condition K. There exist positive constants $c_{α}$ and $c_{β}$ such that for all sufficiently small $ϵ$ and $𝜽_{0}, 𝜽 \in 𝚯$ such that $‖ 𝜽 - 𝜽_{0} ‖ \leq ϵ$ , we have $\log f (𝒙, y | 𝜽) \geq (1 + c_{β} ϵ) \log f (𝒙, y | 𝜽_{0}) - c_{α} ϵ$ .

Theorem 3 (Likelihood concentration on the path).

Assume Condition K hold. Then, for any $κ \in [K_{0} + 1, K]$ , $| {\bar{ℓ}}_{N} (p_{{\hat{G}}_{N}^{(κ)}}) - ℒ (p_{G_{0}}) | ≲ {(\log N / N)}^{1 / (2 \bar{r} ({\hat{G}}_{N}))} .$ Moreover, for $κ^{'} \in [K_{0}]$ , ${\bar{ℓ}}_{N} (p_{{\hat{G}}_{N}^{(κ^{'})}}) \to ℒ (p_{G_{0}^{(κ^{'})}})$ in $ℙ_{G_{0}}$ -probability as $N \to \infty$ .

3.6 Choosing the Number of Experts via a Height-Likelihood Rule

Novel Model Selection Principle. By combining structural signal (heights) and statistical fit (likelihood), our DSC favors models that are both well-separated and well-supported by the data, unlike AIC/BIC/ICL, which ignore the geometry of the fitted atoms.

DSC Definition. For each level $κ$ , define

{DSC}_{N}^{(κ)} := - (0 ​ p ​ t_{N}^{(κ)} + ϵ_{N} ​ {\bar{ℓ}}_{N} ​ (p_{{\hat{G}}_{N}^{(κ)}})),

where the weight $ϵ_{N}$ satifies $1 ≪ ϵ_{N} ≪ {(N / \log N)}^{1 / (2 \bar{r} ({\hat{G}}_{N}))}$ . A practical choice is $ϵ_{N} := \log N$ . Select

{\hat{K}}_{N} := \underset{κ \in [2, K]}{\arg ​ \min} {DSC}_{N}^{(κ)} .

Theorem 4 (Consistency of model selection).

Assume that data are generated by a softmax-gated Gaussian MoE, the parameter space $𝚯$ is compact, the covariate support $𝒳 \subset ℝ^{D}$ is bounded, the DSC uses a penalty $ϵ_{N}$ satisfying as above, and the true component $K_{0} \geq 2$ . Then ${\hat{K}}_{N} \to K_{0}$ in $ℙ_{G_{0}}$ -probability as $N \to \infty$ .

Interpretation. Unlike pure likelihood criteria (AIC/BIC/ICL), ${DSC}_{N}^{(κ)}$ also penalizes structural closeness through $0 p t_{N}^{(κ)}$ . Small heights indicate either redundant atoms (near-duplicates) or atoms with tiny softmax weights; both are symptomatic of over-specification. The joint use of heights and likelihood therefore yields a more robust selection rule in SGMoE.

3.7 Proof Sketches

We sketch the proofs of Lemmas 1, 1, 2, 3 and 4, which together establish monotonicity along the dendrogram path and consistency of the dendrogram-based model selection. We first motivate the fast-rate-aware Voronoi distance in eq. 6. When ${\hat{G}}_{N} \to G_{0}$ , over-specification yields Voronoi cells with $| 𝔸_{k}^{N} | > 1$ . Repeatedly merging such atoms eventually makes every cell singleton, which motivates our construction. Using the density decomposition

	$Q_{N}$	$= [\sum_{k = 1}^{K_{0}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} x + ω_{0 k}^{0} + t_{0})]$
		$\times [p_{G_{N}} (y \| 𝒙) - p_{G_{0}} (y \| 𝒙)],$

we analyze the sums over indices with $| 𝔸_{k}^{N} | > 1$ under $1 \leq | ℓ_{1} | + ℓ_{2} \leq 2 \bar{r} (| 𝔸_{k}^{N} |)$ . For clarity, we also consider $(ℓ_{1}, ℓ_{2})$ with $1 \leq | ℓ_{1} | + ℓ_{2} \leq 2$ , which corresponds to $| 𝔸_{k}^{N} | = 1$ . This reasoning leads to the merging algorithm.

Proof Sketch of Lemma 1. Proceed by induction on $κ \in [K_{0}, K]$ and justify $D_{FRA} (G^{(K)}, G_{0}) ≳ D_{FRA} (G^{(K - 1)}, G_{0})$ . As $D_{FRA} (G^{(K)}, G_{0}) \to 0$ , extract a sequence that satisfies $(𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N}) \to (𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})$ and there exist $t_{0} \in ℝ$ , $𝒕_{1} \in ℝ^{D}$ with $\sum_{ℓ \in 𝔸_{k}^{N}} \exp (ω_{0 ℓ}^{N}) \to \exp (ω_{0 k}^{0} + t_{0})$ and $𝝎_{1 ℓ}^{N} \to 𝝎_{1 k}^{0} + 𝒕_{1}$ for all $ℓ \in 𝔸_{k}^{N}$ . The minimizing pair $(ℓ_{1}, ℓ_{2})$ must belong to a common $𝔸_{k}^{N}$ . Using eq. 8 and Jensen’s inequality for the convex maps $z \mapsto {‖ z ‖}^{{\bar{r}}_{k}}$ and $z \mapsto {‖ z ‖}^{{\bar{r}}_{k} / 2}$ , it suffices to show

	$(\exp ω_{0 ℓ_{1}}^{N} + \exp ω_{0 ℓ_{2}}^{N}) {‖ (Δ_{𝒕_{1}} 𝝎_{1 * k}^{N}, Δ b_{* k}^{N}) ‖}^{{\bar{r}}_{k}}$
	$≲ \sum_{j \in {ℓ_{1}, ℓ_{2}}} \exp ω_{0 j}^{N} {‖ (Δ_{𝒕_{1}} 𝝎_{1 j k}^{N}, Δ b_{j k}^{N}) ‖}^{{\bar{r}}_{k}},$
	$(\exp ω_{0 ℓ_{1}}^{N} + \exp ω_{0 ℓ_{2}}^{N}) {‖ (Δ 𝒂_{* k}^{N}, Δ σ_{* k}^{N}) ‖}^{{\bar{r}}_{k} / 2}$
	$≲ \sum_{j \in {ℓ_{1}, ℓ_{2}}} \exp ω_{0 j}^{N} {‖ (Δ 𝒂_{j k}^{N}, Δ σ_{j k}^{N}) ‖}^{{\bar{r}}_{k} / 2},$

which yields the desired monotonicity.

Proof Sketch of Theorem 1. Combine Lemma 1 with an inverse bound for $D_{FRA} ({\hat{G}}_{N}, G_{0})$ . Following Nguyen et al., 2023a, establish

𝔼_{𝐱} [D_{TV} (p_{G} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] ≳ D_{FRA} (G, G_{0}),

and use Proposition 2 in Nguyen et al., 2023a,

𝔼_{𝐱} [D_{h}^{2} (p_{{\hat{G}}_{N}} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] = 𝒪_{ℙ} ({(\log N / N)}^{1 / 2}),

to derive the rate for ${\hat{G}}_{N}$ . Apply Lemma 1 to obtain the bounds for $κ \in [K_{0} + 1, K]$ . For $κ^{'} \in [K_{0}]$ , combine the previous rate with the merging formula to conclude.

Proof Sketch of Theorem 2. Use Theorem 1 and the fact that any merged pair lies in the same Voronoi cell. Inequalities analogous to those in Lemmas 1 and 1 translate parameter rates into height bounds.

Proof Sketch of Theorem 3. Consider three cases. If $κ \geq K_{0}$ , invoke empirical process tools (van de Geer, 2000) and comparisons between Hellinger and Wasserstein distances (Chen, 1995; Villani, 2003, 2009). If $κ = K_{0}$ , combine Theorem 1 with verification that $u (y | 𝒙; 𝝎_{1}, 𝒂, b, σ) := \exp (𝝎_{1}^{⊤} 𝒙) 𝒩 (y | 𝒂^{⊤} 𝒙 + b, σ)$ satisfies Condition K. If $κ < K_{0}$ , conclude via standard convergence arguments.

Finally, Theorem 4 follows from Theorems 2 and 3.

4 SIMULATION STUDIES

We first show that the dendrogram-based merge yields fast convergence of the mixing measure: starting from an over-fitted estimator that converges slowly, the merged estimator approaches the truth quickly. We then assess model selection via DSC against AIC, BIC, and ICL. Unlike these single-shot selectors, the dendrogram offers a hierarchical view of the fitted atoms, clarifying redundancy and structure. All simulations were run in Python 3.12 on a standard Unix-based system.

Numerical Schemes. The ground-truth mixing measure is

	$G_{0}$	$\equiv G_{0} (2) := \sum_{k = 1}^{2} \exp (ω_{0 k}^{0}) δ_{(𝝎_{1 k}^{0}, 𝒂_{k}, 𝒃_{k}, σ_{k})}$
		$= \exp (- 8) δ_{(25, - 20, 15, 0.3)} + \exp (0) δ_{(0, 20, - 5, 0.4)} .$

For each experiment, $N$ varies on a logarithmic grid from $\log_{10} (N_{\min})$ to $\log_{10} (N_{\max})$ , yielding $N_{num}$ sizes in $[N_{\min}, N_{\max}]$ . At each $N$ , we generate $N_{rep}$ datasets from $G_{0}$ and compute the exact-fitted MLE ${\hat{G}}_{N}^{e} \in ℰ_{2}$ and the over-fitted MLE ${\hat{G}}_{N}^{o} \in 𝒪_{4}$ ( $K = 4$ ) using an EM variant of Chamroukhi et al. (2009). EM stops at tolerance $ϵ = 10^{- 6}$ or 2000 iterations. Because the softmax gate in eq. 1 is translation-invariant, we fix a baseline by setting $ω_{0 K_{0}}^{0} = 0$ and $𝝎_{1 K_{0}}^{0} = 0$ .

To stabilize estimation and highlight asymptotics, EM is favorably initialized. For each replication and $(K, K_{0})$ , split $[K]$ into $K_{0}$ disjoint sets $𝕊_{1}, \dots, 𝕊_{K_{0}}$ , each nonempty. For $k \in 𝕊_{t}$ , draw $𝜼_{k}^{0} = (ω_{0 k}^{0}, 𝝎_{1 k}^{0}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})$ from a Gaussian centered at $𝜼_{t}^{0} = (ω_{0 t}^{0}, 𝝎_{1 t}^{0}, 𝒂_{t}^{0}, b_{t}^{0}, σ_{t}^{0})$ with small covariance. After estimating ${\hat{G}}_{N}^{o}$ , apply the merging procedure in Algorithm 2 to obtain ${\hat{G}}_{N}^{m} \in ℰ_{2}$ .

Fast Parameter Estimation via the Dendrogram. We measure accuracy with the Voronoi distance in eq. 6. For the exact-fitted setting, we use $30$ replicates over $100$ sample sizes with $N \in [10^{2}, 5 \times 10^{4}]$ ; for the over-fitted setting, $40$ replicates over $165$ sizes with $N \in [338, 10^{5}]$ ; for the merged estimator, $40$ replicates over $200$ sizes with $N \in [10^{2}, 10^{5}]$ . The average loss and a reference slope $N^{- 1 / 2}$ are shown in Figure 2. Results match Theorem 1: the exact-fitted and merged estimators attain the optimal $N^{- 1 / 2}$ rate toward $G_{0}$ , while merging drives the over-fitted estimator to the exact-fit level. For illustration of Algorithm 2, Figure 1 considers $G_{0} (3)$ as follows:

e^{- 2} ​ δ_{(3, 1, 0, 1)} + e^{1} ​ δ_{(- 3.5, 8, 7, 0.8)} + e^{0} ​ δ_{(0, 3, 5, 0.6)} .

(11)

Model Selection with DSC. We compare DSC to AIC, BIC, and ICL over $32$ sample sizes with $N \in [10^{3}, 5 \times 10^{4}]$ and $N_{rep} = 25$ . For each method, we report the selection frequency of $K_{0}$ and the average selected size (see Figure 3). AIC/BIC/ICL fit a model for each $κ \in [K]$ via EM and pick the best by the corresponding criterion. DSC fits a single SGMoE with $K = 4$ , builds its dendrogram, and evaluates the criterion with $ω_{N} = \log N$ (Section 3.6). AIC tends to overestimate at small $N$ , while all methods recover $K_{0}$ for large $N$ .

Misspecified Regime. We study $ϵ$ -contamination with $p_{0} = (1 - ϵ) p_{G_{0}} + ϵ q$ , where $q$ is Laplace $(0, 1)$ . Figure 4(a) shows the contaminated sample ( $n = 5000$ ). Figures 4(b) and 4(c) report the proportion of correct selections and the average selected size. AIC/BIC/ICL behave similarly: they may find $K_{0} = 2$ at small $N$ , but tend to overselect as $N$ grows, indicating sensitivity to contamination. DSC, leveraging dendrogram structure, is more robust and continues to select $K_{0}$ with non-negligible frequency even at large $N$ .

5 REAL DATA APPLICATION

We illustrate the dendrogram of mixing measures obtained from our SGMoE model using a real dataset from the study in Blein-Nicolas et al. (2024). The data originate from a large-scale experiment on maize aimed at understanding the genetic and molecular bases of drought-responsive traits from proteins expressed in the leaf (Prado et al., 2018; Blein-Nicolas et al., 2020), where 254 genotypes representing the genetic diversity of dent maize were grown under two watering conditions and phenotyped for seven ecophysiological traits.

After preprocessing and removing missing data as described in Blein-Nicolas et al. (2024), the final dataset consists of 233 maize genotypes ( $N = 233$ ), two ecophysiological traits (outputs), which are water use (WU) and the proteins quantified under the water deficit (WD) condition, and 973 protein variables (inputs, $D = 973$ ). To reduce dimensionality and remove irrelevant features, we apply a Lasso procedure to select $D = 10$ protein variables most associated with the target trait and primarily focus on the ecophysiological trait WU.

We then fit the SGMoE model with $K = 20$ clusters. To ensure a more robust initialization, we first cluster the data into 20 groups using the K-Means algorithm. The resulting cluster assignments are then used to initialize the gating and expert parameters of the SGMoE model, providing a stable starting point for the subsequent steps of the EM algorithm.

Figure 5(a) displays the dendrogram of the fitted mixing measure obtained by Algorithm 2, which reveals the hierarchical structure underlying the data. In this experiment, both BIC and ICL select a single component, while DSC selects 2 components, and AIC overestimates with 18 components. The corresponding heights and average log-likelihoods across levels are shown in Figure 5(b) and Figure 5(c), respectively. We observe that the merging heights generally decrease and approach zero, while the average log-likelihood stabilizes in a few initial levels. Notably, the height at level 2 is much larger than those at subsequent levels, suggesting that there should be two clusters in the data.

The dendrogram not only facilitates effective model selection but also unveils the hierarchical relationships among mixture components, thereby enhancing the interpretability of the estimated parameters in complex biological data settings.

6 CONCLUSION

This work shows that rate-aware geometry, realized through a Voronoi distance together with merging and dendrograms of mixing measures, delivers both fast parameter estimation and consistent, sweep-free model selection in SGMoE. We hope these ideas spur further advances in structured mixture models and expert architectures. Our analysis assumes linear softmax gates, Gaussian experts, compact $𝚯$ , and bounded covariate support. Extending the theory beyond these settings will require additional regularity and tail controls. Exact values of $\bar{r} (M)$ are known for $M \leq 3$ ; for $M \geq 4$ only lower bounds are available. While our guarantees use these bounds, sharper algebraic results would further tighten rates.

Acknowledgments

This project was funded primarily by the Australian Research Council Centre of Excellence for the Mathematical Analysis of Cellular Systems (CE230100001), which supported TrungTin Nguyen and Christopher Drovandi. Christopher Drovandi was also supported by an Australian Research Council Future Fellowship (FT210100260). Additional support was provided by Vietnam National University Ho Chi Minh City (VNU-HCM) under grant number A2025-18-02. The authors also acknowledge Dr. Dat Do (University of Chicago) for helpful discussions about the dendrogram of mixing measures for mixture models (Do et al., 2024).

References

H. Akaike (1974) A new look at the statistical model identification. IEEE Transactions on Automatic Control 19 (6), pp. 716–723. External Links: Document Cited by: §1. ↩ 1
A. Anandkumar, D. Hsu, and S. M. Kakade (2012) A method of moments for mixture models and hidden markov models. In COLT, Cited by: §1. ↩ 1
S. Balakrishnan, M. J. Wainwright, and B. Yu (2017) Statistical guarantees for the EM algorithm: from population to sample-based analysis. Annals of Statistics 45, pp. 77–120. Cited by: §1. ↩ 1
H. Bao, W. Wang, L. Dong, Q. Liu, O-K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei (2022) VLMo: unified vision-language pre-training with mixture-of-modality-experts. In Advances in Neural Information Processing Systems, Cited by: §1. ↩ 1
M. Berrettini, G. Galimberti, S. Ranciati, and T. B. Murphy (2024) Identifying Brexit voting patterns in the British house of commons: an analysis based on Bayesian mixture models with flexible concomitant covariate effects. Journal of the Royal Statistical Society Series C: Applied Statistics 73 (3), pp. 621–638. External Links: ISSN 0035-9254, Link, Document Cited by: §1. ↩ 1
C. Biernacki, G. Celeux, and G. Govaert (2000) Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence 22 (7), pp. 719–725. External Links: Document Cited by: §1. ↩ 1
M. Blein-Nicolas, E. Devijver, M. Gallopin, and E. Perthame (2024) Nonlinear network-based quantitative trait prediction from biological data. Journal of the Royal Statistical Society Series C: Applied Statistics 73 (3), pp. 796–815. External Links: ISSN 0035-9254, Document, Link, https://academic.oup.com/jrsssc/article-pdf/73/3/796/58185677/qlae012.pdf Cited by: Appendix A, Appendix A, Appendix A, §5, §5. ↩ 5 A
M. Blein-Nicolas, S. S. Negro, T. Balliau, C. Welcker, L. Cabrera-Bosquet, S. D. Nicolas, A. Charcosset, and M. Zivy (2020) A systems genetics approach reveals environment-dependent associations between snps, protein coexpression, and drought-related traits in maize. Genome Research 30 (11), pp. 1593–1604. Cited by: Appendix A, §5. ↩ 5 A
F. Chamroukhi, A. Samé, G. Govaert, and P. Aknin (2009) Time series modeling by a regression approach based on a latent process. Neural Networks 22 (5–6), pp. 593–602. External Links: ISSN 0893-6080, Link, Document Cited by: §4. ↩ 4
J. Chen (1995) Optimal Rate of Convergence for Finite Mixture Models. The Annals of Statistics 23 (1), pp. 221 – 233. Note: Publisher: Institute of Mathematical Statistics External Links: Link, Document Cited by: §1, §3.7. ↩ 1 3.7
Z. Chen, Y. Deng, Y. Wu, Q. Gu, and Y. Li (2022) Towards Understanding the Mixture-of-Experts Layer in Deep Learning. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: §1. ↩ 1
M. C. Chong, H. D. Nguyen, and TrungTin Nguyen (2024) Risk Bounds for Mixture Density Estimation on Compact Domains via the h-Lifted Kullback–Leibler Divergence. Transactions on Machine Learning Research. Note: Accept with minor revision External Links: Link Cited by: §1. ↩ 1
D. Dai, C. Deng, C. Zhao, R.x. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y.k. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang (2024) DeepSeekMoE: towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 1280–1297. External Links: Link, Document Cited by: Appendix B. ↩ B
D. Do, L. Do, S. A. McKinley, J. Terhorst, and X. Nguyen (2024) Dendrogram of mixing measures: Learning latent hierarchy and model selection for finite mixture models. arXiv preprint arXiv:2403.01684. Cited by: Appendix B, §1, §6. ↩ 1 6 B
D. Do, L. Do, and X. Nguyen (2025) Strong identifiability and parameter learning in regression with heterogeneous response. Electronic Journal of Statistics 19 (1), pp. 131 – 203. Note: Publisher: Institute of Mathematical Statistics and Bernoulli Society External Links: Link, Document Cited by: §1. ↩ 1
T. G. Do, H. K. Le, T. Nguyen, Q. Pham, B. T. Nguyen, T. Doan, C. Liu, S. Ramasamy, X. Li, and S. HOI (2023) HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore. Cited by: Appendix B, §1. ↩ 1 B
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, External Links: Link Cited by: §1. ↩ 1
N. Doss, Y. Wu, P. Yang, and H. H. Zhou (2023) Optimal estimation of high-dimensional Gaussian location mixtures. The Annals of Statistics 51 (1), pp. 62 – 95. Note: Publisher: Institute of Mathematical Statistics External Links: Link, Document Cited by: §1. ↩ 1
R. Dwivedi, N. Ho, K. Khamaru, M. J. Wainwright, M. I. Jordan, and B. Yu (2020a) Sharp analysis of expectation-maximization for weakly identifiable models. AISTATS. Cited by: §1. ↩ 1
R. Dwivedi, N. Ho, K. Khamaru, M. J. Wainwright, M. I. Jordan, and B. Yu (2020b) Singularity, misspecification, and the convergence rate of EM. Annals of Statistics 44, pp. 2726–2755. Cited by: §1. ↩ 1
D. Eigen, M. Ranzato, and I. Sutskever (2014) Learning factored representations in a deep mixture of experts. In ICLR Workshops, Cited by: §1. ↩ 1
W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, pp. 1–39. Cited by: Appendix B, §1. ↩ 1 B
F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel (2022a) Mixture of expert posterior surrogates for approximate Bayesian computation. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique (SFdS), Lyon, France. Cited by: §1. ↩ 1
F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel (2022b) Summary statistics and discrepancy measures for approximate Bayesian computation via surrogate posteriors. Statistics and Computing 32 (5), pp. 85. External Links: ISSN 1573-1375, Link, Document Cited by: §1. ↩ 1
R. Foygel and M. Drton (2010) Extended Bayesian Information Criteria for Gaussian Graphical Models. In Advances in Neural Information Processing Systems, J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (Eds.), Vol. 23. External Links: Link Cited by: §1. ↩ 1
S. Frühwirth-Schnatter, C. Pamminger, A. Weber, and R. Winter-Ebmer (2012) Labor market entry and earnings dynamics: Bayesian inference using mixtures-of-experts Markov chain clustering. Journal of Applied Econometrics 27 (7), pp. 1116–1137. External Links: Link, Document Cited by: §1. ↩ 1
S. Frühwirth-Schnatter, S. Pittner, A. Weber, and R. Winter-Ebmer (2018) Analysing plant closure effects using time-varying mixture-of-experts Markov chain clustering. The Annals of Applied Statistics 12 (3), pp. 1796 – 1830. Note: Publisher: Institute of Mathematical Statistics External Links: Link, Document Cited by: §1. ↩ 1
S. Frühwirth-Schnatter (2019) Keeping the balance—Bridge sampling for marginal likelihood estimation in finite mixture, mixture of experts and Markov mixture models. Brazilian Journal of Probability and Statistics 33 (4), pp. 706 – 733. External Links: Link, Document Cited by: §1. ↩ 1
C. R. Genovese and L. Wasserman (2000) Rates of convergence for the Gaussian mixture sieve. The Annals of Statistics 28 (4), pp. 1105 – 1127. External Links: Link, Document Cited by: §1. ↩ 1
A. Guha, N. Ho, and X. Nguyen (2021) On posterior contraction of parameters and interpretability in Bayesian mixture modeling. Bernoulli 27 (4), pp. 2159 – 2188. External Links: Link, Document Cited by: §1. ↩ 1
M. Hardt and E. Price (2015) Tight bounds for learning a mixture of two gaussians. In STOC, Cited by: §1. ↩ 1
P. Heinrich and J. Kahn (2018) Strong identifiability and optimal minimax rates for finite mixture estimation. The Annals of Statistics 46 (6), pp. 2844–2870. Cited by: §1. ↩ 1
B. H. Ho, L. N. Chi, T. Nguyen, V. H. Hoang, B. T. Nguyen, and C. Drovandi (2025) A Unified Framework for Variable Selection in Model-Based Clustering with Missing Not at Random. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1. ↩ 1
N. Ho and X. Nguyen (2016a) Convergence rates of parameter estimation for some weakly identifiable finite mixtures. The Annals of Statistics 44 (6), pp. 2726 – 2755. External Links: Link, Document Cited by: §F.4, §1, §1, Fact 7. ↩ 1 F.4
N. Ho and X. Nguyen (2016b) On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics 10 (1), pp. 271–307. Cited by: §1. ↩ 1
N. Ho, C. Yang, and M. I. Jordan (2022) Convergence Rates for Gaussian Mixtures of Experts. Journal of Machine Learning Research 23 (323), pp. 1–81. External Links: Link Cited by: §1, §1. ↩ 1
R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton (1991) Adaptive mixtures of local experts. Neural computation 3 (1), pp. 79–87. Cited by: §1. ↩ 1
W. Jiang and M. A. Tanner (1999) On the identifiability of mixtures-of-experts. Neural Networks 12 (9), pp. 1253–1258. External Links: ISSN 0893-6080, Link, Document Cited by: §F.2. ↩ F.2
M. I. Jordan and R. A. Jacobs (1994) Hierarchical mixtures of experts and the EM algorithm. Neural computation 6 (2), pp. 181–214. Cited by: §1. ↩ 1
A. Khalili, A. Y. Yang, and X. Da (2024) Estimation and group-feature selection in sparse mixture-of-experts with diverging number of parameters. Journal of Statistical Planning and Inference, pp. 106250. External Links: ISSN 0378-3758, Link, Document Cited by: §1. ↩ 1
H. Liang, Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, and Z. Wang (2022) M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design. In NeurIPS, Cited by: §1. ↩ 1
T. Manole and N. Ho (2020) Uniform convergence rates for maximum likelihood estimation under two-component gaussian mixture models. arXiv preprint arXiv:2006.00704. Cited by: §1. ↩ 1
L. Montuelle and E. Le Pennec (2014) Mixture of Gaussian regressions model with logistic weights, a penalized maximum likelihood approach. Electronic Journal of Statistics 8 (1), pp. 1661–1695. Cited by: §1. ↩ 1
D. N. Nguyen and Z. Li (2024) Joint learning of Gaussian graphical models in heterogeneous dependencies of high-dimensional transcriptomic data. In The 16th Asian Conference on Machine Learning (Conference Track), External Links: Link Cited by: §1. ↩ 1
H. D. Nguyen, F. Chamroukhi, and F. Forbes (2019) Approximation results regarding the multiple-output Gaussian gated mixture of linear experts model. Neurocomputing 366, pp. 208–214. External Links: ISSN 0925-2312, Link, Document Cited by: §1. ↩ 1
H. D. Nguyen and F. Chamroukhi (2018) Practical and theoretical aspects of mixture-of-experts modeling: An overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8 (4), pp. e1246. Cited by: §1. ↩ 1
H. D. Nguyen, L. R. Lloyd-Jones, and G. J. McLachlan (2016) A universal approximation theorem for mixture-of-experts models. Neural computation 28 (12), pp. 2585–2593. Cited by: §1. ↩ 1
H. D. Nguyen, M. Gupta, J. Westerhout, and T. Nguyen (2025a) On the large-sample limits of some Bayesian model evaluation statistics. arXiv preprint arXiv:2502.03846. Cited by: §1. ↩ 1
H. D. Nguyen, T. Nguyen, F. Chamroukhi, and G. J. McLachlan (2021a) Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models. Journal of Statistical Distributions and Applications 8 (1), pp. 13. External Links: ISSN 2195-5832, Link, Document Cited by: §1. ↩ 1
H. D. Nguyen, T. Nguyen, J. Westerhout, and X. Guo (2025b) Approximation rates for finite mixtures of location-scale models. arXiv preprint arXiv:2508.10612. Cited by: §1. ↩ 1
H. D. Nguyen and T. Nguyen (2025) Modifications of the BIC for order selection in finite mixture models. 2506.20124. External Links: Link Cited by: §1. ↩ 1
H. Nguyen, P. Akbarian, T. Nguyen, and N. Ho (2024a) A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts. In Proceedings of The 41st International Conference on Machine Learning, Cited by: §1. ↩ 1
H. Nguyen, P. Akbarian, F. Yan, and N. Ho (2024b) Statistical perspective of top-k sparse softmax gating mixture of experts. External Links: 2309.13850, Link Cited by: Appendix B. ↩ B
H. Nguyen, T. Nguyen, and N. Ho (2023a) Demystifying Softmax Gating Function in Gaussian Mixture of Experts. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, pp. 4624–4652. Cited by: Appendix B, Appendix C, Appendix D, §E.2, §E.2, §F.2, §F.2, §F.4, §1, §1, §1, §2, §2, §3.7, §3.7, Fact 1, Fact 2, Fact 3, Fact 4, Fact 5. ↩ 1 2 3.7 B C D E.2 F.2 F.4
H. Nguyen, T. Nguyen, K. Nguyen, and N. Ho (2024c) Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, S. Dasgupta, S. Mandt, and Y. Li (Eds.), Proceedings of Machine Learning Research, Vol. 238, pp. 2683–2691. External Links: Link Cited by: §1. ↩ 1
T. Nguyen, F. Chamroukhi, H. D. Nguyen, and G. J. McLachlan (2023b) Approximation of probability density functions via location-scale finite mixtures in Lebesgue spaces. Communications in Statistics - Theory and Methods 52 (14), pp. 5048–5059. External Links: Link, Document Cited by: §1. ↩ 1
T. Nguyen, F. Chamroukhi, H. D. Nguyen, and F. Forbes (2021b) Non-asymptotic model selection in block-diagonal mixture of polynomial experts models. Preprint. arXiv:2104.08959. External Links: Link Cited by: §1. ↩ 1
T. Nguyen, F. Chamroukhi, H. D. Nguyen, and F. Forbes (2022a) Model selection by penalization in mixture of experts models with a non-asymptotic approach. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique (SFdS), Lyon, France. Cited by: §1. ↩ 1
T. Nguyen, F. Forbes, J. Arbel, and H. Duy Nguyen (2024d) Bayesian nonparametric mixture of experts for inverse problems. Journal of Nonparametric Statistics, pp. 1–60. Note: doi: 10.1080/10485252.2024.2426091 External Links: ISSN 1048-5252, Link, Document Cited by: §1. ↩ 1
T. Nguyen, D. N. Nguyen, H. D. Nguyen, and F. Chamroukhi (2023c) A non-asymptotic theory for model selection in high-dimensional mixture of experts via joint rank and variable selection. In AJCAI Australasian Joint Conference on Artificial Intelligence 2023, External Links: Link Cited by: §1. ↩ 1
T. Nguyen, H. D. Nguyen, F. Chamroukhi, and G. J. McLachlan (2023d) Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts. arXiv:2009.10622. External Links: Link Cited by: §1. ↩ 1
T. Nguyen, H. D. Nguyen, F. Chamroukhi, and F. Forbes (2022b) A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models. Electronic Journal of Statistics 16 (2), pp. 4742 – 4822. External Links: Link, Document Cited by: §1. ↩ 1
T. Nguyen, H. D. Nguyen, F. Chamroukhi, and G. J. McLachlan (2020) Approximation by finite mixtures of continuous density functions that vanish at infinity. Cogent Mathematics & Statistics 7 (1), pp. 1750861. External Links: ISSN null, Link, Document Cited by: §1. ↩ 1
T. Nguyen (2021) Model Selection and Approximation in High-dimensional Mixtures of Experts Models: from Theory to Practice. PhD Thesis, Normandie Université, (en). External Links: Link Cited by: §1. ↩ 1
X. Nguyen (2013) Convergence of latent mixing measures in finite and infinite mixture models. The Annals of Statistics 41 (1), pp. 370–400. External Links: Link, Document Cited by: §1, §1. ↩ 1
A. Norets (2010) Approximation of conditional densities by smooth mixtures of regressions. The Annals of Statistics 38 (3), pp. 1733 – 1766. External Links: Link, Document Cited by: §1. ↩ 1
F. Peng, R. A. Jacobs, and M. A. Tanner (1996) Bayesian Inference in Mixtures-of-Experts and Hierarchical Mixtures-of-Experts Models With an Application to Speech Recognition. Journal of the American Statistical Association 91 (435), pp. 953–960. External Links: ISSN 01621459 Cited by: §1. ↩ 1
Q. Pham, G. Do, H. Nguyen, T. Nguyen, C. Liu, M. Sartipi, B. T. Nguyen, S. Ramasamy, X. Li, S. Hoi, et al. (2024) CompeteSMoE–Effective Training of Sparse Mixture of Experts via Competition. arXiv preprint arXiv:2402.02526. Cited by: Appendix B, §1. ↩ 1 B
S. A. Prado, L. Cabrera-Bosquet, A. Grau, A. Coupel-Ledru, E. J. Millet, C. Welcker, and F. Tardieu (2018) Phenomics allows identification of genomic regions affecting maize stomatal conductance with conditional effects of water deficit and evaporative demand. Plant, Cell & Environment 41 (2), pp. 314–326. Cited by: Appendix A, §5. ↩ 5 A
A. Rakhlin, D. Panchenko, and S. Mukherjee (2005) Risk bounds for mixture density estimation. ESAIM: PS 9, pp. 220–229. External Links: Link, Document Cited by: §1. ↩ 1
G. Schwarz (1978) Estimating the dimension of a model. The Annals of Statistics 6 (2), pp. 461–464. Cited by: §1. ↩ 1
W. Shen, S. T. Tokdar, and S. Ghosal (2013) Adaptive Bayesian multivariate density estimation with Dirichlet mixtures. Biometrika 100 (3), pp. 623–640. Cited by: §1. ↩ 1
C. Sin and H. White (1996) Information criteria for selecting possibly misspecified parametric models. Journal of Econometrics 71 (1), pp. 207–225. External Links: ISSN 0304-4076, Link, Document Cited by: §1. ↩ 1
B. Sturmfels (2002) Solving systems of polynomial equations. CBMS Regional Conference Series in Mathematics, American Mathematical Soc.. Cited by: §2. ↩ 2
T. Thai, T. Nguyen, D. Do, N. Ho, and C. Drovandi (2025) Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures. arXiv preprint arXiv:2505.13052. Cited by: Appendix B, Appendix B, §1. ↩ 1 B
T. Tran, T. Nguyen, M. A. Bashar, N. Ho, R. Nayak, and C. Drovandi (2026a) Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models. Note: _eprint: 2602.07997 External Links: Link Cited by: §1. ↩ 1
T. Tran, T. Nguyen, G. Fort, T. Doan, H. D. Nguyen, B. T. Nguyen, F. Forbes, and C. Drovandi (2026b) Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts. arXiv preprint arXiv:2601.19811. Cited by: §1. ↩ 1
S. van de Geer (2000) Empirical Processes in M-estimation. Vol. 6, Cambridge university press. Cited by: §E.4, §3.7, Fact 6. ↩ 3.7 E.4 F.4
C. Villani (2003) Topics in optimal transportation. Graduate Studies in Mathematics, Vol. 58, American Mathematical Society. External Links: ISBN 0-8218-3312-X Cited by: §3.7. ↩ 3.7
C. Villani (2009) Optimal transport: old and new. Vol. 338, Springer. Cited by: §3.7. ↩ 3.7
J. Westerhout, T. Nguyen, X. Guo, and H. D. Nguyen (2024) On the Asymptotic Distribution of the Minimum Empirical Risk. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: §1. ↩ 1
Y. Wu and H. H. Zhou (2021) Randomly initialized EM algorithm for two-component Gaussian mixture achieves near optimality in $O (\sqrt{n})$ iterations. Mathematical Statistics and Learning 4, pp. 143–220. Cited by: §1. ↩ 1
Y. Wu and P. Yang (2020) Optimal estimation of Gaussian mixtures via denoised method of moments. The Annals of Statistics 48, pp. 1987–2007. Cited by: §1. ↩ 1
Z. You, S. Feng, D. Su, and D. Yu (2021) SpeechMoE: scaling to large acoustic models with dynamic routing mixture of experts. In Interspeech, Cited by: §1. ↩ 1
Z. You, S. Feng, D. Su, and D. Yu (2022) Speechmoe2: mixture-of-experts model with improved routing. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7217–7221. External Links: Document Cited by: §1. ↩ 1
S. E. Yuksel, J. N. Wilson, and P. D. Gader (2012) Twenty Years of Mixture of Experts. IEEE Transactions on Neural Networks and Learning Systems 23 (8), pp. 1177–1193. External Links: ISSN 2162-2388 VO - 23, Document Cited by: §1. ↩ 1
G. Zens (2019) Bayesian shrinkage in mixture-of-experts models: identifying robust determinants of class membership. Advances in Data Analysis and Classification 13 (4), pp. 1019–1051. External Links: ISSN 1862-5355, Link, Document Cited by: §1. ↩ 1

Appendix A ADDITIONAL DETAILS ON THE MAIZE DROUGHT-RESPONSE DATA

This appendix provides additional biological context and a more explicit description of the data-processing pipeline used for the real-data illustration in the main paper. The goal is to clarify the provenance of the dataset, the meaning of the response and predictor variables, and the rationale for the preprocessing choices, while avoiding repetition of the main-text discussion.

Biological motivation and data provenance.

The dataset comes from a broader systems-genetics effort on dent maize aimed at linking molecular variation in the leaf proteome to drought-related ecophysiological traits. In that line of work, a genetically diverse maize panel was grown under contrasting watering conditions and characterised using both high-throughput phenotyping and proteomics, with the broader objective of understanding how genotype-dependent molecular responses are related to drought adaptation and plant water-use behaviour (Prado et al., 2018; Blein-Nicolas et al., 2020). The statistical prediction study of Blein-Nicolas et al. (2024) used these biological measurements as a benchmark for multivariate trait prediction from high-dimensional proteomic covariates, specifically considering drought-related traits measured on a panel of 233 maize genotypes with 973 protein predictors (Blein-Nicolas et al., 2024). Systems-genetics reports associated with the same experimental programme also emphasise that the maize data were designed to study drought-related traits by integrating proteomic and genomic information (Blein-Nicolas et al., 2020).

Why this dataset is relevant here.

This dataset is well suited to our SGMoE framework for three reasons. First, the sample is biologically heterogeneous: the maize panel spans substantial genetic diversity, so one should not expect all genotypes to follow a single homogeneous regression relationship. Second, drought response is known to be multi-mechanistic, with different molecular programmes potentially associated with different water-use strategies or stress-response profiles. Third, the predictor space is high-dimensional relative to the sample size, which makes model structure and interpretability especially important. These features make the dataset a natural test bed for a method that combines flexible conditional modelling with hierarchical aggregation and model selection. The resulting fitted components can then be interpreted as latent subgroups of genotypes sharing similar proteomic-to-phenotypic relationships rather than as merely algorithmic clusters.

Raw variables used in the illustration.

Following Blein-Nicolas et al. (2024), we work with a cleaned subset of the original experiment after removing observations with missing values. The final analysis set contains $N = 233$ maize genotypes. The biological study recorded two drought-related ecophysiological outputs together with quantitative protein abundances measured under water-deficit conditions, yielding a predictor matrix with 973 protein variables before dimension reduction. In the present illustration, we focus primarily on the ecophysiological trait water use (WU), while the predictors are the leaf protein abundances measured under the water-deficit regime. This choice is scientifically meaningful because WU is directly linked to drought adaptation and integrates the cumulative effect of genotype-specific physiological regulation under stress.

Preprocessing strategy.

The preprocessing follows the protocol used for the statistical benchmark in Blein-Nicolas et al. (2024), with the same starting point of a cleaned matrix after exclusion of incomplete observations. Since the original proteomic representation is very high-dimensional compared with the number of genotypes, we apply a supervised screening step before fitting the SGMoE. Concretely, we use a Lasso-based variable-selection procedure to extract a smaller subset of proteins that are most strongly associated with the target trait, and we retain $D = 10$ proteins for the analysis shown in the main paper. This reduction serves two purposes. Statistically, it improves stability in the small- $N$ , large- $D$ regime and reduces the risk that the fitted experts are driven by noise dimensions. Biologically, it yields a more interpretable model by restricting attention to a compact set of drought-informative protein signals. We stress that this Lasso step is used only as a preprocessing device; the clustering, aggregation path, and model selection are all performed by the SGMoE methodology thereafter.

Model fitting for the SGMoE path.

After preprocessing, we fit an over-specified SGMoE with $K = 20$ initial components. Because mixture models can be sensitive to starting values, we initialise the fit using a preliminary $K$ -means partition of the genotypes. These initial groups are then used to seed the gating and expert parameters before running the estimation procedure. The purpose of this intentionally over-specified fit is not to interpret all 20 initial components literally, but rather to create a rich starting representation from which the dendrogram path can merge redundant atoms and reveal a more stable low-dimensional structure. In this sense, the over-specified fit plays the same exploratory role as in the synthetic studies: it allows the subsequent aggregation path to separate persistent large-scale structure from small within-cell duplications.

Interpretation of the fitted path.

In the main-text illustration, the fitted dendrogram suggests a pronounced split at level $2$ , while the average log-likelihood stabilises quickly along the path. From a biological viewpoint, this pattern is consistent with the idea that the maize panel contains a small number of broad genotype groups with distinct proteomic-response profiles under drought, rather than many sharply separated subpopulations. Thus, the selected two-expert solution should be read as a parsimonious summary of two dominant genotype–phenotype response regimes. The value of the dendrogram is therefore twofold: it provides a data-driven model-selection tool, and it offers a hierarchical view of how more complex over-specified representations collapse into a small number of biologically interpretable regimes.

Why the real-data example is informative for our methodology.

Unlike the synthetic experiments, this dataset does not come with a known ground-truth number of experts. Its role is instead to illustrate the practical behaviour of the pathwise procedure on a genuinely heterogeneous biological problem. In particular, it shows that the dendrogram can remain informative even when standard information criteria disagree strongly, and that the selected solution can still be interpreted in domain terms through genotype–phenotype structure and early likelihood stabilisation. This complements the theory by demonstrating that the SGMoE aggregation path is not only a technical device for proving rates, but also a practically useful summary of heterogeneity in complex omics-assisted prediction problems.

Appendix B OVERVIEW OF MIXTURE AND MOE GEOMETRY

This section provides a unified overview of unconditional mixtures, MoE, and SGMoE, clarifying the geometric and statistical differences that motivate our dendrogram framework.

First, we recall the definitions of unconditional mixtures, MoE, and covariate-free gates.

•

Unconditional mixture:

$p (y) = \sum_{k = 1}^{K} π_{k} \times f (y; 𝜼_{k}) .$
•

MoE:

$p (y ∣ 𝒙) = \sum_{k = 1}^{K} π_{k} (𝒙) \times f (y; 𝜼_{k} (𝒙)),$

where both the weights and the experts depend on $𝒙$ .
•

Covariate-free gates:

$π_{k} (𝒙) \equiv π_{k},$

which reduces to a mixture of regressions when $𝜼_{k} (x)$ is a regression map.

Next, we analyze the difference from existing dendrogram approaches and compare them to Gaussian-gated Gaussian MoE (GGMoE) (Thai et al., 2025).

Difference from existing dendrogram approaches.

Our framework differs from classical dendrogram methods in three key aspects. First, we introduce Voronoi-type losses in the gate space that respect softmax symmetry (common translations). Second, our method is tailored to conditional SGMoE geometry and provides finite-sample predictive parameter rates, building on Nguyen et al. (2023a). Third, earlier dendrogram approaches were developed for unconditional finite mixtures (Do et al., 2024) and rely on standard Wasserstein-type losses; they are not tailored to the conditional geometry and softmax-induced couplings of SGMoE. We introduce a fast-rate-aware Voronoi loss $D_{FRA}$ that (i) reduces to the exact-fit loss when cells are singletons and (ii) adds merged-moment block sums precisely in the slow directions created by Voronoi multi-coverage. This is motivated by the insufficiency of Wasserstein for SGMoE parameter geometry (and even the limitations of early Voronoi losses in Nguyen et al. (2023a)) and is spelled out in our appendix overview and the formal $D_{FRA}$ /merge analysis.

Compare to GGMoE.

GGMoE is a generative MoE that models covariates and gates via Bayes’ rule, enabling closed-form EM M-steps but not matching modern deep MoE practice. Our framework aligns with contemporary discriminative softmax/top- $k$ gating learned end-to-end over features: we directly optimize the conditional density $p (y ∣ 𝒙)$ without a generative model for $𝒙$ (Dai et al., 2024; Nguyen et al., 2024b; Fedus et al., 2022; Pham et al., 2024; Do et al., 2023). This conditional focus aligns with predictive use but is analytically harder: softmax gating introduces a tight numerator–denominator coupling and nontrivial gate–expert interactions that do not arise in GGMoE’s EM updates. Beyond this objective mismatch, we contribute Voronoi-type losses aligned with the gate-induced partition and establish finite-sample MLE convergence rates for SGMoE in both exact-fit and over-specified regimes, addressing the conditional SGMoE geometry directly rather than relying on a generative model for $x$ . Empirically, beyond synthetic studies, we analyze a maize proteomics dataset of drought-responsive traits: the dendrogram-guided SGMoE path selects two experts, stabilizes the likelihood early, reveals a clear hierarchical structure in the mixing measure, and yields interpretable genotype–phenotype mappings, complementing GGMoE-centric work whose experiments are primarily synthetic (Thai et al., 2025).

Appendix C ILLUSTRATION OF VORONOI CELLS AND MERGE STEPS FOR SGMoE

For a candidate mixing measure $G = \sum_{k = 1}^{K} \exp (ω_{0 k}) δ_{(𝝎_{1 k}, 𝒂_{k}, b_{k}, σ_{k})}$ and the true $G_{0} = \sum_{k = 1}^{K_{0}} \exp (ω_{0 k}^{0}) δ_{(𝝎_{1 k}^{0}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})}$ , define, for $k \in [K_{0}]$ , the (parameter-space) Voronoi cell

𝔸_{k} ​ (G) := {ℓ \in [K] : ‖ 𝜽_{ℓ} - 𝜽_{k}^{0} ‖ \leq ‖ 𝜽_{ℓ} - 𝜽_{j}^{0} ‖, \forall j \neq k},

(12)

where $𝜽_{ℓ} := (𝝎_{1 ℓ}, 𝒂_{ℓ}, b_{ℓ}, σ_{ℓ})$ . We use the softmax translation $(t_{0}, 𝒕_{1})$ from identifiability (cf. Proposition 1 of Nguyen et al., 2023a) and the shorthand $Δ_{𝒕_{1}} 𝝎_{1 ℓ k} := 𝝎_{1 ℓ} - 𝝎_{1 k}^{0} - 𝒕_{1}$ , $Δ 𝒂_{ℓ k} := 𝒂_{ℓ} - 𝒂_{k}^{0}$ , $Δ b_{ℓ k} := b_{ℓ} - b_{k}^{0}$ , $Δ σ_{ℓ k} := σ_{ℓ} - σ_{k}^{0}$ . For brevity we write $𝔸_{k}$ for $𝔸_{k} (G)$ . (We restate eq. 12 only for completeness; throughout we reference the main-paper definition eq. 2.)

Explanation. Figure 6 summarizes the geometry and the merge step used by our method for an example with $K_{0} = 6$ and $K = 10$ : red squares denote true atoms of $G_{0}$ , blue circles denote fitted atoms of $G$ . Each Voronoi cell is generated by one true atom, and its cardinality $| 𝔸_{k} |$ equals the number of fitted atoms assigned to that true atom (e.g., two circles in a cell imply $| 𝔸_{k} | = 2$ ). Panel Figure 6(a) shows the Voronoi partition ${𝔸_{k}}_{k \in [K_{0}]}$ induced by $G_{0}$ as in eq. 2. Cells with $| 𝔸_{k} | > 1$ reveal redundancy: multiple fitted atoms approximate the same truth and create slow directions. Panel Figure 6(b) zooms into one such multi-covered cell and depicts the merge step at a visual level: the closest pair (w.r.t. our rate-weighted dissimilarity) is merged into a single aggregate; iterating this operation produces the aggregation path. Panel Figure 6(c) links the visuals to the mathematics: labels “fitted i,” “fitted j,” and “merged *” correspond to $\exp (ω_{0 i}) δ_{𝜽_{i}}$ , $\exp (ω_{0 j}) δ_{𝜽_{j}}$ , and $\exp (ω_{0, *}) δ_{𝜽_{*}}$ . Pair selection uses $𝖽$ from eq. 7, and the softmax-weighted update rules are given in eq. 8. Together, these steps collapse slow directions within a cell, strengthen the loss along the path $D_{FRA}$ (eq. 6), and enable our fast pathwise guarantees and sweep-free model selection via DSC.

Appendix D THEORETICAL CHALLENGES: MORE DETAILS

The geometric picture above motivates the analytic tools below. We now detail three fundamental challenges in the statistical analysis of SGMoE that create substantial obstacles for parameter estimation and model selection:

(i) Softmax translation invariance. Gating parameters are identifiable only up to common translations. Unlike covariate-independent gating functions, the softmax gate is invariant under simultaneous shifts of intercepts and slopes, which makes the parameterization non-unique. As a result, standard identifiability arguments break down, and it becomes necessary to design translation-invariant loss functions. We address this by introducing the Voronoi partition and loss (see eq. 6), which takes an infimum over translations and thereby aligns the loss with the geometry of gating partitions.

(ii) Gate-expert PDE couplings. The likelihood function exhibits intrinsic gate-expert interactions that induce coupled differential relations among parameters. These relations lead to numerous linear dependencies among derivative terms in Taylor expansions, which prevents a direct decomposition of density discrepancies $p_{{\hat{G}}_{N}} (y | 𝒙) - p_{G_{0}} (y | 𝒙)$ into independent components. Moreover, the parameters of the softmax gating numerators and the Gaussian experts are intrinsically linked through explicit PDEs,

\frac{\partial^{2} u}{\partial 𝝎_{1} ​ \partial b} = \frac{\partial u}{\partial 𝒂}, \frac{\partial^{2} u}{\partial b^{2}} = 2 ​ \frac{\partial u}{\partial σ},

(13)

where $u (y | 𝒙; 𝝎, 𝒂, b, σ) := \exp (𝝎_{1}^{⊤} 𝒙) 𝒩 (y | 𝒂^{⊤} 𝒙 + b, σ)$ . Our analysis requires a systematic reorganization of these dependent terms to recover a meaningful set of independent directions.

(iii) Algebraic cancellations. Due to the tight coupling between numerators and denominators in the softmax-induced conditional density, higher-order cancellations in the expansions give rise to systems of polynomial equations introduced in eq. 3. The solvability of these systems determines the order of the first non-vanishing terms and directly controls the convergence rates of the MLE in over-specified models. This algebraic obstruction is a key source of non-standard, slower rates unique to SGMoE.

These challenges indicate that previously used loss functions, such as the Wasserstein distance, are insufficient for analyzing parameter quantities in either standard mixture models or mixtures with covariate-free gating functions. Moreover, the convergence rates of parameter estimates, as reported in Nguyen et al. (2023a), remain relatively slow due to the influence of the associated polynomial systems. Therefore, developing a dedicated method or algorithm, such as our DSC approach in Section 3, for models of this type is well motivated.

In addition, we also clarify the relationship between polynomial equations in eq. 3 and SGMoE in the over - specified case. Following Theorem 1, at $κ = K$ , we can see that the convergence rate of Fast-Rate-Aware Voronoi distance $D_{FRA}$ is

D_{FRA} ({\hat{G}}_{N}, G_{0}) ≲ {(\frac{\log N}{N})}^{1 / 2},

this is an "optimal" rate for a mixing measure. However, the convergence rate of some parameters such as $𝝎_{1}, 𝒂, b, σ$ are not ${(\frac{\log N}{N})}^{1 / 2}$ (following the definition of $D_{FRA}$ ). In particular, in the over-specified case, respectively $| 𝔸_{k} | \geq 2$ , then the associated parameters suffer slower rates of the order $N^{- 1 / (2 \bar{r} (| 𝔸_{k} |))}$ or $N^{- 1 / \bar{r} (| 𝔸_{k} |)}$ (see Table 1).

To explain this connection, we revisit our proof for over-specified case. Firstly, we want to show that $𝔼_{𝐱} [V (p_{G} (\cdot ∣ 𝒙), p_{G_{0}} (\cdot ∣ 𝒙))] ≳ D_{FRA} (G, G_{0})$ because we can see that if we obtain this argument, we will get the "optimal" convergence rate of $D_{FRA}$ . We can rewrite the quantity $Q_{N}$ as follows:

	$Q_{N}$	$= \sum_{k = 1}^{K_{0}} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [u (y \| 𝒙; 𝝎_{1 ℓ}^{N}, 𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N}) - u (y \| 𝒙; 𝝎_{1 k}^{0}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}) - v (y \| 𝒙; 𝝎_{1 ℓ}^{N})$
		$+ v (y \| 𝒙; 𝝎_{1 k}^{0})] + \sum_{k = 1}^{K_{0}} (\sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) - \exp (ω_{0 k}^{0})) [u (y \| 𝒙; ω_{0 k}^{0}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}) - v (y \| 𝒙; 𝝎_{1 k}^{0})],$

where we define $u (y | 𝒙; 𝝎_{1}, 𝒂, b, σ) := \exp (𝝎_{1}^{⊤} 𝒙) 𝒩 (y | 𝒂^{⊤} 𝒙 + b, σ)$ and $v (y | 𝒙; 𝝎_{1}) := \exp (𝝎_{1}^{⊤} 𝒙) p_{G_{N}} (y | 𝒙)$ . Next, for each $k \in [K_{0}]$ and $ℓ \in 𝔸_{k}$ , we denote $h_{1} (𝒙, 𝒂_{k}^{0}, b_{k}^{0}) := {(𝒂_{k}^{0})}^{⊤} 𝒙 + b_{k}^{0}$ and then apply the Taylor expansions to the functions $u (y | 𝒙; 𝝎_{1 ℓ}^{N}, 𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N})$ and $v (y | 𝒙; 𝝎_{1 ℓ}^{N})$ up to orders $r_{1 k}$ and $r_{2 k}$ (which we will choose later), respectively, as follows:

	$u (y \| 𝒙; 𝝎_{1 ℓ}^{N}, 𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N}) - u (y \| 𝒙; 𝝎_{1 k}^{0}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})$
	$= \sum_{\| ℓ_{1} \| + ℓ_{2} = 1}^{2 r_{1 k}} T_{ℓ_{1}, ℓ_{2}}^{N} (k) 𝒙^{ℓ_{1}} \exp ({(𝝎_{1 k}^{0})}^{⊤} 𝒙) \frac{\partial^{ℓ_{2}} 𝒩}{\partial h_{1}^{ℓ_{2}}} (y \| {(𝒂_{k}^{0})}^{⊤} 𝒙 + b_{k}^{0}, σ_{k}^{0}) + R_{1 ℓ k} (𝒙, y),$
	$v (y \| 𝒙; 𝝎_{1 ℓ}^{N}) - v (y \| 𝒙; 𝝎_{1 k}^{0}) = \sum_{\| γ \| = 1}^{r_{2 k}} S_{γ}^{N} (k) 𝒙^{γ} \exp ({(𝝎_{1 k}^{0})}^{⊤} 𝒙) p_{G_{N}} (y \| 𝒙) + R_{2 ℓ k} (𝒙, y),$

where $R_{1 ℓ k} (𝒙, y)$ and $R_{2 ℓ k} (𝒙, y)$ are Taylor remainders such that $R_{ρ ℓ k} (𝒙, y) / D_{FRA} (G_{N}, G_{0})$ vanishes as $N \to \infty$ for $ρ \in {1, 2}$ . As a result, the limit of $Q_{N} / D_{FRA} (G_{N}, G_{0})$ when $n$ goes to infinity can be seen as a linear combination of elements of the following set:

	$𝒲$	$:= {𝒙^{ℓ_{1}} \exp ({(𝝎_{1 k}^{0})}^{⊤} 𝒙) \frac{\partial^{ℓ_{2}} 𝒩}{\partial h_{1}^{ℓ_{2}}} (y \| {(𝒂_{k}^{0})}^{⊤} 𝒙 + b_{k}^{0}, σ_{k}^{0}) : k \in [K_{0}], 0 \leq 2 \| ℓ_{1} \| + ℓ_{2} \leq 2 r_{1 k}}$
		$\cup {𝒙^{γ} \exp ({(𝝎_{1 k}^{0})}^{⊤} 𝒙) p_{G_{0}} (y \| 𝒙) : k \in [K_{0}], 0 \leq \| γ \| \leq r_{2 k}}$

which is shown to be linearly independent. By the Fatou’s lemma, we demonstrate that $Q_{N} / D_{FRA} (G_{N}, G_{0})$ goes to zero as $N \to \infty$ , implying that all the coefficients in the representation of $Q_{N} / D_{FRA} (G_{N}, G_{0})$ , denoted by $T_{ℓ_{1}, ℓ_{2}}^{N} (k) / D_{FRA} (G_{N}, G_{0})$ and $S_{γ}^{N} (k) / D_{FRA} (G_{N}, G_{0})$ , vanish when $N \to \infty$ . Given that result, we aim to select the Taylor orders $r_{1 k}$ and $r_{2 k}$ such that at least one among the limits of $T_{ℓ_{1}, ℓ_{2}}^{N} (k) / D_{FRA} (G_{N}, G_{0})$ and $S_{γ}^{N} (k) / D_{FRA} (G_{N}, G_{0})$ is different from zero, which leads to a contradiction. In the over-specified case, we assume that all the limits of $T_{ℓ_{1}, ℓ_{2}}^{N} (k) / D_{FRA} (G_{N}, G_{0})$ and $S_{γ}^{N} (k) / D_{FRA} (G_{N}, G_{0})$ equal zero. After some steps of considering typical limits as in the previous setting which requires $r_{2 k} = 2$ for all $k \in [K_{0}]$ , we encounter the following system of polynomial equations:

\sum_{ℓ \in 𝔸_{k}} \sum_{(𝜶_{1}, 𝜶_{2}, α_{3}, α_{4}) \in 𝕀_{ℓ_{1}, ℓ_{2}}} \frac{p_{5 ​ ℓ}^{2} ​ p_{1 ​ ℓ}^{𝜶_{1}} ​ p_{2 ​ ℓ}^{𝜶_{2}} ​ p_{3 ​ ℓ}^{α_{3}} ​ p_{4 ​ ℓ}^{α_{4}}}{𝜶_{1}! ​ 𝜶_{2}! ​ α_{3}! ​ α_{4}!} = 0

for all $(ℓ_{1}, ℓ_{2}) \in ℕ^{D} \times ℕ$ such that $0 \leq | ℓ_{1} | \leq r_{1 k}, 0 \leq ℓ_{2} \leq r_{1 k} - | ℓ_{1} |$ and $| ℓ_{1} | + ℓ_{2} \geq 1$ for some $k \in [K_{0}]$ . Due to the construction of this system, it must have at least one non-trivial solution. Therefore, we choose $r_{1 k} = \bar{r} (| 𝔸_{k} |)$ for all $k \in [K_{0}]$ .

To discuss about the value of $\bar{r} (M)$ with $M \geq 2$ in general, by Fact 1, we obtain $\bar{r} (M) = 2 M$ for $M = 2, 3$ and as $M$ increases, so does $\bar{r} (M)$ . Hence, we predict that $\bar{r} (M) = 2 M$ . With this conjecture, we can see that the slow convergence rate of parameter estimation of SGMoE before we apply the merging atoms process.

Appendix E PROOF SKETCHES

In this section we expand the sketches for Lemma 1, Theorems 1, 2 and 3.

Why the $D_{FRA}$ loss in eq. 6?

When ${\hat{G}}_{N} \to G_{0}$ with $K > K_{0}$ , some Voronoi cells $𝔸_{k}$ are multi-covered. The slow directions in $D_{O}$ (with exponents $\bar{r} (| 𝔸_{k} |)$ ) arise from these cells. $D_{FRA}$ augments $D_{O}$ with first-order merged-moment block-sums that vanish when a cell behaves as a single aggregate. Thus $D_{FRA}$ is simultaneously (i) exact-fit consistent, it reduces to $D_{E}$ when $| 𝔸_{k} | = 1$ , and (ii) overfit-aware, penalizing precisely the slow directions that merging removes. In the over-specified case, cells with $| 𝔸_{k} | > 1$ may persist; repeatedly merging atoms within such cells yields singletons and restores first-order behavior. Formally, using the density decomposition

Q_{N} = [\sum_{k = 1}^{K_{0}} \exp ({(𝝎_{1 ​ k}^{0} + 𝒕_{1})}^{⊤} ​ 𝒙 + ω_{0 ​ k}^{0} + t_{0})] \cdot [p_{G_{N}} ​ (y | 𝒙) - p_{G_{0}} ​ (y | 𝒙)],

we analyze the sums over indices with $| 𝔸_{k} | > 1$ under $1 \leq | ℓ_{1} | + ℓ_{2} \leq 2 \bar{r} (| 𝔸_{k} |)$ ; for clarity, we also isolate the case $1 \leq | ℓ_{1} | + ℓ_{2} \leq 2$ , corresponding to $| 𝔸_{k} | = 1$ . This leads directly to the merge operator and the aggregation path.

E.1 Proof Sketch of Lemma 1

We argue for the first merge $G^{(K)} \to G^{(K - 1)}$ ; the rest follows by induction. Assume $D_{FRA} (G^{(K)}, G_{0}) \to 0$ . Then, for the Voronoi partition ${𝔸_{k}}$ , there exist $(t_{0}, 𝒕_{1})$ such that, for every $k$ ,

\sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ​ ℓ}) \to \exp (ω_{0 ​ k}^{0} + t_{0}), (𝝎_{1 ​ ℓ}, 𝒂_{ℓ}, b_{ℓ}, σ_{ℓ}) \to (𝝎_{1 ​ k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}) .

The minimizing pair $(i, j)$ of $𝖽$ must lie in the same cell $𝔸_{k}$ . Let the merged atom be $\exp (ω_{0, *}) δ_{(𝝎_{1, *}, 𝒂_{*}, b_{*}, σ_{*})}$ as in eq. 8. Using the convexity of $z \mapsto {‖ z ‖}^{m}$ for $m \in {\bar{r} (| 𝔸_{k} |), \bar{r} (| 𝔸_{k} |) / 2}$ and the identities implicit in eq. 8, we obtain the two key comparisons

	$(\exp ω_{0 i} + \exp ω_{0 j}) {‖ (Δ_{𝒕_{1}} 𝝎_{1 * k}, Δ b_{* k}) ‖}^{\bar{r} (\| 𝔸_{k} \|)}$	$≲ \sum_{t \in {i, j}} \exp ω_{0 t} {‖ (Δ_{𝒕_{1}} 𝝎_{1 t k}, Δ b_{t k}) ‖}^{\bar{r} (\| 𝔸_{k} \|)},$
	$(\exp ω_{0 i} + \exp ω_{0 j}) {‖ (Δ 𝒂_{* k}, Δ σ_{* k}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2}$	$≲ \sum_{t \in {i, j}} \exp ω_{0 t} {‖ (Δ 𝒂_{t k}, Δ σ_{t k}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2} .$

The block-sum terms in $D_{FRA}$ also decrease since the merged parameters are softmax-weighted averages. Collecting terms yields $D_{FRA} (G^{(K)}, G_{0}) ≳ D_{FRA} (G^{(K - 1)}, G_{0})$ , proving monotonicity.

E.2 Proof Sketch of Theorem 1

(A) Inverse bound.

We first prove an inverse inequality: there exists $C > 0$ depending only on $G_{0}$ and $𝚯$ such that, for any $G \in 𝒪_{K} (𝚯)$ ,

𝔼_{𝐱} [D_{TV} (p_{G} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] \geq C D_{FRA} (G, G_{0}) .

(14)

The proof follows the density decomposition strategy in Nguyen et al. (2023a) but keeps all merged-moment block-sums that define $D_{FRA}$ . Let

Q_{N} ​ (𝒙, y) = [\sum_{k = 1}^{K_{0}} \exp ({(𝝎_{1 ​ k}^{0} + 𝒕_{1})}^{⊤} ​ 𝒙 + ω_{0 ​ k}^{0} + t_{0})] \cdot [p_{G} ​ (y | 𝒙) - p_{G_{0}} ​ (y | 𝒙)] .

A multi-index Taylor expansion (around $(𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})$ within each cell $𝔸_{k}$ ) up to order $\bar{r} (| 𝔸_{k} |)$ , together with the PDE identities $\partial^{2} u / \partial 𝝎_{1} \partial b = \partial u / \partial 𝒂$ and $\partial^{2} u / \partial b^{2} = 2 \partial u / \partial σ$ , rewrites $Q_{N}$ as a linear combination of basis functions

𝒙^{ℓ_{1}} ​ \exp ({(𝝎_{1 ​ k}^{0} + 𝒕_{1})}^{⊤} ​ 𝒙) ​ \frac{\partial^{ℓ_{2}}}{\partial h_{1}^{ℓ_{2}}} ​ 𝒩 ​ (y | 𝒂_{k}^{0 ⊤} ​ 𝒙 + b_{k}^{0}, σ_{k}^{0}), 1 \leq | ℓ_{1} | + ℓ_{2} \leq 2 ​ \bar{r} ​ (| 𝔸_{k} |),

with coefficients that are precisely the atomwise sums appearing in $D_{FRA}$ (up to constants). If eq. 14 failed, all these coefficients would have to vanish at a rate faster than $D_{FRA} (G, G_{0})$ , forcing a non-trivial solution to the polynomial system of eq. 3, in contradiction with the definition of $\bar{r} (\cdot)$ (Fact 1). This yields eq. 14.

(B) Applying density rates.

By Proposition 2 of Nguyen et al. (2023a), $𝔼_{𝐱} [D_{h}^{2} (p_{{\hat{G}}_{N}} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] = 𝒪_{ℙ} ({(\log N / N)}^{1 / 2}) .$ Using the inequality $D_{TV} \leq \sqrt{2} D_{h}^{2}^{1 / 2}$ and eq. 14 with $G = {\hat{G}}_{N}$ , we obtain

D_{FRA} ({\hat{G}}_{N}, G_{0}) = 𝒪_{ℙ} ​ ({(\log N / N)}^{1 / 2}) .

Now apply Lemma 1 along the aggregation path: for every $κ \in [K_{0} + 1, K]$ ,

D_{FRA} ({\hat{G}}_{N}^{(κ)}, G_{0}) ≲ D_{FRA} ({\hat{G}}_{N}, G_{0}) = 𝒪_{ℙ} ​ ({(\log N / N)}^{1 / 2}) .

For the exact-fit and under-fit levels $κ^{'} \leq K_{0}$ , $D_{FRA} = D_{E}$ by definition, which gives the second claim.

E.3 Proof Sketch of Theorem 2

For $κ \in [K_{0} + 1, K]$ , the height $0 p t_{N}^{(κ)}$ is the minimum $𝖽$ -distance between any two atoms of ${\hat{G}}_{N}^{(κ)}$ . Inside a multi-covered cell $𝔸_{k} ({\hat{G}}_{N})$ , the Taylor/merged-moment analysis from the proof of eq. 14 implies that

𝖽 ​ (\exp ({\hat{ω}}_{0 ​ i}) ​ δ_{{\hat{𝜽}}_{i}}, \exp ({\hat{ω}}_{0 ​ j}) ​ δ_{{\hat{𝜽}}_{j}}) ≲ {‖ (Δ_{𝒕_{1}} ​ {\hat{𝝎}}_{1 ​ i ​ k}, Δ ​ {\hat{b}}_{i ​ k}) ‖}^{2} + ‖ (Δ ​ {\hat{𝒂}}_{i ​ k}, Δ ​ {\hat{σ}}_{i ​ k}) ‖ .

The right-hand side is controlled by $D_{FRA} ({\hat{G}}_{N}^{(κ)}, G_{0})$ with the exponents $\bar{r} (| 𝔸_{k} |)$ , hence $0 p t_{N}^{(κ)} ≲ {(\log N / N)}^{1 / \bar{r} ({\hat{G}}_{N})}$ . For $κ^{'} \leq K_{0}$ , heights converge at parametric rate because atoms are separated and $D_{E} ({\hat{G}}_{N}^{(κ^{'})}, G_{0}^{(κ^{'})}) = 𝒪_{ℙ} ({(\log N / N)}^{1 / 2})$ .

E.4 Proof Sketch of Theorem 3

Let ${\bar{ℓ}}_{N} (p_{G}) = N^{- 1} \sum_{n = 1}^{N} \log p_{G} (y_{n} | 𝐱_{n})$ and $ℒ (p_{G}) = 𝔼_{(𝐱, y) \sim P_{G_{0}}} [\log p_{G} (y | 𝒙)]$ . Under Condition K, a local Lipschitz/curvature argument yields

| {\bar{ℓ}}_{N} (p_{G}) - ℒ (p_{G}) | ≲ 𝔼_{(𝐱, y) \sim P_{G_{0}}} [D_{TV} (p_{G} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] + empirical fluctuation .

For $κ \geq K_{0}$ , combine the inverse bound $𝔼 [D_{TV}] ≳ D_{FRA}$ with $D_{FRA} ({\hat{G}}_{N}^{(κ)}, G_{0}) = 𝒪_{ℙ} ({(\log N / N)}^{1 / 2})$ and standard empirical-process bounds (e.g., van de Geer, 2000) to obtain $| {\bar{ℓ}}_{N} (p_{{\hat{G}}_{N}^{(κ)}}) - ℒ (p_{G_{0}}) | ≲ {(\log N / N)}^{1 / (2 \bar{r} ({\hat{G}}_{N}))}$ . For $κ^{'} \leq K_{0}$ , ${\hat{G}}_{N}^{(κ^{'})}$ is exact/under-fit and converges at parametric rate, hence ${\bar{ℓ}}_{N} (p_{{\hat{G}}_{N}^{(κ^{'})}}) \to ℒ (p_{G_{0}^{(κ^{'})}})$ in probability.

Appendix F PROOF OF MAIN RESULTS

Before proving the main results, we fix notation used throughout this appendix. For any natural number $N \in ℕ$ , write $[N] := {1, 2, \dots, N}$ . Given two sequences of positive real numbers ${a_{N}}_{N = 1}^{\infty}$ and ${b_{N}}_{N = 1}^{\infty}$ , we write $a_{N} = 𝒪 (b_{N})$ (equivalently, $a_{N} ≲ b_{N}$ ) to mean that there exists a constant $C > 0$ such that $a_{N} \leq C b_{N}$ for all $N \in ℕ$ . For a vector $𝒗 \in ℝ^{D}$ and any multi-index $𝒑 \in ℕ^{D}$ , set $| 𝒑 | := p_{1} + \dots + p_{D}$ , $𝒗^{𝒑} := v_{1}^{p_{1}} v_{2}^{p_{2}} \dots v_{D}^{p_{D}}$ , $𝒑! := p_{1}! p_{2}! \dots p_{D}!$ , and let ${‖ 𝒗 ‖}_{p}$ denote its $p$ -norm; by default, $‖ 𝒗 ‖$ refers to the $2$ -norm unless otherwise stated. We also use $‖ 𝑨 ‖$ for the Frobenius norm of a matrix $𝑨 \in ℝ^{D \times D}$ . For any set $𝕊$ , $| 𝕊 |$ denotes its cardinality. For two probability density functions $p$ and $q$ with respect to the Lebesgue measure $μ$ , define $D_{TV} (p, q) := \frac{1}{2} \int | p - q | d μ$ as their total variation distance, while $D_{h}^{2} (p, q) := \frac{1}{2} \int {(\sqrt{p} - \sqrt{q})}^{2} d μ$ denotes the squared Hellinger distance. Moreover, for $𝝁 \in ℝ^{D}$ , $𝜶 \in ℕ^{D}$ , and a differentiable function $f$ of $𝝁$ , we write the partial derivative of order $| 𝜶 |$ as

\frac{\partial^{| 𝜶 |}}{\partial^{𝜶} 𝝁} ​ f ​ (𝝁) := \frac{\partial^{| 𝜶 |}}{\partial μ_{1}^{𝜶_{1}} ​ \partial μ_{2}^{𝜶_{2}} ​ \dots ​ \partial μ_{D}^{𝜶_{D}}} ​ f ​ (𝝁) .

Let $𝚯$ be the parameter space. Write $ℰ_{K} (𝚯)$ for the collection of discrete probability measures on $𝚯$ with exactly $K$ atoms, and $𝒪_{K} (𝚯) := ⋃_{K^{'} \leq K} ℰ_{K^{'}} (𝚯)$ for those with at most $K$ atoms. For a mixing measure $G = \sum_{k = 1}^{K} π_{k} δ_{𝜽_{k}}$ , we (slightly abusively) refer to each component $π_{k} δ_{𝜽_{k}}$ as an “atom,” comprising both its weight $π_{k}$ and parameter $𝜽_{k}$ . Finally, the domain of parameters in the SGMoE is $𝚯$ , where $𝜼_{k}^{0} := (ω_{0 k}^{0}, 𝝎_{1 k}^{0}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}) \in 𝚯 \subset ℝ \times ℝ^{D} \times ℝ^{D} \times ℝ \times ℝ_{> 0}$ . Furthermore, assume $𝚯$ is compact and $𝒳 \subset ℝ^{D}$ , the support of $𝐱$ , is bounded. When clear from context, we drop $𝚯$ and simply write $ℰ_{K}$ and $𝒪_{K}$ .

F.1 Proof of Lemma 1

We prove the inequality $D_{FRA} (G^{(K)}, G_{0}) ≳ D_{FRA} (G^{(K - 1)}, G_{0})$ , and the rest are similar.

Assume that $G_{N} := G_{N}^{(K)} = \sum_{k = 1}^{K} \exp (ω_{0 k}^{N}) δ_{(𝝎_{1 k}^{N}, 𝒂_{k}^{N}, b_{k}^{N}, σ_{k}^{N})} \in ℰ_{K}$ varies so that $D_{FRA} (G_{N}, G_{0}) \to 0$ . We consider the Voronoi cells $𝔸_{k}^{N} := 𝔸_{k} (G_{N})$ , for $k \in [K_{0}]$ , of the mixing measure $G_{N}$ generated by the true components of $G_{0}$ . Since the argument in this proof is asymptotic, we assume without loss of generality that those Voronoi cells are independent of $N$ for all $N \in ℕ$ , i.e, $𝔸_{k} = 𝔸_{k}^{N}$ .

Then, we have $(𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N}) \to (𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})$ , and there exist $t_{0} \in ℝ$ and $𝒕_{1} \in ℝ^{D}$ such that $\sum_{ℓ \in 𝔸_{k}^{N}} \exp (ω_{0 ℓ}^{N}) \to \exp (ω_{0 k}^{0} + t_{0})$ and $𝝎_{1 ℓ}^{N} \to 𝝎_{1 k}^{0} + 𝒕_{1}$ for any $ℓ \in 𝔸_{k}$ and $k \in [K_{0}]$ as $N$ approaches infinity.

We are going to show that the merging pair of indices $(ℓ_{1}, ℓ_{2})$ must belong to a common $𝔸_{k}$ . Indeed, for every pair $(ℓ_{1}, ℓ_{2})$ in a common $𝔸_{k}$ , since $(𝒂_{ℓ_{1}}^{N}, b_{ℓ_{1}}^{N}, σ_{ℓ_{1}}^{N}) \to (𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})$ and $𝝎_{1 ℓ_{1}}^{N} \to 𝝎_{1 k}^{0} + 𝒕_{1}$ , and $(𝒂_{ℓ_{2}}^{N}, b_{ℓ_{2}}^{N}, σ_{ℓ_{2}}^{N}) \to (𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})$ and $𝝎_{1 ℓ_{2}}^{N} \to 𝝎_{1 k}^{0} + 𝒕_{1}$ , we have

𝖽 ​ (\exp (ω_{0 ​ ℓ_{1}}^{N}) ​ δ_{(𝝎_{1 ​ ℓ_{1}}^{N}, 𝒂_{ℓ_{1}}^{N}, b_{ℓ_{1}}^{N}, σ_{ℓ_{1}}^{N})}, \exp (ω_{0 ​ ℓ_{2}}^{N}) ​ δ_{(𝝎_{1 ​ ℓ_{2}}^{N}, 𝒂_{ℓ_{2}}^{N}, b_{ℓ_{2}}^{N}, σ_{ℓ_{2}}^{N})}) \to 0, as N \to \infty .

On the other hand, for every pair $(ℓ, ℓ^{'}) \in 𝔸_{k} \times 𝔸_{k^{'}}$ , where $k \neq k^{'}$ , because $(𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N}) \to (𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})$ and $𝝎_{1 ℓ}^{N} \to 𝝎_{1 k}^{0} + 𝒕_{1}$ , and $(𝒂_{ℓ^{'}}^{N}, b_{ℓ^{'}}^{N}, σ_{ℓ^{'}}^{N}) \to (𝒂_{k^{'}}^{0}, b_{k^{'}}^{0}, σ_{k^{'}}^{0})$ and $𝝎_{1 ℓ^{'}}^{N} \to 𝝎_{1 k^{'}}^{0} + 𝒕_{1}$ , we have

𝖽 ​ (\exp (ω_{0 ​ ℓ}^{N}) ​ δ_{(𝝎_{1 ​ ℓ}^{N}, 𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N})}, \exp (ω_{0 ​ ℓ^{'}}^{N}) ​ δ_{(𝝎_{1 ​ ℓ^{'}}^{N}, 𝒂_{ℓ^{'}}^{N}, b_{ℓ^{'}}^{N}, σ_{ℓ^{'}}^{N})}) ≳ {‖ (𝝎_{1 ​ k}^{0}, b_{k}^{0}) - (𝝎_{1 ​ k^{'}}^{0}, b_{k^{'}}^{0}) ‖}^{2} + ‖ (𝒂_{k}^{0}, σ_{k}^{0}) - (𝒂_{k^{'}}^{0}, σ_{k^{'}}^{0}) ‖,

where the multiplicative constant is not dependent on $N$ . Hence, the merging pair must belong to a common $𝔸_{k}$ .

Next, for any $(t_{0}, 𝒕_{1}) \in ℝ \times ℝ^{D}$ such that $ω_{0 k}^{0} + t_{0}$ and $𝝎_{1 k}^{0} + 𝒕_{1}$ still lie inside the domain of the parameter space $𝚯$ , we define $𝒟 (G_{N}, G_{0}, t_{0}, 𝒕_{1})$ as

	$𝒟 (G_{N}, G_{0}, t_{0}, 𝒕_{1}) := \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) ({‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}, Δ b_{ℓ k}^{N}) ‖}^{{\bar{r}}_{k}} + {‖ (Δ 𝒂_{ℓ k}^{N}, Δ σ_{ℓ k}^{N}) ‖}^{{\bar{r}}_{k} / 2})$
	$+ \sum_{k : \| 𝔸_{k} \| = 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) ‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}, Δ 𝒂_{ℓ k}^{N}, Δ b_{ℓ k}^{N}, Δ σ_{ℓ k}^{N}) ‖ + \sum_{k = 1}^{K_{0}} \| \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) - \exp (ω_{0 k}^{0} + t_{0}) \|$
	$+ \sum_{k : \| 𝔸_{k} \| > 1} (∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ b_{ℓ k}^{N}) ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [{(Δ b_{ℓ k}^{N})}^{2} + (Δ σ_{ℓ k}^{N})] ∥$
	$+ ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) (Δ b_{ℓ k}^{N}) + (Δ 𝒂_{ℓ k}^{N})] ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{⊤} ∥),$

in which $Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N} := 𝝎_{1 ℓ}^{N} - 𝝎_{1 k}^{0} - 𝒕_{1}$ , $Δ 𝒂_{ℓ k}^{N} := 𝒂_{ℓ}^{N} - 𝒂_{k}^{0}$ , $Δ b_{ℓ k}^{N} := b_{ℓ}^{N} - b_{k}^{0}$ , $Δ σ_{ℓ k}^{N} := σ_{ℓ}^{N} - σ_{k}^{0}$ , and ${\bar{r}}_{k} := \bar{r} (𝔸_{k} ({\hat{G}}_{N}))$ .

We prove that $𝒟 (G_{N}^{(K - 1)}, G_{0}, t_{0}, 𝒕_{1}) ≲ 𝒟 (G_{N}^{(K)}, G_{0}, t_{0}, 𝒕_{1})$ . Let the merging pair of indices $(ℓ_{1}, ℓ_{2})$ in the Voronoi cell $𝔸_{k}$ , then $| 𝔸_{k} | > 1$ and the merged atom is $\exp (ω_{0, *}^{N}) δ_{(𝝎_{1, *}^{N}, 𝒂_{*}^{N}, b_{*}^{N}, σ_{*}^{N})}$ , i.e,

	$ω_{0, *}^{N}$	$= \log (\exp ω_{0 ℓ_{1}}^{N} + \exp ω_{0 ℓ_{2}}^{N}),$
	$𝝎_{1, *}^{N}$	$= \exp (ω_{0 ℓ_{1}}^{N} - ω_{0, }^{N}) 𝝎_{1 ℓ_{1}}^{N} + \exp (ω_{0 ℓ_{2}}^{N} - ω_{0, }^{N}) 𝝎_{1 ℓ_{2}}^{N},$
	$b_{*}^{N}$	$= \exp (ω_{0 ℓ_{1}}^{N} - ω_{0, }^{N}) b_{ℓ_{1}}^{N} + \exp (ω_{0 ℓ_{2}}^{N} - ω_{0, }^{N}) b_{ℓ_{2}}^{N},$
	$𝒂_{*}^{N}$	$= \frac{\exp (ω_{0 ℓ_{1}}^{N})}{\exp (ω_{0, }^{N})} [(𝝎_{1 ℓ_{1}}^{N} - 𝝎_{1, }^{N}) (b_{ℓ_{1}}^{N} - b_{}^{N}) + 𝒂_{ℓ_{1}}^{N}] + \frac{\exp (ω_{0 ℓ_{2}}^{N})}{\exp (ω_{0, }^{N})} [(𝝎_{1 ℓ_{2}}^{N} - 𝝎_{1, }^{N}) (b_{ℓ_{2}}^{N} - b_{}^{N}) + 𝒂_{ℓ_{2}}^{N}],$
	$σ_{*}^{N}$	$= \frac{\exp (ω_{0 ℓ_{1}}^{N})}{\exp (ω_{0, }^{N})} [{(b_{ℓ_{1}}^{N} - b_{}^{N})}^{2} + σ_{ℓ_{1}}^{N}] + \frac{\exp (ω_{0 ℓ_{2}}^{N})}{\exp (ω_{0, }^{N})} [{(b_{ℓ_{2}}^{N} - b_{}^{N})}^{2} + σ_{ℓ_{2}}^{N}] .$

Hence, we have that

	$\| \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) - \exp (ω_{0 k}^{0} + t_{0}) \|$	$= \| \sum_{ℓ \in 𝔸_{k}, ℓ \notin {ℓ_{1}, ℓ_{2}}} \exp (ω_{0 ℓ}^{N}) + \exp (ω_{0, *}^{N}) - \exp (ω_{0 k}^{0} + t_{0}) \|,$
	$\exp (ω_{0, }^{N}) Δ b_{ k}^{N}$	$= \exp (ω_{0, }^{N}) (b_{}^{N} - b_{k}^{0})$
		$= \exp (ω_{0, }^{N}) (\exp (ω_{0 ℓ_{1}}^{N} - ω_{0, }^{N}) b_{ℓ_{1}}^{N} + \exp (ω_{0 ℓ_{2}}^{N} - ω_{0, *}^{N}) b_{ℓ_{2}}^{N} - b_{k}^{0})$
		$= \exp (ω_{0 ℓ_{1}}^{N}) b_{ℓ_{1}}^{N} + \exp (ω_{0 ℓ_{2}}^{N}) b_{ℓ_{2}}^{N} - \exp (ω_{0, *}^{N}) b_{k}^{0}$
		$= \exp (ω_{0 ℓ_{1}}^{N}) (b_{ℓ_{1}}^{N} - b_{k}^{0}) + \exp (ω_{0 ℓ_{2}}^{N}) (b_{ℓ_{2}}^{N} - b_{k}^{0})$
		$= \exp (ω_{0 ℓ_{1}}^{N}) Δ b_{ℓ_{1} k}^{N} + \exp (ω_{0 ℓ_{2}}^{N}) Δ b_{ℓ_{2} k}^{N} .$

It follows that the term

‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ​ ℓ}^{N}) ​ (Δ ​ b_{ℓ ​ k}^{N}) ‖ = ‖ \sum_{ℓ \in 𝔸_{k} ∖ {ℓ_{1}, ℓ_{2}}} \exp (ω_{0 ​ ℓ}^{N}) ​ (Δ ​ b_{ℓ ​ k}^{N}) + \exp (ω_{0, *}^{N}) ​ (Δ ​ b_{* k}^{N}) ‖ .

Similarly, we can show that

	$‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) ‖$	$= ‖ \sum_{ℓ \in 𝔸_{k} ∖ {ℓ_{1}, ℓ_{2}}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) + \exp (ω_{0, }^{N}) (Δ_{𝒕_{1}} 𝝎_{1 k}^{N}) ‖,$
	$‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [{(Δ b_{ℓ k}^{N})}^{2} + (Δ σ_{ℓ k}^{N})] ‖$	$= ∥ \sum_{ℓ \in 𝔸_{k} ∖ {ℓ_{1}, ℓ_{2}}} \exp (ω_{0 ℓ}^{N}) [{(Δ b_{ℓ k}^{N})}^{2} + (Δ σ_{ℓ k}^{N})]$
		$+ \exp (ω_{0, }^{N}) [{(Δ b_{ k}^{N})}^{2} + (Δ σ_{* k}^{N})] ∥,$
	$‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) (Δ b_{ℓ k}^{N}) + (Δ 𝒂_{ℓ k}^{N})] ‖$	$= ∥ \sum_{ℓ \in 𝔸_{k} ∖ {ℓ_{1}, ℓ_{2}}} \exp (ω_{0 ℓ}^{N}) [(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) (Δ b_{ℓ k}^{N}) + (Δ 𝒂_{ℓ k}^{N})]$
		$+ \exp (ω_{0, }^{N}) [(Δ_{𝒕_{1}} 𝝎_{1 k}^{N}) (Δ b_{* k}^{N}) + (Δ 𝒂_{* k}^{N})] ∥,$
	$‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{⊤} ‖$	$= ∥ \sum_{ℓ \in 𝔸_{k} ∖ {ℓ_{1}, ℓ_{2}}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{⊤}$
		$+ \exp (ω_{0, }^{N}) (Δ_{𝒕_{1}} 𝝎_{1 k}^{N}) {(Δ_{𝒕_{1}} 𝝎_{1 * k}^{N})}^{⊤} ∥ .$

To this end, we show the key convexity step in detail. Firstly, we define $α_{i}$ and $𝒙_{i}$ ( $i = 1, 2$ ) as follow:

	$α_{i}$	$:= \exp (ω_{0 ℓ_{i}}^{N} - ω_{0, *}^{N}), i = 1, 2,$
	$𝒙_{i}$	$:= (𝝎_{1 ℓ_{i}}^{N} - 𝝎_{1 k}^{0} - 𝒕_{1}, b_{ℓ_{i}}^{N} - b_{k}^{0}) = (Δ_{𝒕_{1}} 𝝎_{1 ℓ_{i} k}^{N}, Δ b_{ℓ_{i} k}^{N}), i = 1, 2 .$

Note that $𝜶_{1}, 𝜶_{2} \in (0, 1)$ and $𝜶_{1} + 𝜶_{2} = 1$ and by the definition of the merged atom eq. 8, we have the convex combination identity $(Δ_{𝒕_{1}} 𝝎_{1 * k}^{N}, Δ b_{* k}^{N}) = 𝜶_{1} 𝒙_{1} + 𝜶_{2} 𝒙_{2}$ .

By the fact that $\bar{r} \geq 4$ (since $| 𝔸_{k} | > 1$ ), so we can use Jensen’s inequality (convexity of the map $z \mapsto {‖ z ‖}^{m}$ with $m = {\bar{r}}_{k}$ ):

{‖ 𝜶_{1} ​ 𝒙_{1} + 𝜶_{2} ​ 𝒙_{2} ‖}^{{\bar{r}}_{k}} \leq 𝜶_{1} ​ {‖ 𝒙_{1} ‖}^{{\bar{r}}_{k}} + 𝜶_{2} ​ {‖ 𝒙_{2} ‖}^{{\bar{r}}_{k}} .

Multiply both sides by $\exp (ω_{0, *}^{N})$ and substitute $α_{i} = \exp (ω_{0 ℓ_{i}}^{N} - ω_{0, *}^{N})$ :

	$(\exp (ω_{0 ℓ_{1}}^{N}) + \exp (ω_{0 ℓ_{2}}^{N})) ∥ Δ_{𝒕_{1}} 𝝎_{1 * k}^{N}, Δ b_{* k}^{N} ∥^{{\bar{r}}_{k}}$	$= \exp (ω_{0, *}^{N}) {‖ 𝜶_{1} 𝒙_{1} + 𝜶_{2} 𝒙_{2} ‖}^{{\bar{r}}_{k}}$
		$\leq \exp (ω_{0, *}^{N}) (𝜶_{1} {‖ 𝒙_{1} ‖}^{{\bar{r}}_{k}} + 𝜶_{2} {‖ 𝒙_{2} ‖}^{{\bar{r}}_{k}})$
		$= \exp (ω_{0 ℓ_{1}}^{N}) {‖ 𝒙_{1} ‖}^{{\bar{r}}_{k}} + \exp (ω_{0 ℓ_{2}}^{N}) {‖ 𝒙_{2} ‖}^{{\bar{r}}_{k}}$
		$= \exp (ω_{0 ℓ_{1}}^{N}) {‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ_{1} k}^{N}, Δ b_{ℓ_{1} k}^{N}) ‖}^{{\bar{r}}_{k}} + \exp (ω_{0 ℓ_{2}}^{N}) {‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ_{2} k}^{N}, Δ b_{ℓ_{2} k}^{N}) ‖}^{{\bar{r}}_{k}} .$

Analogously, we can show that

\exp (ω_{0 ​ ℓ_{1}}^{N}) ​ {‖ (Δ ​ 𝒂_{ℓ_{1} ​ k}^{N}, Δ ​ σ_{ℓ_{1} ​ k}^{N}) ‖}^{{\bar{r}}_{k} / 2} + \exp (ω_{0 ​ ℓ_{2}}^{N}) ​ {‖ (Δ ​ 𝒂_{ℓ_{2} ​ k}^{N}, Δ ​ σ_{ℓ_{2} ​ k}^{N}) ‖}^{{\bar{r}}_{k} / 2} ≳ (\exp (ω_{0 ​ ℓ_{1}}^{N}) + \exp (ω_{0 ​ ℓ_{2}}^{N})) ​ {‖ (Δ ​ 𝒂_{* k}^{N}, Δ ​ σ_{* k}^{N}) ‖}^{{\bar{r}}_{k} / 2} .

Combining the two inequalities above gives the claimed comparison between the contribution of the merged atom and the contributions of the two original atoms:

	$\exp (ω_{0 ℓ_{1}}^{N}) (∥ (Δ_{𝒕_{1}} 𝝎_{1 ℓ_{1} k}^{N}, Δ b_{ℓ_{1} k}^{N}) ∥^{{\bar{r}}_{k}}$	$+ ∥ (Δ 𝒂_{ℓ_{1} k}^{N}, Δ σ_{ℓ_{1} k}^{N}) ∥^{{\bar{r}}_{k} / 2}) + \exp (ω_{0 ℓ_{2}}^{N}) (∥ (Δ_{𝒕_{1}} 𝝎_{1 ℓ_{2} k}^{N}, Δ b_{ℓ_{2} k}^{N}) ∥^{{\bar{r}}_{k}} + ∥ (Δ 𝒂_{ℓ_{2} k}^{N}, Δ σ_{ℓ_{2} k}^{N}) ∥^{{\bar{r}}_{k} / 2})$
		$≳ (\exp (ω_{0 ℓ_{1}}^{N}) + \exp (ω_{0 ℓ_{2}}^{N})) ({‖ (Δ_{𝒕_{1}} 𝝎_{1 * k}^{N}, Δ b_{* k}^{N}) ‖}^{{\bar{r}}_{k}} + {‖ (Δ 𝒂_{* k}^{N}, Δ σ_{* k}^{N}) ‖}^{{\bar{r}}_{k} / 2}) .$

Hence

𝒟 ​ (G_{N}^{(K)}, G_{0}, t_{0}, 𝒕_{1}) ≳ 𝒟 ​ (G_{N}^{(K - 1)}, G_{0}, t_{0}, 𝒕_{1}),

and therefore

D_{FRA} (G_{N}^{(K - 1)}, G_{0}) ≲ D_{FRA} (G_{N}^{(K)}, G_{0}) .

F.2 Proof of Theorem 1

First of all, we study the convergence rate of the MLE ${\hat{G}}_{N} \in ℰ_{K}$ of the SGMoE; that is, we will show the inverse bound for SGMoE. We revisit the following result on the identifiability of the SGMoE models, which was previously studied in Nguyen et al. (2023a); Jiang and Tanner (1999).

Fact 3 (Nguyen et al., 2023a, Proposition 1).

For any mixing measures $G = \sum_{k = 1}^{K} \exp (ω_{0 k}) δ_{(𝛚_{1 k}, 𝐚_{k}, b_{k}, σ_{k})}$ and $G^{'} = \sum_{k = 1}^{K^{'}} \exp (ω_{0 k}^{'}) δ_{(𝛚_{1 k}^{'}, 𝐚_{k}^{'}, b_{k}^{'}, σ_{k}^{'})}$ , if we have $p_{G} (y | 𝐱) = p_{G^{'}} (y | 𝐱)$ for almost surely $(𝐱, y)$ , then it follows that $K = K^{'}$ and $G \equiv G_{t_{0}, 𝐭_{1}}^{'}$ where $G_{t_{0}, 𝐭_{1}}^{'} := \sum_{k = 1}^{K^{'}} \exp (ω_{0 k}^{'} + t_{0}) δ_{(𝛚_{1 k}^{'} + 𝐭_{1}, 𝐚_{k}^{'}, b_{k}^{'}, σ_{k}^{'})}$ for some $t_{0} \in ℝ$ and $𝐭_{1} \in ℝ^{D}$ .

The identifiability of the softmax gating Gaussian mixture of experts guarantees that the MLE ${\hat{G}}_{N}$ converges to the true mixing measure $G_{0}$ (up to the translation of the parameters in the softmax gating).

Given the consistency of the MLE, it is natural to ask about its convergence rate to the true parameters. Our next result establishes the convergence rate of conditional density estimation $p_{{\hat{G}}_{N}} (y | 𝒙)$ to the true conditional density $p_{G_{0}} (y | 𝒙)$ , which lays an important foundation for the study of MLE’s convergence rate.

Fact 4 (Nguyen et al., 2023a, Proposition 2).

The density estimation $p_{{\hat{G}}_{N}} (y | 𝐱)$ converges to the true density $p_{G_{0}} (y | 𝐱)$ under the Hellinger distance $D_{h}^{2} (\cdot, \cdot)$ at the following rate:

𝔼_{𝐱} [D_{h}^{2} (p_{{\hat{G}}_{N}} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] = 𝒪_{P} (\sqrt{\log (N) / N}) .

That is,

ℙ (𝔼_{𝐱} [D_{h}^{2} (p_{{\hat{G}}_{N}} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] > C {(\log (N) / N)}^{1 / 2}) ≲ \exp (- c \log N),

where $c$ and $C$ are universal constants.

The result of Fact 4 indicates that under either the exact-specified or over-specified cases of the SGMoE, the rate of the conditional density function $p_{{\hat{G}}_{N}} (y | 𝒙)$ to the true one $p_{G_{0}} (y | 𝒙)$ under Hellinger distance is of order $𝒪 (N^{- 1 / 2})$ (up to some logarithmic factors), which is parametric on the sample size.

Now, we establish the convergence rate of the MLE under the over-specified case of the SGMoE via the Fast-Rate-Aware Voronoi Distance $D_{FRA}$ .

Theorem 5.

Under the over-specified case of the SGMoE, namely, when $K > K_{0}$ , we obtain that

𝔼_{𝐱} [D_{h}^{2} (p_{G} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] \geq C \cdot D_{FRA} (G, G_{0}),

for any $G \in 𝒪_{K}$ where $C$ is some universal constant depending only on $G_{0}$ and $𝚯$ . Therefore, that lower bound leads to the following convergence rate of the MLE:

ℙ ​ (D_{FRA} ({\hat{G}}_{N}, G_{0}) > C^{'} ​ {(\log (N) / N)}^{1 / 2}) ≲ \exp (- c ​ \log N),

(15)

where $C^{'}$ and $c$ are some universal constants.

Proof of Theorem 5.

We are going to prove that there exists a constant $C > 0$ depending only on $G_{0}$ and $𝚯$ such that, for any $G \in 𝒪_{K}$ ,

𝔼_{𝐱} [D_{TV} (p_{G} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] ≳ D_{FRA} (G, G_{0}) .

(16)

Then, by the Fact 4, we get the convergence rate of the MLE of SGMoE.

Local version: Firstly, we prove the local version of the eq. 16:

lim_{ε \to 0} inf_{G \in 𝒪_{K} : D_{FRA} (G, G_{0}) \leq ε} 𝔼_{𝐱} [D_{TV} (p_{G} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] / D_{FRA} (G, G_{0}) > 0 .

(17)

Assume that the inequality in eq. 17 does not hold true, there exists a sequence of mixing measures $G_{N} := \sum_{k = 1}^{K_{N}} \exp (ω_{0 k}^{N}) δ_{(𝝎_{1 k}^{N}, 𝒂_{k}^{N}, b_{k}^{N}, σ_{k}^{N})} \in 𝒪_{K}$ such that

	$𝔼_{𝐱} [D_{TV} (p_{G_{N}} (\cdot \| x), p_{G_{0}} (\cdot \| x)] / D_{FRA} (G_{N}, G_{0})$	$\to 0,$
	$D_{FRA} (G_{N}, G_{0})$	$\to 0,$

when $N$ to infinity. Since the proof argument is asymptotic, we also assume that $K_{N} = K^{'} \leq K$ for all $N \geq 1$ . Next, we consider the Voronoi cells $𝔸_{k}^{N} := 𝔸_{k} (G_{N})$ , for $k \in [K_{0}]$ , of the mixing measure $G_{N}$ generated by the true components of $G_{0}$ . And we can assume without loss of generality (WLOG) that those Voronoi cells are independent of $N$ for all $N \in ℕ$ , i.e. $𝔸_{k} = 𝔸_{k}^{N}$ . Additionally, since $D_{FRA} (G_{N}, G_{0}) \to 0$ , we have $(𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N}) \to (𝒂_{ℓ}^{0}, b_{ℓ}^{0}, σ_{ℓ}^{0})$ for any $ℓ \in 𝔸_{k}$ as $N \to \infty$ . Furthermore, there exist $t_{0} \in ℝ$ and $𝒕_{1} \in ℝ^{D}$ such that $\sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) \to \exp (ω_{0 k}^{0} + t_{0})$ and $𝝎_{1 ℓ}^{N} \to 𝝎_{1 k}^{0} + 𝒕_{1}$ as $N$ approaches infinity for any $ℓ \in 𝔸_{k}$ and $k \in [K_{0}]$ . It suggests that we can upper bound $D_{FRA}$ as $D_{FRA} (G_{N}, G_{0}) \leq D_{V} (G_{N}, G_{0})$ , where

	$D_{V} (G_{N}, G_{0}) := \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) ({‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}, Δ b_{ℓ k}^{N}) ‖}^{\bar{r} (\| 𝔸_{k} \|)} + {‖ (Δ 𝒂_{ℓ k}^{N}, Δ σ_{ℓ k}^{N}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2})$
	$+ \sum_{k : \| 𝔸_{k} \| = 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) ‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}, Δ 𝒂_{ℓ k}^{N}, Δ b_{ℓ k}^{N}, Δ σ_{ℓ k}^{N}) ‖ + \sum_{k = 1}^{K_{0}} \| \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) - \exp (ω_{0 k}^{0} + t_{0}) \|$
	$+ \sum_{k : \| 𝔸_{k} \| > 1} (∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ b_{ℓ k}^{N}) ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [{(Δ b_{ℓ k}^{N})}^{2} + (Δ σ_{ℓ k}^{N})] ∥$
	$+ ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) (Δ b_{ℓ k}^{N}) + (Δ 𝒂_{ℓ k}^{N})] ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{⊤} ∥),$

in which $Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N} := 𝝎_{1 ℓ}^{N} - 𝝎_{1 k}^{0} - 𝒕_{1}$ , $Δ 𝒂_{ℓ k}^{N} := 𝒂_{ℓ}^{N} - 𝒂_{k}^{0}$ , $Δ b_{ℓ k}^{N} := b_{ℓ}^{N} - b_{k}^{0}$ , $Δ σ_{ℓ k}^{N} := σ_{ℓ}^{N} - σ_{k}^{0}$ .

Step 1: Density Decomposition

In this step, we try to find a density decomposition for the quatity $Q_{N} = [\sum_{k = 1}^{K_{0}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙 + ω_{0 k}^{0} + t_{0})] \cdot [p_{G_{N}} (y | 𝒙) - p_{G_{0}} (y | 𝒙)]$ :

	$Q_{N}$	$= \sum_{k = 1}^{K_{0}} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [u (y \| 𝒙; 𝝎_{1 ℓ}^{N}, 𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N}) - u (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})]$
		$- \sum_{k = 1}^{K_{0}} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [v (y \| 𝒙; 𝝎_{1 ℓ}^{N}) - v (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1})]$
		$+ \sum_{k = 1}^{K_{0}} (\sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) - \exp (ω_{0 k}^{0} + t_{0})) [u (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}) - v (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1})],$
		$:= A_{N} + B_{N} + E_{N},$

where we denote $u (y | 𝒙; 𝝎_{1}, 𝒂, b, σ) := \exp (𝝎_{1}^{⊤} 𝒙) 𝒩 (y | 𝒂^{⊤} 𝒙 + b, σ)$ and $v (y | 𝒙; 𝝎_{1}) := \exp (𝝎_{1}^{⊤} 𝒙) p_{G_{N}} (y | 𝒙)$ .

Since each Voronoi cell $𝔸_{k}$ possibly has more than one element, we continue to decompose $A_{N}$ as follows:

	$A_{N}$	$= \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [u (y \| 𝒙; 𝝎_{1 ℓ}^{N}, 𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N}) - u (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})]$
		$+ \sum_{k : \| 𝔸_{k} \| = 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [u (y \| 𝒙; 𝝎_{1 ℓ}^{N}, 𝒂_{ℓ}^{N}, b_{ℓ}^{N}, σ_{ℓ}^{N}) - u (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})]$
		$:= A_{N, 1} + A_{N, 2} .$

Now, we perform Taylor expansion up to the $\bar{r} (| 𝔸_{k} |) -$ th order, and then rewrite $A_{N, 1}$ with a note that $𝜶 = (𝜶_{1}, 𝜶_{2}, α_{3}, α_{4}) \in ℕ^{D} \times ℕ^{D} \times ℕ \times ℕ$ as follows:

	$A_{N, 1}$	$= \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \sum_{\| 𝜶 \| = 1}^{\bar{r} (\| 𝔸_{k} \|)} \frac{\exp (ω_{0 ℓ}^{N})}{𝜶!} {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{𝜶_{1}} {(Δ 𝒂_{ℓ k}^{N})}^{𝜶_{2}} {(Δ b_{ℓ k}^{N})}^{α_{3}} {(Δ σ_{ℓ k}^{N})}^{α_{4}}$
		$\times \frac{\partial^{\| 𝜶_{1} \| + \| 𝜶_{2} \| + α_{3} + α_{4}}}{\partial 𝝎_{1}^{𝜶_{1}} \partial 𝒂^{𝜶_{2}} \partial b^{α_{3}} \partial σ^{α_{4}}} u (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}) + R_{1}^{N} (𝒙, y),$

where $R_{1}^{N} (𝒙, y)$ is the remainder term such that

R_{1}^{N} ​ (𝒙, y) = o ​ (\sum_{k : | 𝔸_{k} | > 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ​ ℓ}^{N}) ​ ({‖ Δ_{𝒕_{1}} ​ 𝝎_{1 ​ ℓ ​ k}^{N} ‖}^{\bar{r} ​ (| 𝔸_{k} |)} + {‖ Δ ​ 𝒂_{ℓ ​ k}^{N} ‖}^{\bar{r} ​ (| 𝔸_{k} |)} + {‖ Δ ​ b_{ℓ ​ k}^{N} ‖}^{\bar{r} ​ (| 𝔸_{k} |)} + {‖ Δ ​ σ_{ℓ ​ k}^{N} ‖}^{\bar{r} ​ (| 𝔸_{k} |)})) .

Next, for each $k \in [K_{0}]$ and $ℓ \in 𝔸_{k}$ , we denote $h_{1} (𝒙, 𝒂, b) := {(𝒂)}^{⊤} 𝒙 + b$ . By the partial differential equations

\frac{\partial^{2} u}{\partial 𝝎_{1} ​ \partial b} = \frac{\partial u}{\partial 𝒂}; \frac{\partial^{2} u}{\partial b^{2}} = 2 ​ \frac{\partial u}{\partial σ},

we have

\frac{\partial^{| 𝜶_{2} |} u}{\partial 𝒂^{𝜶_{2}}} = \frac{\partial^{2 ​ | 𝜶_{2} |} u}{\partial 𝝎_{1}^{𝜶_{2}} ​ \partial b^{| 𝜶_{2} |}}; \frac{\partial^{α_{4}} u}{\partial σ^{α_{4}}} = \frac{1}{2^{α_{4}}} \cdot \frac{\partial^{2 ​ α_{4}} u}{\partial b^{2 ​ α_{4}}} .

Hence

\frac{\partial^{| 𝜶_{1} | + | 𝜶_{2} | + α_{3} + α_{4}} u}{\partial 𝝎_{1}^{𝜶_{1}} ​ \partial 𝒂^{𝜶_{2}} ​ \partial b^{α_{3}} ​ \partial σ^{α_{4}}} = \frac{1}{2^{α_{4}}} \cdot \frac{\partial^{(| 𝜶_{1} | + | 𝜶_{2} |) + (| 𝜶_{2} | + α_{3} + 2 ​ α_{4})} u}{\partial 𝝎_{1}^{𝜶_{1} + 𝜶_{2}} ​ \partial b^{| 𝜶_{2} | + α_{3} + 2 ​ α_{4}}} .

It follows that

	$A_{N, 1}$	$= \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \sum_{\| 𝜶 \| = 1}^{\bar{r} (\| 𝔸_{k} \|)} \frac{\exp (ω_{0 ℓ}^{N})}{𝜶!} {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{𝜶_{1}} {(Δ 𝒂_{ℓ k}^{N})}^{𝜶_{2}} {(Δ b_{ℓ k}^{N})}^{α_{3}} {(Δ σ_{ℓ k}^{N})}^{α_{4}}$
		$\times \frac{1}{2^{α_{4}}} \cdot \frac{\partial^{(\| 𝜶_{1} \| + \| 𝜶_{2} \|) + (\| 𝜶_{2} \| + α_{3} + 2 α_{4})}}{\partial 𝝎_{1}^{𝜶_{1} + 𝜶_{2}} \partial b^{\| 𝜶_{2} \| + α_{3} + 2 α_{4}}} u (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}) + R_{1}^{N} (𝒙, y)$
		$= \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \sum_{\| ℓ_{1} \| + ℓ_{2} = 1}^{2 \bar{r} (\| 𝔸_{k} \|)} \sum_{𝜶 \in 𝕀_{ℓ_{1}, ℓ_{2}}} \frac{\exp (ω_{0 ℓ}^{N})}{2^{α_{4}} 𝜶!} {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{𝜶_{1}} {(Δ 𝒂_{ℓ k}^{N})}^{𝜶_{2}} {(Δ b_{ℓ k}^{N})}^{α_{3}} {(Δ σ_{ℓ k}^{N})}^{α_{4}}$
		$\times 𝒙^{ℓ_{1}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) \cdot \frac{\partial^{ℓ_{2}} 𝒩}{\partial h_{1}^{ℓ_{2}}} (y \| {(𝒂_{k}^{0})}^{⊤} 𝒙 + b_{k}^{0}, σ_{k}^{0}) + R_{1}^{N} (𝒙, y),$

where $𝕀_{ℓ_{1}, ℓ_{2}} = {𝜶 = (𝜶_{1}, 𝜶_{2}, α_{3}, α_{4}) \in ℕ^{D} \times ℕ^{D} \times ℕ \times ℕ : 𝜶_{1} + 𝜶_{2} = ℓ_{1}, | 𝜶_{2} | + α_{3} + 2 α_{4} = ℓ_{2}}$ .

Similarly, we can decompose $A_{N, 2}$ by the first-order Taylor expansion as

	$A_{N, 2}$	$= \sum_{k : \| 𝔸_{k} \| = 1} \sum_{ℓ \in 𝔸_{k}} \sum_{\| ℓ_{1} \| + ℓ_{2} = 1}^{2} \sum_{𝜶 \in 𝕀_{ℓ_{1}, ℓ_{2}}} \frac{\exp (ω_{0 ℓ}^{N})}{2^{α_{4}} 𝜶!} {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{𝜶_{1}} {(Δ 𝒂_{ℓ k}^{N})}^{𝜶_{2}} {(Δ b_{ℓ k}^{N})}^{α_{3}} {(Δ σ_{ℓ k}^{N})}^{α_{4}}$
		$\times 𝒙^{ℓ_{1}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) \cdot \frac{\partial^{\| ℓ_{2} \|} 𝒩}{\partial h_{1}^{\| ℓ_{2} \|}} (y \| {(𝒂_{k}^{0})}^{⊤} 𝒙 + b_{k}^{0}, σ_{k}^{0}) + R_{2}^{N} (𝒙, y),$

where

R_{2}^{N} ​ (𝒙, y) = o ​ (\sum_{k : | 𝔸_{k} | = 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ​ ℓ}^{N}) ​ (‖ Δ_{𝒕_{1}} ​ 𝝎_{1 ​ ℓ ​ k}^{N} ‖ + ‖ Δ ​ 𝒂_{ℓ ​ k}^{N} ‖ + ‖ Δ ​ b_{ℓ ​ k}^{N} ‖ + ‖ Δ ​ σ_{ℓ ​ k}^{N} ‖)) .

Analogously, $B_{N}$ can be rewritten as

	$B_{N}$	$= B_{N, 1} + B_{N, 2}$
		$= - \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \sum_{\| γ \| = 1}^{2} \frac{\exp (ω_{0 ℓ}^{N})}{γ!} {(Δ_{𝒕_{1}} 𝝎_{1 i k}^{N})}^{γ} \cdot 𝒙^{γ} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) p_{G_{N}} (y \| 𝒙) + R_{3}^{N} (𝒙, y)$
		$- \sum_{k : \| 𝔸_{k} \| = 1} \sum_{ℓ \in 𝔸_{k}} \sum_{\| γ \| = 1} \frac{\exp (ω_{0 ℓ}^{N})}{γ!} {(Δ_{𝒕_{1}} 𝝎_{1 i k}^{N})}^{γ} \cdot 𝒙^{γ} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) p_{G_{N}} (y \| 𝒙) + R_{4}^{N} (𝒙, y)$

where

	$R_{3}^{N} (𝒙, y)$	$= o (\sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) ({‖ Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N} ‖}^{2})),$
	$R_{4}^{N} (𝒙, y)$	$= o (\sum_{k : \| 𝔸_{k} \| = 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (‖ Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N} ‖)) .$

Therefore, $Q_{N}$ can be represented as

$Q_{N}$	$= \sum_{k = 1}^{K_{0}} \sum_{\| ℓ_{1} \| + ℓ_{2} = 1}^{2 \bar{r} (\| 𝔸_{k} \|)} T_{ℓ_{1}, ℓ_{2}}^{N} (k) \cdot 𝒙^{ℓ_{1}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) \frac{\partial^{ℓ_{2}} 𝒩}{\partial h_{1}^{ℓ_{2}}} (y \| 𝒂_{k}^{0 ⊤} 𝒙 + b_{k}^{0}, σ_{k}^{0})$
	$+ \sum_{k = 1}^{K_{0}} \sum_{\| γ \| = 1}^{1 + 𝟏_{{\| 𝔸_{k} \| > 1}}} S_{γ}^{N} (k) \cdot 𝒙^{γ} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) p_{G_{N}} (y \| 𝒙) + \sum_{ρ = 1}^{4} R_{ρ}^{N} (𝒙, y)$
	$+ \sum_{k = 1}^{K_{0}} (\sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) - \exp (ω_{0 k}^{0} + t_{0})) [u (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}) - v (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1})]$
	$= \sum_{k = 1}^{K_{0}} \sum_{\| ℓ_{1} \| + ℓ_{2} = 0}^{2 \bar{r} (\| 𝔸_{k} \|)} T_{ℓ_{1}, ℓ_{2}}^{N} (k) \cdot 𝒙^{ℓ_{1}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) \frac{\partial^{ℓ_{2}} 𝒩}{\partial h_{1}^{ℓ_{2}}} (y \| 𝒂_{k}^{0 ⊤} 𝒙 + b_{k}^{0}, σ_{k}^{0})$
	$+ \sum_{k = 1}^{K_{0}} \sum_{\| γ \| = 0}^{1 + 𝟏_{{\| 𝔸_{k} \| > 1}}} S_{γ}^{N} (k) \cdot 𝒙^{γ} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) p_{G_{N}} (y \| 𝒙) + \sum_{ρ = 1}^{4} R_{ρ}^{N} (𝒙, y),$	(18)

with coefficients $T_{ℓ_{1}, ℓ_{2}}^{N} (k)$ and $S_{γ}^{N} (k)$ are defined for any $k \in [K_{0}]$ , $0 \leq | ℓ_{1} | + ℓ_{2} \leq 2 \bar{r} (| 𝔸_{k} |)$ and $0 \leq | γ | \leq 2$ as

	$T_{ℓ_{1}, ℓ_{2}}^{N} (k)$	$= {\begin{matrix} \sum_{ℓ \in 𝔸_{k}} \sum_{𝜶 \in 𝕀_{ℓ_{1}, ℓ_{2}}} \frac{\exp (ω_{0 ℓ}^{N})}{2^{α_{4}} 𝜶!} {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{𝜶_{1}} {(Δ 𝒂_{ℓ k}^{N})}^{𝜶_{2}} {(Δ b_{ℓ k}^{N})}^{α_{3}} {(Δ σ_{ℓ k}^{N})}^{α_{4}}, & (ℓ_{1}, ℓ_{2}) \neq (0_{D}, 0), \\ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) - \exp (ω_{0 k}^{0} + t_{0}), & (ℓ_{1}, ℓ_{2}) = (0_{D}, 0), \end{matrix}$
	$S_{γ}^{N} (k)$	$= {\begin{matrix} - \sum_{ℓ \in 𝔸_{k}} \frac{\exp (ω_{0 ℓ}^{N})}{γ!} {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{γ}, & \| γ \| \neq 0, \\ - \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) + \exp (ω_{0 k}^{0} + t_{0}), & \| γ \| = 0 . \end{matrix}$

Step 2: Non-vanishing coefficients

Next, we will show that not all the quatities $T_{ℓ_{1}, ℓ_{2}}^{N} (k) / D_{V} (G_{N}, G_{0})$ and $S_{γ}^{N} (k) / D_{V} (G_{N}, G_{0})$ go to $0$ as $N \to \infty$ . We assume that all of them go to $0$ as $N \to \infty$ . Then, by assumption $T_{0_{D}, 0}^{N} (k) / D_{V} (G_{N}, G_{0}) \to 0$ , we have

\frac{1}{D_{V} (G_{N}, G_{0})} ​ \sum_{k = 1}^{K_{0}} | \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ​ ℓ}^{N}) - \exp (ω_{0 ​ k}^{0} + t_{0}) | \to 0 .

(19)

For any $k$ such that $| 𝔸_{k} | = 1$ , consider all $(| ℓ_{1} |, ℓ_{2})$ implying $1 \leq | ℓ_{1} | + ℓ_{2} \leq 2$ , we have $T_{ℓ_{1}, ℓ_{2}}^{N} (k) / D_{V} (G_{N}, G_{0}) \to 0$ for all $k$ such that $| 𝔸_{k} | = 1$ . Hence

\frac{1}{D_{V} (G_{N}, G_{0})} ​ (\sum_{k : | 𝔸_{k} | = 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ​ ℓ}^{N}) ​ ‖ (Δ_{𝒕_{1}} ​ 𝝎_{1 ​ ℓ ​ k}^{N}, Δ ​ 𝒂_{ℓ ​ k}^{N}, Δ ​ b_{ℓ ​ k}^{N}, Δ ​ σ_{ℓ ​ k}^{N}) ‖) \to 0 .

(20)

Next, we consider $k$ such that $| 𝔸_{k} | > 1$ and $(| ℓ_{1} |, ℓ_{2})$ such that $1 \leq | ℓ_{1} | + ℓ_{2} \leq 2$ :

•

For $(| ℓ_{1} |, ℓ_{2}) = (0, 1)$ , then

$\frac{1}{D_{V} (G_{N}, G_{0})} ‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ b_{ℓ k}^{N}) ‖ \to 0 .$
•

For $(| ℓ_{1} |, ℓ_{2}) = (1, 0)$ , then

$\frac{1}{D_{V} (G_{N}, G_{0})} ‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) ‖ \to 0 .$
•

For $(| ℓ_{1} |, ℓ_{2}) = (1, 1)$ , then

$\frac{1}{D_{V} (G_{N}, G_{0})} ‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) (Δ b_{ℓ k}^{N}) + (Δ 𝒂_{ℓ k}^{N})] ‖ \to 0 .$
•

For $(| ℓ_{1} |, ℓ_{2}) = (0, 2)$ , then

$\frac{1}{D_{V} (G_{N}, G_{0})} ‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [{(Δ b_{ℓ k}^{N})}^{2} + (Δ σ_{ℓ k}^{N})] ‖ \to 0 .$
•

For $(| ℓ_{1} |, ℓ_{2}) = (2, 0)$ , then

$\frac{1}{D_{V} (G_{N}, G_{0})} ‖ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{⊤} ‖ \to 0 .$

Combining the above limit and the formulation of $D_{FRA} (G_{N}, G_{0})$ together, it follows that

\frac{1}{D_{V} (G_{N}, G_{0})} \cdot \sum_{k : | 𝔸_{k} | > 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ​ ℓ}) ​ ({‖ (Δ_{𝒕_{1}} ​ 𝝎_{1 ​ ℓ ​ k}^{N}, Δ ​ b_{ℓ ​ k}^{N}) ‖}^{\bar{r} ​ (| 𝔸_{k} |)} + {‖ (Δ ​ 𝒂_{ℓ ​ k}^{N}, Δ ​ σ_{ℓ ​ k}^{N}) ‖}^{\bar{r} ​ (| 𝔸_{k} |) / 2}) ↛ 0

which implies that there exists some index $k^{*} \in [K_{0}]$ such that $| 𝔸_{k^{*}} | > 1$ and

\frac{1}{D_{V} (G_{N}, G_{0})} \cdot \sum_{ℓ \in 𝔸_{k^{*}}} \exp (ω_{0 ​ ℓ}) ​ ({‖ (Δ_{v ​ t_{1}} ​ 𝝎_{1 ​ e ​ l ​ l ​ k^{*}}^{N}, Δ ​ b_{ℓ ​ k^{*}}^{N}) ‖}^{\bar{r} ​ (| 𝔸_{k} |)} + {‖ (Δ ​ 𝒂_{ℓ ​ k^{*}}^{N}, Δ ​ σ_{ℓ ​ k^{*}}^{N}) ‖}^{\bar{r} ​ (| 𝔸_{k} |) / 2}) ↛ 0

for all $𝒕_{1} \in ℝ^{D}$ . WLOG, we assume that $k^{*} = 1$ . For $(ℓ_{1}, ℓ_{2}) \in ℕ^{D} \times ℕ$ such that $1 \leq | ℓ_{1} | + ℓ_{2} \leq \bar{r} (| 𝔸_{1} |)$ , we have $T_{ℓ_{1}, ℓ_{2}}^{N} (1) / D_{V} (G_{N}, G_{0}) \to 0$ as $N \to \infty$ . Thus, by dividing this ratio and the left hand side of the above equation and let $𝒕_{1} = 0$ , we have

\frac{\sum_{ℓ \in 𝔸_{1}} \sum_{𝜶 \in 𝕀_{ℓ_{1}, ℓ_{2}}} \frac{\exp (ω_{0 ​ ℓ}^{N})}{2^{α_{4}} ​ α!} ​ {(Δ_{𝒕_{1}} ​ 𝝎_{1 ​ ℓ ​ 1}^{N})}^{𝜶_{1}} ​ {(Δ ​ 𝒂_{ℓ ​ 1}^{N})}^{𝜶_{2}} ​ {(Δ ​ b_{ℓ ​ 1}^{N})}^{α_{3}} ​ {(Δ ​ σ_{ℓ ​ 1}^{N})}^{α_{4}}}{\sum_{ℓ \in 𝔸_{1}} \exp (ω_{0 ​ ℓ}^{N}) ​ ({‖ (Δ_{𝒕_{1}} ​ 𝝎_{1 ​ ℓ ​ 1}^{N}, Δ ​ b_{ℓ ​ 1}^{N}) ‖}^{\bar{r} ​ (| 𝔸_{1} |)} + {‖ (Δ ​ 𝒂_{ℓ ​ 1}^{N}, Δ ​ σ_{ℓ ​ 1}^{N}) ‖}^{\bar{r} ​ (| 𝔸_{1} |) / 2})} \to 0

(21)

for all $(ℓ_{1}, ℓ_{2})$ such that $1 \leq | ℓ_{1} | + ℓ_{2} \leq \bar{r} (| 𝔸_{1} |)$ .
Let us define ${\bar{M}}_{N} := \max {‖ Δ_{𝒕_{1}} 𝝎_{1 ℓ 1}^{N} ‖, {‖ Δ 𝒂_{ℓ 1}^{N} ‖}^{1 / 2}, | Δ b_{ℓ 1}^{N} |, {| Δ σ_{ℓ 1}^{N} |}^{1 / 2} : ℓ \in 𝔸_{1}}$ and ${\bar{ω}}_{N} := \max_{ℓ \in 𝔸_{1}} \exp (ω_{0 ℓ}^{N})$ . Since the sequence $\exp (ω_{0 ℓ}^{N}) / {\bar{ω}}_{N}$ is bounded, we can replace it by its subsequence that has a positive limit $p_{5 ℓ}^{2} := {lim}_{N \to \infty} \exp (ω_{0 ℓ}^{N}) / {\bar{ω}}_{N}$ . Hence, at least one among $p_{5 ℓ}^{2}$ , for $ℓ \in 𝔸_{1}$ , equals $1$ .

Similarly, we also define

	$(Δ_{𝒕_{1}} 𝝎_{1 ℓ 1}^{N}) / {\bar{M}}_{N} \to p_{1 ℓ},$	$(Δ 𝒂_{ℓ 1}^{N}) / {\bar{M}}_{N} \to p_{2 ℓ},$
	$(Δ b_{ℓ 1}^{N}) / {\bar{M}}_{N} \to p_{3 ℓ},$	$(Δ σ_{ℓ 1}^{N}) / [2 {\bar{M}}_{N}] \to p_{4 ℓ} .$

Here, at least one of $p_{1 ℓ}, p_{2 ℓ}, p_{3 ℓ}$ and $p_{4 ℓ}$ for $ℓ \in 𝔸_{1}$ equals either $1$ or $- 1$ . Next, we divide both the numerator and the denominator of the ratio in eq. 21 by ${\bar{ω}}_{N} {\bar{M}}_{N}^{ℓ_{1} + ℓ_{2}}$ , and then achieve the following system of polynomial equations:

\sum_{ℓ \in 𝔸_{1}} \sum_{𝜶 \in 𝕀_{ℓ_{1}, ℓ_{2}}} \frac{1}{α!} \cdot p_{5 ​ ℓ}^{2} ​ p_{1 ​ ℓ}^{𝜶_{1}} ​ p_{2 ​ ℓ}^{𝜶_{2}} ​ p_{3 ​ ℓ}^{α_{3}} ​ p_{4 ​ ℓ}^{α_{4}} = 0

for all $(ℓ_{1}, ℓ_{2}) \in ℕ^{D} \times ℕ$ such that $1 \leq | ℓ_{1} | + ℓ_{2} \leq \bar{r} (| 𝔸_{1} |)$ . However, based on the definition of $\bar{r} (| 𝔸_{1} |)$ , the above system has no non-trivial solutions, which is a contradiction. Thus, not all the quantities $T_{ℓ_{1}, ℓ_{2}}^{N} (k) / D_{V} (G_{N}, G_{0})$ and $S_{γ}^{N} (k) / D_{V} (G_{N}, G_{0})$ go to $0$ as $N \to \infty$ .

Step 3: Fatou’s lemma involvement

Following this, we define by $m_{N}$ be the maximum of the absolute values of those quantities. Based on the result in Step 2, we know that $1 / m_{N} ↛ \infty$ . Then, by applying the Fatou’s lemma, we obtain that

lim_{N \to \infty} \frac{𝔼_{𝐱} [D_{TV} (p_{G} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))]}{m_{N} \cdot D_{V} (G_{N}, G_{0})} \geq \int \underset{N \to \infty}{lim inf} \frac{| p_{G} (y | 𝒙), p_{G_{0}} (y | 𝒙) |}{2 ​ m_{N} \cdot D_{V} (G_{N}, G_{0})} ​ d ​ (𝒙, y) .

(22)

By assumption, the left-hand side of eq. 22 equals to $0$ , so the integrand in the right-hand side also equals to $0$ for almost surely $(𝒙, y)$ . Hence, we get that $Q_{N} / [m_{N} D_{V} (G_{N}, G_{0})] \to 0$ as $N \to \infty$ for almost surely $(𝒙, y)$ . It follows from the decomposition of $Q_{N}$ in eq. 18 that

	$\sum_{k = 1}^{K_{0}} \sum_{\| ℓ_{1} \| + ℓ_{2} = 0}^{2 \bar{r} (\| 𝔸_{k} \|)}$	$τ_{ℓ_{1}, ℓ_{2}} (k) \cdot 𝒙^{ℓ_{1}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) \frac{\partial^{ℓ_{2}} 𝒩}{\partial h_{1}^{ℓ_{2}}} (y \| {(𝒂_{k}^{0})}^{⊤} 𝒙 + b_{k}^{0}, σ_{k}^{0})$
		$+ \sum_{k = 1}^{K_{0}} \sum_{\| γ \| = 0}^{1 + 𝟏_{{\| 𝔸_{k} \| > 1}}} ξ_{γ} (j) \cdot 𝒙^{γ} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) p_{G_{0}} (y \| 𝒙) = 0,$

for almost surely $(𝒙, y)$ , where $τ_{ℓ_{1}, ℓ_{2}} (k)$ and $ξ_{γ} (k)$ denote the limits of $T_{ℓ_{1}, ℓ_{2}}^{N} (k) / [m_{N} D_{V} (G_{N}, G_{0})]$ and $S_{γ}^{N} (j) / [m_{N} D_{V} (G_{N}, G_{0})]$ as $N \to \infty$ , respectively, for all $k \in [K_{0}], 0 \leq 2 | ℓ_{1} | + ℓ_{2} \leq 2 \bar{r} (| 𝔸_{k} |)$ and $0 \leq | γ | \leq 1 + 𝟏_{{| 𝔸_{k} | > 1}}$ . By definition, at least one among $τ_{ℓ_{1}, ℓ_{2}} (k)$ and $ξ_{γ} (k)$ is different from zero.

Furthermore, we denote the set $𝒲$ as follows:

	$𝒲$	$:= {𝒙^{ℓ_{1}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) \frac{\partial^{ℓ_{2}} 𝒩}{\partial h_{1}^{ℓ_{2}}} (y \| {(𝒂_{k}^{0})}^{⊤} 𝒙 + b_{k}^{0}, σ_{k}^{0}) : k \in [K_{0}], 0 \leq \| ℓ_{1} \| + ℓ_{2} \leq 2 \bar{r} (\| 𝔸_{k} \|)}$
		$\cup {𝒙^{γ} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) p_{G_{0}} (y \| 𝒙) : k \in [K_{0}], 0 \leq \| γ \| \leq 1 + 𝟏_{{\| 𝔸_{k} \| > 1}}} .$

Similarly to the proof of Fact 5 in Nguyen et al., 2023a:

Fact 5 (Nguyen et al., 2023a, Lemma 2).

The set $𝒲_{1}$ is linearly indeqendent w.r.t $𝐱$ and $y$ , where $𝒲_{1}$ is denoted as follows:

	$𝒲_{1}$	$:= {𝒙^{ℓ_{1}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) \frac{\partial^{ℓ_{2}} 𝒩}{\partial h_{1}^{ℓ_{2}}} (y \| {(𝒂_{k}^{0})}^{⊤} 𝒙 + b_{k}^{0}, σ_{k}^{0}) : k \in [K_{0}], 0 \leq \| ℓ_{1} \| + ℓ_{2} \leq 2}$
		$\cup {𝒙^{γ} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙) p_{G_{0}} (y \| 𝒙) : k \in [K_{0}], 0 \leq \| γ \| \leq 1},$

the set $𝒲$ is linearly independent w.r.t $𝒙$ and $y$ , it follows that

τ_{ℓ_{1}, ℓ_{2}} ​ (k) = ξ_{γ} ​ (k) = 0

for all $k \in [K_{0}], 0 \leq 2 | ℓ_{1} | + ℓ_{2} \leq 2 \bar{r} (| 𝔸_{k} |)$ and $0 \leq | γ | \leq 1 + 𝟏_{{| 𝔸_{k} | > 1}}$ , which is a contradiction. Hence, we achieve the eq. 17.

Global version: Hence, it is sufficient to prove its following global inequality:

inf_{G \in 𝒪_{K} : D_{FRA} (G, G_{0}) > ε^{'}} 𝔼_{𝐱} [D_{TV} (p_{G} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] / D_{FRA} (G, G_{0}) > 0 .

(23)

Assume by contrary that there exists a sequence $G_{N}^{'} \in 𝒪_{K}$ that satisfies

{\begin{matrix} {lim}_{N \to \infty} 𝔼_{𝐱} [D_{TV} (p_{G_{N}^{'}} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] / D_{FRA} (G_{N}^{'}, G_{0}) = 0, \\ D_{FRA} (G_{N}^{'}, G_{0}) > ε^{'} . \end{matrix}

Then, we get that $𝔼_{𝐱} [D_{TV} (p_{G_{N}^{'}} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] \to 0$ as $N \to \infty$ . Since the set $𝚯$ is compact, we can replace the sequence $G_{N}^{'}$ by its subsequence which converges to some mixing measure $G^{'} \in 𝒪_{K}$ such that $D_{FRA} (G^{'}, G_{0}) > ε^{'}$ . Then, by the Fatou’s lemma, we get

lim_{N \to \infty} 𝔼_{𝐱} [D_{TV} (p_{G_{N}^{'}} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] \geq \frac{1}{2} \int \underset{N \to \infty}{lim inf} | p_{G_{N}^{'}} (y | 𝒙) - p_{G_{0}} (y | 𝒙) | d (𝒙, y) .

It follows that

\int | p_{G^{'}} (y | 𝒙) - p_{G_{0}} (y | 𝒙) | d (𝒙, y) = 0 .

Thus, we obtain that $p_{G^{'}} (y | 𝒙) = p_{G_{0}} (y | 𝒙)$ for almost surely $(𝒙, y)$ . By Fact 3, the mixing measure $G^{'}$ admits the form $G^{'} = \sum_{k = 1}^{K_{0}} \exp (ω_{0 ν (k)}^{0} + t_{0}) δ_{(𝝎_{1 ν (k)}^{0} + 𝒕_{1}, 𝒂_{ν (k)}^{0}, b_{ν (k)}^{0}, σ_{ν (k)}^{0})}$ for some $(t_{0}, 𝒕_{1}) \in ℝ \times ℝ^{D}$ , where $ν$ is some permutation of the set ${1, 2, \dots, K_{0}}$ . It follows that $D_{FRA} (G^{'}, G_{0}) = 0$ , which contradicts the hypothesis $D_{FRA} (G^{'}, G_{0}) > ε^{'} > 0$ . Hence, we obtain the inequality in eq. 16. ∎

Next, assume that ${\hat{G}}_{N} \in ℰ_{K}$ with $K > K_{0}$ . From Fact 4, there exists a constant $c (𝚯, K)$ depending on $𝚯$ and $K$ so that on an event, we call $A_{N}$ , with probability at least $1 - C N^{- c}$ , we have

𝔼_{𝐱} [D_{TV} (p_{{\hat{G}}_{N}} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] \leq \sqrt{2} 𝔼_{𝐱} [D_{h}^{2} (p_{{\hat{G}}_{N}} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙))] \leq c (𝚯, K) \cdot {(\frac{\log N}{N})}^{1 / 2} .

Now, we prove Theorem 1.

Proof of Theorem 1.

Firstly, we prove for the over-specified case. By Lemma 1 and Theorem 5, we have the first statement.

To prove the rest, we need to consider the exact-specified case. When $κ^{'} = K_{0}$ , by definition of $D_{FRA} ({\hat{G}}_{N}^{(κ^{'})}, G_{0})$ , we obtain that $D_{FRA} ({\hat{G}}_{N}^{(K_{0})}, G_{0}) = D_{E} ({\hat{G}}_{N}^{(K_{0})}, G_{0})$ . Hence, by Lemma 1, we get the convergence rate

D_{E} ({\hat{G}}_{N}^{(K_{0})}, G_{0}) ≲ {(\frac{\log N}{N})}^{1 / 2} .

Assume that ${\hat{G}}_{N}^{(K_{0})} = \sum_{k = 1}^{K_{0}} \exp (ω_{0 k}^{N}) δ_{(𝝎_{1 k}^{N}, 𝒂_{k}^{N}, b_{k}^{N}, σ_{k}^{N})} \in ℰ_{K_{0}}$ . Building on our previous work, there exist $t_{0} \in ℝ$ and $𝒕_{1} \in ℝ^{D}$ such that for large $N$ enough, we get

	$\| \exp (ω_{0 k}^{N}) - \exp (ω_{0 k}^{0} + t_{0}) \|$	$≲ {(\frac{\log N}{N})}^{1 / 2},$
	$‖ (Δ_{𝒕_{1}} 𝝎_{1 k}^{N}, Δ 𝒂_{k}^{N}, Δ b_{k}^{N}, Δ σ_{k}^{N}) ‖$	$≲ {(\frac{\log N}{N})}^{1 / 2},$

for every $k \in [K_{0}]$ , where $Δ_{𝒕_{1}}^{N} 𝝎_{1 k}^{N} := 𝝎_{1 k}^{N} - 𝝎_{1 k}^{0} - 𝒕_{1}$ , $Δ 𝒂_{k}^{N} := 𝒂_{k}^{N} - 𝒂_{k}^{0}$ , $Δ b_{k}^{N} := b_{k}^{N} - b_{k}^{0}$ and $Δ σ_{k}^{N} := σ_{k}^{N} - σ_{k}^{0}$ .

This implies that for every $(i, j) \in {[K_{0}]}^{2}$ , by the triangle inequality, we have

	$\| ∥ (𝝎_{1 i}^{N} - 𝝎_{1 j}^{N}, b_{i}^{N} - b_{j}^{N}) ∥$	$- ∥ (𝝎_{1 i}^{0} - 𝝎_{1 j}^{0}, b_{i}^{0} - b_{j}^{0}) ∥ \| \leq ∥ (𝝎_{1 i}^{N} - 𝝎_{1 j}^{N} - 𝝎_{1 i}^{0} + 𝝎_{1 j}^{0}, b_{i}^{N} - b_{j}^{N} - b_{i}^{0} + b_{j}^{0}) ∥$
		$\leq ‖ (𝝎_{1 i}^{N} - 𝝎_{1 i}^{0} - 𝒕_{1}, b_{i}^{N} - b_{i}^{0}) ‖ + ‖ (𝝎_{1 j}^{N} - 𝝎_{1 j}^{0} - 𝒕_{1}, b_{j}^{N} - b_{j}^{0}) ‖$
		$≲ {(\frac{\log N}{N})}^{1 / 2} .$

Similarly, we have

| ‖ (𝒂_{i}^{N} - 𝒂_{j}^{N}, σ_{i}^{N} - σ_{j}^{N}) ‖ - ‖ (𝒂_{i}^{0} - 𝒂_{j}^{0}, σ_{i}^{0} - σ_{j}^{0}) ‖ | ≲ {(\frac{\log N}{N})}^{1 / 2} .

Hence, we obtain that

$\|$	$\frac{1}{\exp (- ω_{0 i}^{N}) + \exp (- ω_{0 j}^{N})} ({‖ (𝝎_{1 i}^{N} - 𝝎_{1 j}^{N}, b_{i}^{N} - b_{j}^{N}) ‖}^{2} + ‖ (𝒂_{i}^{N} - 𝒂_{j}^{N}, σ_{i}^{N} - σ_{j}^{N}) ‖)$
	$- \frac{1}{\exp (- ω_{0 i}^{0} - t_{0}) + \exp (- ω_{0 j}^{0} - t_{0})} (∥ (𝝎_{1 i}^{0} - 𝝎_{1 j}^{0}, b_{i}^{0} - b_{j}^{0}) ∥^{2} + ∥ (𝒂_{i}^{0} - 𝒂_{j}^{0}, σ_{i}^{0} - σ_{j}^{0}) ∥) \|$
	$≲ {(\frac{\log N}{N})}^{1 / 2}, \forall (i, j) \in {[K_{0}]}^{2} .$	(24)

Hence, on $A_{N}$ , the optimal choice of indices $(ℓ_{1}, ℓ_{2})$ to merge for ${\hat{G}}_{N}^{(K_{0})}$ will be the same as $G_{0}$ for every $N$ large enough. It follows that we have two merged atoms are $\exp (ω_{0, *}^{N}) δ_{(𝝎_{1, *}^{N}, 𝒂_{*}^{N}, b_{*}^{N}, σ_{*}^{N})}$ and $\exp (ω_{0, *}^{0}) δ_{(𝝎_{1, *}^{0}, 𝒂_{*}^{0}, b_{*}^{0}, σ_{*}^{0})}$ denoted as follows:

	$ω_{0, *}^{N}$	$= \log (\exp ω_{0 ℓ_{1}}^{N} + \exp ω_{0 ℓ_{2}}^{N}),$
	$𝝎_{1, *}^{N}$	$= \exp (ω_{0 ℓ_{1}}^{N} - ω_{0, }^{N}) 𝝎_{1 ℓ_{1}}^{N} + \exp (ω_{0 ℓ_{2}}^{N} - ω_{0, }^{N}) 𝝎_{1 ℓ_{2}}^{N},$
	$b_{*}^{N}$	$= \exp (ω_{0 ℓ_{1}}^{N} - ω_{0, }^{N}) b_{ℓ_{1}}^{N} + \exp (ω_{0 ℓ_{2}}^{N} - ω_{0, }^{N}) b_{ℓ_{2}}^{N},$
	$𝒂_{*}^{N}$	$= \frac{\exp (ω_{0 ℓ_{1}}^{N})}{\exp (ω_{0, }^{N})} [(𝝎_{1 ℓ_{1}}^{N} - 𝝎_{1, }^{N}) (b_{ℓ_{1}}^{N} - b_{}^{N}) + 𝒂_{ℓ_{1}}^{N}] + \frac{\exp (ω_{0 ℓ_{2}}^{N})}{\exp (ω_{0, }^{N})} [(𝝎_{1 ℓ_{2}}^{N} - 𝝎_{1, }^{N}) (b_{ℓ_{2}}^{N} - b_{}^{N}) + 𝒂_{ℓ_{2}}^{N}],$
	$σ_{*}^{N}$	$= \frac{\exp (ω_{0 ℓ_{1}}^{N})}{\exp (ω_{0, }^{N})} [{(b_{ℓ_{1}}^{N} - b_{}^{N})}^{2} + σ_{ℓ_{1}}^{N}] + \frac{\exp (ω_{0 ℓ_{2}}^{N})}{\exp (ω_{0, }^{N})} [{(b_{ℓ_{2}}^{N} - b_{}^{N})}^{2} + σ_{ℓ_{2}}^{N}],$

and

	$ω_{0, *}^{0}$	$= \log (\exp ω_{0 ℓ_{1}}^{0} + \exp ω_{0 ℓ_{2}}^{0}),$
	$𝝎_{1, *}^{0}$	$= \exp (ω_{0 ℓ_{1}}^{0} - ω_{0, }^{0}) 𝝎_{1 ℓ_{1}}^{0} + \exp (ω_{0 ℓ_{2}}^{0} - ω_{0, }^{0}) 𝝎_{1 ℓ_{2}}^{0},$
	$b_{*}^{0}$	$= \exp (ω_{0 ℓ_{1}}^{0} - ω_{0, }^{0}) b_{ℓ_{1}}^{0} + \exp (ω_{0 ℓ_{2}}^{0} - ω_{0, }^{0}) b_{ℓ_{2}}^{0},$
	$𝒂_{*}^{0}$	$= \frac{\exp (ω_{0 ℓ_{1}}^{0})}{\exp (ω_{0, }^{0})} [(𝝎_{1 ℓ_{1}}^{0} - 𝝎_{1, }^{0}) (b_{ℓ_{1}}^{0} - b_{}^{0}) + 𝒂_{ℓ_{1}}^{0}] + \frac{\exp (ω_{0 ℓ_{2}}^{0})}{\exp (ω_{0, }^{0})} [(𝝎_{1 ℓ_{2}}^{0} - 𝝎_{1, }^{0}) (b_{ℓ_{2}}^{0} - b_{}^{0}) + 𝒂_{ℓ_{2}}^{0}],$
	$σ_{*}^{0}$	$= \frac{\exp (ω_{0 ℓ_{1}}^{0})}{\exp (ω_{0, }^{0})} [{(b_{ℓ_{1}}^{0} - b_{}^{0})}^{2} + σ_{ℓ_{1}}^{0}] + \frac{\exp (ω_{0 ℓ_{2}}^{0})}{\exp (ω_{0, }^{0})} [{(b_{ℓ_{2}}^{0} - b_{}^{0})}^{2} + σ_{ℓ_{2}}^{0}] .$

After merging, we also have

$\| \exp (ω_{0, }^{N}) - \exp (ω_{0, }^{0} + t_{0}) \|$	$= \| \exp (ω_{0 ℓ_{1}}^{N}) + \exp (ω_{0 ℓ_{2}}^{N}) - \exp (ω_{0 ℓ_{1}}^{0} + t_{0}) - \exp (ω_{0 ℓ_{1}}^{0} + t_{0}) \|$
	$\leq \| \exp (ω_{0 ℓ_{1}}^{N}) - \exp (ω_{0 ℓ_{1}}^{0} + t_{0}) \| + \| \exp (ω_{0 ℓ_{1}}^{N}) - \exp (ω_{0 ℓ_{1}}^{0} + t_{0}) \|$
	$≲ {(\frac{\log N}{N})}^{1 / 2},$	(25)

and

	$\exp (ω_{0, *}^{N})$	$‖ (Δ_{𝒕_{1}}^{N} 𝝎_{1, }^{N}, Δ 𝒂_{}^{N}, Δ b_{}^{N}, Δ σ_{}^{N}) ‖$
		$\leq \exp (ω_{0, }^{N}) \times \exp (ω_{0 ℓ_{1}}^{N} - ω_{0, }^{N}) ‖ (Δ_{𝒕_{1}}^{N} 𝝎_{1 ℓ_{1}}^{N}, Δ 𝒂_{ℓ_{1}}^{N}, Δ b_{ℓ_{1}}^{N}, Δ σ_{ℓ_{1}}^{N}) ‖$
		$+ \exp (ω_{0, }^{N}) \times \exp (ω_{0 ℓ_{2}}^{N} - ω_{0, }^{N}) ‖ (Δ_{𝒕_{1}}^{N} 𝝎_{1 ℓ_{2}}^{N}, Δ 𝒂_{ℓ_{2}}^{N}, Δ b_{ℓ_{2}}^{N}, Δ σ_{ℓ_{2}}^{N}) ‖$
		$≲ {(\frac{\log N}{N})}^{1 / 2} .$

Hence, $D_{E} ({\hat{G}}_{N}^{(K_{0} - 1)}, G_{0}^{(K_{0} - 1)}) ≲ {(\frac{\log N}{N})}^{1 / 2}$ . By the induction, we have the rest statement. ∎

F.3 Proof of Theorem 2

For the convergence rate of the height at all levels $κ \geq K_{0} + 1$ , from Theorem 1, we have

D_{FRA} ({\hat{G}}_{N}^{(κ)}, G_{0}) ≲ {(\frac{\log N}{N})}^{1 / 2} .

Because $κ \geq K_{0} + 1$ , by the pigeonhole principle, there exists at least two $i, j \in [κ]$ such that two atoms $\exp (ω_{0 i}^{N}) δ_{(𝝎_{1 i}^{N}, 𝒂_{i}^{N}, b_{i}^{N}, σ_{i}^{N})}$ and $\exp (ω_{0 j}^{N}) δ_{(𝝎_{1 j}^{N}, 𝒂_{j}^{N}, b_{j}^{N}, σ_{j}^{N})}$ belongs to a common Voronoi cell of some $𝜽_{k}^{0}$ (we suppress the dependence of $i, j$ , and $𝔸_{k}$ on $N$ for ease of notation). Hence,

	$inf_{𝒕_{1}} \exp (ω_{0 i}) ({‖ (Δ_{𝒕_{1}} 𝝎_{1 i k}, Δ b_{i k}) ‖}^{\bar{r} (\| 𝔸_{k} \|)} + {‖ (Δ 𝒂_{i k}, Δ σ_{i k}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2})$
	$+ \exp (ω_{0 j}) ({‖ (Δ_{𝒕_{1}} 𝝎_{1 j k}, Δ b_{j k}) ‖}^{\bar{r} (\| 𝔸_{k} \|)} + {‖ (Δ 𝒂_{j k}, Δ σ_{j k}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2}) ≲ {(\frac{\log N}{N})}^{1 / 2} .$

Using the fact that $\min {\exp (ω_{0 i}), \exp (ω_{0 j})} \geq \frac{1}{\exp (- ω_{0 i}) + \exp (- ω_{0 j})}$ , $\bar{r} ({\hat{G}}_{N}) \geq \bar{r} (| 𝔸_{k} |) \geq \bar{r} (2) = 4$ , and using the Hölder’s inequality, for every $𝒕_{1} \in ℝ^{D}$ we have

	$\exp (ω_{0 i}) ({‖ (Δ_{𝒕_{1}} 𝝎_{1 i k}, Δ b_{i k}) ‖}^{\bar{r} (\| 𝔸_{k} \|)} + {‖ (Δ 𝒂_{i k}, Δ σ_{i k}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2})$
	$+ \exp (ω_{0 j}) ({‖ (Δ_{𝒕_{1}} 𝝎_{1 j k}, Δ b_{j k}) ‖}^{\bar{r} (\| 𝔸_{k} \|)} + {‖ (Δ 𝒂_{j k}, Δ σ_{j k}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2})$
	$\geq \frac{1}{\exp (- ω_{0 i}) + \exp (- ω_{0 j})} [(∥ (Δ_{𝒕_{1}} 𝝎_{1 i k}, Δ b_{i k}) ∥^{\bar{r} (\| 𝔸_{k} \|)} + ∥ (Δ_{𝒕_{1}} 𝝎_{1 j k}, Δ b_{j k}) ∥^{\bar{r} (\| 𝔸_{k} \|)})$
	$+ (∥ (Δ 𝒂_{i k}, Δ σ_{i k}) ∥^{\bar{r} (\| 𝔸_{k} \|) / 2} + ∥ (Δ 𝒂_{j k}, Δ σ_{j k}) ∥^{\bar{r} (\| 𝔸_{k} \|) / 2})]$
	$≳ \frac{1}{\exp (- ω_{0 i}) + \exp (- ω_{0 j})} ({‖ (𝝎_{1 i} - 𝝎_{1 j}, b_{i} - b_{j}) ‖}^{\bar{r} (\| 𝔸_{k} \|)} + {‖ (𝒂_{i} - 𝒂_{j}, σ_{i} - σ_{j}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2})$
	$≳ {(\frac{1}{\exp (- ω_{0 i}) + \exp (- ω_{0 j})} ({‖ (𝝎_{1 i} - 𝝎_{1 j}, b_{i} - b_{j}) ‖}^{2} + ‖ (𝒂_{i} - 𝒂_{j}, σ_{i} - σ_{j}) ‖))}^{\bar{r} ({\hat{G}}_{N}) / 2} .$

Since the height of the dendrogram is the minimum of $𝖽$ over all pairs $(i, j)$ , we obtain that

0 ​ p ​ t_{N}^{(κ)} ≲ \frac{1}{\exp (- ω_{0 ​ i}) + \exp (- ω_{0 ​ j})} ​ ({‖ (𝝎_{1 ​ i} - 𝝎_{1 ​ j}, b_{i} - b_{j}) ‖}^{2} + ‖ (𝒂_{i} - 𝒂_{j}, σ_{i} - σ_{j}) ‖) ≲ {(\frac{\log N}{N})}^{1 / \bar{r} ​ ({\hat{G}}_{N})},

for all $κ \geq K_{0} + 1$ .

When $κ \leq K_{0}$ , the conclusion follows from inequality in eq. 24 in the proof of Theorem 1.

F.4 Proof of Theorem 3

Before we prove Theorem 3, we revisit preliminary on empirical process theory and connection between the Hellinger distance and the Wasserstein metric.

Preliminary on Empirical Process Theory.

Suppose $𝐱_{1}, \dots, 𝐱_{N} \sim P_{G_{0}}$ . Denote $P_{N} := \frac{1}{N} \sum_{n = 1}^{N} δ_{𝐱_{n}}$ is the empirical measure. Denote the empirical process for $G$ :

ν_{N} ​ (G) := \sqrt{N} ​ (P_{N} - P_{G_{0}}) ​ \log \frac{{\bar{p}}_{G}}{p_{G_{0}}} .

The following results is important in proof below.

Fact 6 (van de Geer, 2000, Theorem 5.11).

Let positive numbers $R, C, C_{1}, a$ satisfy:

a \leq C_{1} ​ \sqrt{N} ​ R^{2} \land 8 ​ \sqrt{N} ​ R,

and

a \geq \sqrt{C^{2} ​ (C_{1} + 1)} ​ (\int_{a / (2^{6} ​ \sqrt{N})}^{R} H_{B}^{1 / 2} ​ (\frac{u}{\sqrt{2}}, {p_{G} : G \in 𝒪_{K}, D_{h}^{2} (p_{G}, p_{G_{0}}) \leq R}, ν) ​ d u \lor R),

then

ℙ_{G_{0}} ​ (sup_{G \in 𝒪_{K}, D_{h}^{2} (p_{G}, p_{G_{0}}) \leq R} | ν_{N} ​ (G) | \geq a) \leq C ​ \exp (- \frac{a^{2}}{C^{2} ​ (C_{1} + 1) ​ R^{2}}) .

Connection between the Hellinger distance and the Wasserstein metric.

We introduce the Wasserstein distances to measure the difference between two measures. For two mixing measure $G = \sum_{k = 1}^{K} p_{k} δ_{θ_{k}}$ and $G^{'} = \sum_{ℓ = 1}^{K^{'}} p_{ℓ}^{'} δ_{θ_{ℓ}^{'}}$ , the Wasserstein- $r$ distance (for $r \geq 1$ ) between $G$ and $G^{'}$ is defined as

W_{r} ​ (G, G^{'}) := {(inf_{𝒒 \in Π ​ (𝒑, 𝒑^{'})} ​ \sum_{k, ℓ = 1}^{K, K^{'}} q_{k ​ ℓ} ​ {‖ θ_{k} - θ_{ℓ}^{'} ‖}^{r})}^{1 / r},

(26)

where $Π (𝒑, 𝒑^{'})$ is the set of all couplings between $𝒑 = (p_{1}, \dots, p_{K})$ and $𝒑^{'} = (p_{1}^{'}, \dots, p_{K^{'}}^{'})$ , i.e, $Π (𝒑, 𝒑^{'}) = {𝒒 \in ℝ_{+}^{K \times K^{'}} : \sum_{k = 1}^{K} q_{k ℓ} = p_{ℓ}^{'}, \sum_{l = 1}^{K^{'}} q_{k ℓ} = p_{k}, \forall k \in [K], ℓ \in [K^{'}]}$ . Fix $G_{0} = \sum_{k = 1}^{K_{0}} π_{k}^{0} δ_{θ_{k}^{0}} \in ℰ_{K_{0}}$ , and consider $G = \sum_{ℓ = 1}^{K} π_{ℓ} δ_{θ_{ℓ}}$ such that $W_{r} (G, G_{0}) \to 0$ , we obtain that

W_{r}^{r} ​ (G, G_{0}) ≍ \sum_{k = 1}^{K_{0}} (| \sum_{ℓ \in 𝔸_{k} ​ (G)} π_{ℓ} - π_{k}^{0} | + \sum_{ℓ \in 𝔸_{k} ​ (G)} π_{ℓ} ​ {‖ θ_{ℓ} - θ_{k}^{0} ‖}^{r}) .

Now, we remind Lemma 1 in Ho and Nguyen (2016a).

Fact 7 (Ho and Nguyen, 2016a, Lemma 1).

Let $G = \sum_{i = 1}^{k} p_{i} δ_{θ_{i}}$ denote a discrete probability measure and $p_{G} (x) = \sum_{i = 1}^{k} p_{i} f (x | θ_{i})$ be the mixture density. According to the Lemma 1: Let $G, G^{'} \in 𝒪_{k} (Θ)$ such that both $ρ_{ϕ} (p_{G}, p_{G^{'}})$ and $d_{ρ_{ϕ}} (G, G^{'})$ are finite for some convex function $ϕ$ . Then, $ρ_{ϕ} (p_{G}, p_{G^{'}}) \leq d_{ρ_{ϕ}} (G, G^{'})$ .

By Fact 7, we can compare the expectation of Hellinger distance between $p_{G} (y | 𝒙)$ and $p_{G^{'}} (y | 𝒙)$ with the Wasserstein metric between $G$ and $G^{'}$ following:

𝔼_{𝐱} (D_{h}^{2} (p_{G} (\cdot | 𝒙), p_{G^{'}} (\cdot | 𝒙)) ≲ W_{2} (G, G^{'}) .

Now, we are going to prove Theorem 3.

Proof of Theorem 3.

Firstly, we recall the empirical average log-likelihood and population average log-likelihood as follows:

	${\bar{ℓ}}_{N} (p_{G})$	$= \frac{1}{N} \sum_{n = 1}^{N} \log p_{G} (y_{n} \| 𝐱_{n}) = : P_{N} \log p_{G},$
	$ℒ (p_{G})$	$= 𝔼_{(𝐱, y) \sim P_{G_{0}}} [\log p_{G} (y \| 𝒙)] = \int \log p_{G} (y \| 𝒙) d P_{G_{0}} (𝒙, y) = : P_{G_{0}} \log p_{G},$

where $P_{N} := \frac{1}{N} \sum_{n = 1}^{N} δ_{(𝐱_{n}, y_{n})}$ is the empirical measure from data, and the joint distribution $P_{G_{0}}$ over $(𝒙, y)$ is then constructed by first sampling $𝒙 \sim P_{𝐱}$ and then $y | 𝒙 \sim p_{G_{0}} (y | 𝒙)$ .

We divide into three cases.

Case 1: $κ \geq K_{0}$ . For any $G$ , we denote $P_{G}$ by the distribution of $p_{G}$ . By the concavity of log function, we have

\frac{1}{2} ​ \log \frac{p_{G}}{p_{G_{0}}} \leq \log \frac{p_{G} + p_{G_{0}}}{2 ​ p_{G_{0}}} = \log \frac{{\bar{p}}_{G}}{p_{G_{0}}}, \forall G \in 𝒪_{K} .

Therefore, for all $κ > K_{0}$ we have

	$\frac{1}{2} P_{N} \log \frac{p_{{\hat{G}}_{N}^{(κ)}}}{p_{G_{0}}}$	$\leq P_{N} \log \frac{{\bar{p}}_{{\hat{G}}_{N}^{(κ)}}}{p_{G_{0}}}$
		$= (P_{N} - P_{G_{0}}) \log \frac{{\bar{p}}_{{\hat{G}}_{N}^{(κ)}}}{p_{G_{0}}} - KL (p_{G_{0}} ∥ {\bar{p}}_{{\hat{G}}_{N}^{(κ)}})$
		$\leq (P_{N} - P_{G_{0}}) \log \frac{{\bar{p}}_{{\hat{G}}_{N}^{(κ)}}}{p_{G_{0}}} .$

Hence,

	$P_{N} \log p_{{\hat{G}}_{N}^{(κ)}} - P_{G_{0}} \log p_{G_{0}}$	$= P_{N} \log \frac{p_{{\hat{G}}_{N}^{(κ)}}}{p_{G_{0}}} + (P_{N} - P_{G_{0}}) \log p_{G_{0}}$
		$\leq 2 (P_{N} - P_{G_{0}}) \log \frac{{\bar{p}}_{{\hat{G}}_{N}^{(κ)}}}{p_{G_{0}}} + (P_{N} - P_{G_{0}}) \log p_{G_{0}} .$

By Theorem 1, we obtain that $D_{FRA} ({\hat{G}}_{N}^{(κ)}, G_{0}) ≲ {(\log N / N)}^{1 / 2}$ , and obviously we have

inf_{t_{0}, 𝒕_{1}} W_{\bar{r} ​ ({\hat{G}}_{N})}^{\bar{r} ​ ({\hat{G}}_{N})} ​ ({\hat{G}}_{N}^{(κ)}, G_{0, t_{0}, 𝒕_{1}}) \leq D_{FRA} ({\hat{G}}_{N}^{(κ)}, G_{0}),

where $G_{0, t_{0}, 𝒕_{1}} = \sum_{k = 1}^{K_{0}} \exp (ω_{0 k}^{0} + t_{0}) δ_{(𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0})}$ , so there exists a constant $D$ such that

ℙ_{G_{0}} ​ (inf_{t_{0}, 𝒕_{1}} W_{\bar{r} ​ ({\hat{G}}_{N})} ​ ({\hat{G}}_{N}^{(κ)}, G_{0, t_{0}, 𝒕_{1}}) \leq D ​ {(\frac{\log N}{N})}^{1 / 2 ​ \bar{r} ​ ({\hat{G}}_{N})}) \geq 1 - c_{1} ​ N^{- c_{2}}, κ \in [K_{0}, K] .

Now, we compare Wasserstein metrics $W_{2}$ and $W_{\bar{r} ({\hat{G}}_{N})}$ . Since $2 / \bar{r} ({\hat{G}}_{N}) \leq 2 / 4 < 1$ , with a note that for a probability $q_{i, j}$ , we have $q_{i, j} \leq q_{i, j}^{2 / \bar{r} ({\hat{G}}_{N})}$ . Combining with all norms on finite space is equivalent, we obtain that

{(\sum_{i, j} q_{i, j} ​ {‖ θ_{k} - θ_{ℓ}^{'} ‖}^{2})}^{1 / 2} \leq {(\sum_{i, j} q_{i, j}^{2 / \bar{r} ​ ({\hat{G}}_{N})} ​ {‖ θ_{k} - θ_{ℓ}^{'} ‖}^{2})}^{1 / 2} ≲ {(\sum_{i, j} q_{i, j} ​ {‖ θ_{k} - θ_{ℓ}^{'} ‖}^{\bar{r} ​ ({\hat{G}}_{N})})}^{1 / \bar{r} ​ ({\hat{G}}_{N})} .

Then, we get $W_{2} ≲ W_{\bar{r} ({\hat{G}}_{N})}$ . Using the fact that $𝔼_{𝐱} (D_{h}^{2} (p_{G} (\cdot | 𝒙), p_{G^{'}} (\cdot | 𝒙)) ≲ W_{2} (G, G^{'})$ and $p_{G_{0}} = p_{G_{0, t_{0}, 𝒕_{1}}}, \forall (t_{0}, 𝒕_{1}) \in ℝ \times ℝ^{D}$ , we also have

	$ℙ_{G_{0}}$	$(𝔼 (D_{h}^{2} (p_{{\hat{G}}_{N}^{(κ)}} (\cdot \| 𝒙), p_{G_{0}} (\cdot \| 𝒙)) \leq D {(\frac{\log N}{N})}^{1 / 2 \bar{r} ({\hat{G}}_{N})})$
		$= ℙ_{G_{0}} (inf_{t_{0}, 𝒕_{1}} 𝔼_{𝐱} (D_{h}^{2} (p_{{\hat{G}}_{N}^{(κ)}} (\cdot \| 𝒙), p_{G_{0, t_{0}, 𝒕_{1}}} (\cdot \| 𝒙)) \leq D {(\frac{\log N}{N})}^{1 / 2 \bar{r} ({\hat{G}}_{N})})$
		$\geq ℙ_{G_{0}} (inf_{t_{0}, 𝒕_{1}} W_{\bar{r} ({\hat{G}}_{N})} ({\hat{G}}_{N}^{(κ)}, G_{0, t_{0}, 𝒕_{1}}) \leq D {(\frac{\log N}{N})}^{1 / 2 \bar{r} ({\hat{G}}_{N})}) \geq 1 - c_{1} N^{- c_{2}}, κ \in [K_{0}, K] .$

Let $𝒫_{K} (Θ) := {p_{G} (y | 𝒙) : G \in 𝒪_{K} (Θ)}$ and $H_{B} (ε, 𝒫_{K} (Θ), h)$ denotes the bracketing entropy of $𝒫_{K} (Θ)$ under the Hellinger distance. By the Lemma 3 in Nguyen et al. (2023a), there is a constant $C > 0$ such that $H_{B} (ε, 𝒫_{K} (Θ), h) ≲ \log (1 / ε)$ for any $0 \leq ε \leq 1 / 2$ .

Define, $α := 1 / 2 \bar{r} ({\hat{G}}_{N}) \leq 1 / 4$ , substitute $R = D {(\frac{\log N}{N})}^{α}$ , $a = D \frac{\log^{α + 1 / 2} N}{N^{α}}$ , then for any positive number $ε < R$ , we have $0 \leq ε \leq 1 / e < 1 / 2$ and $\log (1 / ε) > 1$ for large $N$ enough. Therefore, for large $N$ enough, we obtain that $a \leq \sqrt{N} R^{2} \leq \sqrt{N} R$ and

	$a \geq R (\log (\frac{2^{6} \sqrt{N}}{a}))$	$\geq \int_{a / (2^{6} \sqrt{N})}^{R} \log \frac{1}{ε} d ε$
		$\geq \int_{a / (2^{6} \sqrt{N})}^{R} \log^{1 / 2} \frac{1}{ε} d ε$
		$\geq \int_{a / (2^{6} \sqrt{N})}^{R} H_{B}^{1 / 2} (ε, 𝒫_{K} (Θ), h) 𝑑 ε$
		$\geq \int_{a / (2^{6} \sqrt{N})}^{R} H_{B}^{1 / 2} (ε, {p_{G} : G \in 𝒪_{K} (Θ), 𝔼_{𝐱} (D_{h}^{2} (p_{G} (\cdot \| 𝒙), p_{G_{0}} (\cdot \| 𝒙)) \leq R}, ν) d ε .$

By Fact 6, we get

ℙ_{G_{0}} ​ (sup_{𝔼_{𝐱} (D_{h}^{2} (p_{G} (\cdot | 𝒙), p_{G_{0}} (\cdot | 𝒙)) \leq D {(\log N / N)}^{α}} | \sqrt{N} ​ (P_{N} - P_{G_{0}}) ​ \log \frac{{\bar{p}}_{G}}{p_{G_{0}}} | \geq D ​ \frac{\log^{α + 1 / 2} N}{N^{α}}) \leq N^{- c_{2}} .

Combining with the bound on Hellinger distance, we have

	$ℙ_{G_{0}} (\| (P_{N} - P_{G_{0}}) \log \frac{{\bar{p}}_{{\hat{G}}_{N}^{(κ)}}}{p_{G_{0}}} \| \geq D \frac{\log^{α + 1 / 2} N}{N^{α + 1 / 2}})$
	$\leq ℙ_{G_{0}} (𝔼_{𝐱} (D_{h}^{2} (p_{{\hat{G}}_{N}^{(κ)}} (\cdot \| 𝒙), p_{G_{0}} (\cdot \| 𝒙)) \geq D {(\log N / N)}^{α})$
	$+ ℙ_{G_{0}} (\| (P_{N} - P_{G_{0}}) \log \frac{{\bar{p}}_{{\hat{G}}_{N}^{(κ)}}}{p_{G_{0}}} \| \geq D \frac{\log^{α + 1 / 2} N}{N^{α + 1 / 2}}, 𝔼_{𝐱} (D_{h}^{2} (p_{{\hat{G}}_{N}^{(κ)}} (\cdot \| 𝒙), p_{G_{0}} (\cdot \| 𝒙)) \leq D {(\log N / N)}^{α})$
	$\leq c_{1} N^{- c_{2}} + ℙ_{G_{0}} (sup_{𝔼_{𝐱} (D_{h}^{2} (p_{G} (\cdot \| 𝒙), p_{G_{0}} (\cdot \| 𝒙)) \leq D {(\log N / N)}^{α}} \| \sqrt{N} (P_{N} - P_{G_{0}}) \log \frac{{\bar{p}}_{G}}{p_{G_{0}}} \| \geq D \frac{\log^{α + 1 / 2} N}{N^{α}})$
	$\leq c_{1}^{'} N^{- c_{2}} .$

For the second term, by the Chebyshev inequality, we have

ℙ_{G_{0}} ​ (| (P_{N} - P_{G_{0}}) ​ \log p_{G_{0}} | \geq t) \leq \frac{Var ​ (\log p_{G_{0}})}{N ​ t^{2}} .

(27)

Choose $t = {(\log N / N)}^{α}$ , we have

ℙ_{G_{0}} ​ (| (P_{N} - P_{G_{0}}) ​ \log p_{G_{0}} | \leq {(\log N / N)}^{α}) \geq 1 - c_{1} ​ N^{- c_{2}} .

Hence, we conclude that

ℙ_{G_{0}} ​ ({\bar{ℓ}}_{N} ​ ({\hat{G}}_{N}^{(κ)}) - ℒ ​ (p_{G_{0}}) \leq {(\frac{\log N}{N})}^{1 / 2 ​ \bar{r} ​ ({\hat{G}}_{N})}) \geq 1 - c_{1} ​ N^{- c_{2}} .

Case 2: $κ = K_{0}$ . By the Theorem 1, we have

D_{E} ({\hat{G}}_{N}^{(K_{0})}, G_{0}) ≲ {(\frac{\log N}{N})}^{1 / 2} .

Assume that ${\hat{G}}_{N}^{(K_{0})} = \sum_{k = 1}^{K_{0}} \exp (ω_{0 k}^{N}) δ_{(𝝎_{1 k}^{N}, 𝒂_{k}^{N}, b_{k}^{N}, σ_{k}^{N})}$ , since $D_{E} ({\hat{G}}_{N}^{(K_{0})}, G_{0}) \to 0$ as $N \to \infty$ , the Voronoi cell $𝔸_{k}$ has only one element for any $k \in [K_{0}]$ . WLOG, we suppose that $𝔸_{k} = {k}$ for all $k \in [K_{0}]$ . Moreover, there exist $t_{0} \in ℝ$ and $𝒕_{1} \in ℝ^{D}$ independent of $N$ such that $\exp (ω_{0 k}^{N}) \to \exp (ω_{0 k}^{0} + t_{0})$ and $𝝎_{1 k}^{N} \to 𝝎_{1 k}^{0} + 𝒕_{1}$ as $N \to \infty$ for all $k \in [K_{0}]$ . By the definition of $D_{E} ({\hat{G}}_{N}^{(K_{0})}, G_{0})$ , we get for large $N$ enough

| \exp (ω_{0 ​ k}^{N}) - \exp (ω_{0 ​ k}^{0} + t_{0}) | ≲ {(\frac{\log N}{N})}^{1 / 2}, ‖ (Δ_{𝒕_{1}} ​ 𝝎_{1 ​ k}^{N}, Δ ​ 𝒂_{k}^{N}, Δ ​ b_{k}^{N}, Δ ​ σ_{k}^{N}) ‖ ≲ {(\frac{\log N}{N})}^{1 / 2},

for every $k \in [K_{0}]$ , where $Δ_{𝒕_{1}}^{N} 𝝎_{1 k}^{N} := 𝝎_{1 k}^{N} - 𝝎_{1 k}^{0} - 𝒕_{1}$ , $Δ 𝒂_{k}^{N} := 𝒂_{k}^{N} - 𝒂_{k}^{0}$ , $Δ b_{k}^{N} := b_{k}^{N} - b_{k}^{0}$ and $Δ σ_{k}^{N} := σ_{k}^{N} - σ_{k}^{0}$ .

Because the function $f (𝒙, y | 𝜽) = u (y | 𝒙; 𝝎_{1}, 𝒂, b, σ)$ satisfies Condition K (see Lemma 2), let $ϵ_{N} = {(\log N / N)}^{1 / 2} \to 0$ , from condition K, there exist $c_{α}$ and $c_{ω}$ such that

u ​ (y | 𝒙; 𝝎_{1 ​ k}^{N}, 𝒂_{k}^{N}, b_{k}^{N}, σ_{k}^{N}) \geq {(u ​ (y | 𝒙; 𝝎_{1 ​ k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}))}^{(1 + c_{ω} ​ ϵ_{N})} ​ e^{- c_{α} ​ ϵ_{N}}, \forall k \in [K_{0}] .

Besides, we can find constant $c_{q} > 0$ and $c_{p} > 0$ such that

	$\exp (ω_{0 k}^{N})$	$\geq (1 - c_{p} ϵ_{N}) \exp (ω_{0 k}^{0} + t_{0}),$	$\forall k \in [K_{0}],$
	$\sum_{k = 1}^{K_{0}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙 + ω_{0 k}^{0} + t_{0})$	$\geq (1 - c_{q} ϵ_{N}) \sum_{k = 1}^{K_{0}} \exp ({(𝝎_{1 k}^{N})}^{⊤} 𝒙 + ω_{0 k}^{N}),$	$\forall k \in [K_{0}] .$

Hence, we have

	$[\sum_{k = 1}^{K_{0}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙 + ω_{0 k}^{0} + t_{0})] \cdot p_{{\hat{G}}_{N}^{(K_{0})}} (y \| 𝒙)$
	$= \frac{\sum_{k = 1}^{K_{0}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙 + ω_{0 k}^{0} + t_{0})}{\sum_{k = 1}^{K_{0}} \exp ({(𝝎_{1 k}^{N})}^{⊤} 𝒙 + ω_{0 k}^{N})} \cdot \sum_{k = 1}^{K_{0}} \exp (ω_{0 k}^{N}) u (y \| 𝒙; 𝝎_{1 k}^{N}, 𝒂_{k}^{N}, b_{k}^{N}, σ_{k}^{N})$
	$\geq (1 - c_{q} ϵ) \sum_{k = 1}^{K_{0}} (1 - c_{p} ϵ) \exp (ω_{0 k}^{0} + t_{0}) {(u (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}))}^{(1 + c_{ω} ϵ_{N})} e^{- c_{α} ϵ_{N}} .$

With the fact that $g (t) = t^{1 + c_{ω} ϵ_{N}}$ is a convex function, we get

	$p_{{\hat{G}}_{N}^{(K_{0})}} (y \| 𝒙)$	$\geq (1 - c_{q} ϵ) (1 - c_{p} ϵ) \frac{1}{\sum_{k = 1}^{K_{0}} \exp ({(𝝎_{1 k}^{0} + 𝒕_{1})}^{⊤} 𝒙 + ω_{0 k}^{0} + t_{0})}$
		$\times \sum_{k = 1}^{K_{0}} \exp (ω_{0 k}^{0} + t_{0}) {(u (y \| 𝒙; 𝝎_{1 k}^{0} + 𝒕_{1}, 𝒂_{k}^{0}, b_{k}^{0}, σ_{k}^{0}))}^{(1 + c_{ω} ϵ_{N})} e^{- c_{α} ϵ_{N}}$
		$\geq (1 - c_{q} ϵ) (1 - c_{p} ϵ) e^{- c_{α} ϵ_{N}} \sum_{k = 1}^{K_{0}} \frac{\exp ({(ω_{1 k}^{0})}^{⊤} 𝒙 + ω_{0 k}^{0})}{\sum_{j = 1}^{K_{0}} \exp ({(ω_{1 j}^{0})}^{⊤} 𝒙 + ω_{0 j}^{0})} \cdot 𝒩 {(y \| 𝒂_{k}^{0} 𝒙 + b_{k}^{0}, σ_{k}^{0})}^{(1 + c_{ω} ϵ_{N})}$
		$\geq (1 - c_{q} ϵ) (1 - c_{p} ϵ) e^{- c_{α} ϵ_{N}} p_{G_{0}} {(y \| 𝒙)}^{(1 + c_{ω} ϵ_{N})} .$

Therefore, we have

\frac{1}{N} ​ \sum_{i = 1}^{N} \log \frac{p_{{\hat{G}}_{N}^{(K_{0})}}}{p_{G_{0}}} ​ (y_{i} | 𝒙_{i}) \geq \log ((1 - c_{q} ​ ϵ) ​ (1 - c_{p} ​ ϵ)) - (c_{α} ​ ϵ_{N}) + (c_{ω} ​ ϵ_{N}) ​ \frac{1}{N} ​ \sum_{i = 1}^{N} \log p_{G_{0}} ​ (y_{i} | 𝒙_{i}) .

Hence

\bar{ℓ} ​ (p_{{\hat{G}}_{N}^{(K_{0})}}) - ℒ ​ (p_{G_{0}})

\geq \log ((1 - c_{q} ​ ϵ) ​ (1 - c_{p} ​ ϵ)) - (c_{α} ​ ϵ_{N}) + (c_{ω} ​ ϵ_{N}) ​ P_{G_{0}} ​ \log p_{G_{0}} + (1 + c_{ω} ​ ϵ_{N}) ​ (P_{N} - P_{G_{0}}) ​ \log p_{G_{0}} .

(28)

Now, we will bound the right-hand side of above equation, from Chebyshev inequality from eq. 27, choose $t = {(\log N / N)}^{1 / 2}$ , we get that

ℙ_{G_{0}} ​ (| (P_{N} - P_{G_{0}}) ​ \log p_{G_{0}} | \geq {(\frac{\log N}{N})}^{1 / 2}) \leq \frac{Var ​ (\log p_{G_{0}})}{\log N} .

Obviously, the terms $| \log ((1 - c_{q} ϵ) (1 - c_{p} ϵ)) - (c_{α} ϵ_{N}) + (c_{ω} ϵ_{N}) P_{G_{0}} \log p_{G_{0}} | ≲ ϵ_{N} = {(\log N / N)}^{1 / 2}$ , thus there exist a constant $C > 0$ such that $\log ((1 - c_{q} ϵ) (1 - c_{p} ϵ)) - (c_{α} ϵ_{N}) + (c_{ω} ϵ_{N}) P_{G_{0}} \log p_{G_{0}} \geq - C {(\log N / N)}^{1 / 2}$ . Then, for some constant $C_{e} > 0$ , we have

ℙ_{G_{0}} ​ (RHS of eq. 28 \geq - C_{e} ​ {(\frac{\log N}{N})}^{1 / 2}) \geq 1 - \frac{Var ​ (\log p_{G_{0}})}{\log N} .

Call the event under above case is $B$ , then we obtain that

	$ℙ_{G_{0}} (\bar{ℓ} (p_{{\hat{G}}_{N}^{(K_{0})}}) - ℒ (p_{G_{0}}) \geq - C_{e} {(\frac{\log N}{N})}^{1 / 2})$	$\geq ℙ_{G_{0}} (A_{N} \cap B) = ℙ_{G_{0}} (B) - ℙ_{G_{0}} (B \cap A_{N}^{c})$
		$\geq ℙ_{G_{0}} (B) - ℙ_{G_{0}} (A_{N}^{c}) = 1 - \frac{Var (\log p_{G_{0}})}{\log N} - c_{1} N^{- c_{2}}$

approach $1$ when $N \to \infty$ , where $A_{N}$ is defined in Section F.2. Therefore, combine both results, we can conclude that

| \bar{ℓ} ​ (p_{{\hat{G}}_{N}^{(K_{0})}}) - ℒ ​ (p_{G_{0}}) | ≲ {(\frac{\log N}{N})}^{1 / 2 ​ \bar{r} ​ ({\hat{G}}_{N})} .

Case 3: $κ < K_{0}$ . Since $| \log p_{G} (y | 𝒙) | \leq m (y | 𝒙)$ for a measurable function $m$ for all $G \in 𝒪_{κ}$ , we can use uniform law of large number to get that

sup_{G \in 𝒪_{κ}} | {\bar{ℓ}}_{N} ​ (G) - P_{G_{0}} ​ \log p_{G} | ​ \overset{ℙ}{⟶} ​ 0,

where $\overset{ℙ}{⟶}$ means convergence in probability. Therefore,

| {\bar{ℓ}}_{N} ​ ({\hat{G}}_{N}^{(κ)}) - P_{G_{0}} ​ \log p_{{\hat{G}}_{N}^{(κ)}} | ​ \overset{ℙ}{⟶} ​ 0 .

We know that $\log p_{{\hat{G}}_{N}^{(κ)}} \to \log p_{G_{0}^{(κ)}}$ in probability, by application of Dominated Convergence theorem, we obtain

P_{G_{0}} ​ \log p_{{\hat{G}}_{N}^{(κ)}} ​ \overset{ℙ}{⟶} ​ P_{G_{0}} ​ \log p_{G_{0}^{(κ)}} .

Combining the above results together, we get

{\bar{ℓ}}_{N} ​ ({\hat{G}}_{N}^{(κ)}) ​ \overset{ℙ}{⟶} ​ P_{G_{0}} ​ \log p_{G_{0}^{(κ)}} = ℒ ​ (\log P_{G_{0}^{(κ)}}) .

∎

Checking condition K.

Finally, we check condition K for the function $f (𝒙, y | 𝜽) := \exp (𝝎_{1}^{⊤} 𝒙) 𝒩 (y | 𝒂^{⊤} 𝒙 + b, σ)$ .

Lemma 2.

The condition K is satisfied for $f (𝐱, y | 𝛉) := \exp (𝛚_{1}^{⊤} 𝐱) 𝒩 (y | 𝐚^{⊤} 𝐱 + b, σ)$ , where $𝛉 = (𝛚_{1}, 𝐚, b, σ) \in ℝ^{D} \times ℝ^{D} \times ℝ \times ℝ$ and $𝒳$ are bounded as from the initial setup, and the eigenvalues of $σ$ are bounded below and above by the positive constants $σ_{\min}$ and $σ_{\max}$ .

Proof of Lemma 2.

When $‖ 𝜽 - 𝜽^{0} ‖ \leq ϵ$ with $𝜽^{0} = (𝝎_{1}^{0}, 𝒂^{0}, b^{0}, σ^{0})$ , by the equivalence of the norm, we can consider the cases where $‖ 𝝎_{1} - 𝝎_{1}^{0} ‖, ‖ 𝒂 - 𝒂^{0} ‖, ‖ b - b^{0} ‖, ‖ σ - σ^{0} ‖ \leq ϵ$ . We aim to show that for sufficiently small $ϵ$ , there exist $c_{α}, c_{β} > 0$ such that

\log (\exp (𝝎_{1}^{⊤} ​ 𝒙) ​ 𝒩 ​ (y | 𝒂^{⊤} ​ 𝒙 + b, σ)) \geq (1 + c_{β} ​ ϵ) ​ \log (\exp ({(𝝎_{1}^{0})}^{⊤} ​ 𝒙) ​ 𝒩 ​ (y | {(𝒂^{0})}^{⊤} ​ 𝒙 + b^{0}, σ^{0})) - c_{α} ​ ϵ .

which is equivalent to

		$[(1 + c_{β} ϵ) {(𝝎_{1}^{0})}^{⊤} 𝒙 - {(𝝎_{1})}^{⊤} 𝒙] + [(1 + c_{β} ϵ) \log (\| σ^{0} \|) - \log (\| σ \|)]$
	$+$	$[(1 + c_{β} ϵ) {(y - {(𝒂^{0})}^{⊤} 𝒙 - b^{0})}^{⊤} {(σ^{0})}^{- 1} (y - {(𝒂^{0})}^{⊤} 𝒙 - b^{0}) - {(y - 𝒂^{⊤} 𝒙 - b)}^{⊤} {(σ)}^{- 1} (y - 𝒂^{⊤} 𝒙 - b)] + c_{α} ϵ \geq 0 .$

Firstly, since $𝒳$ is bounded, we can omit the term $[(1 + c_{β} ϵ) {(𝝎_{1}^{0})}^{⊤} 𝒙 - {(𝝎_{1})}^{⊤} 𝒙]$ . Next, we note that

\frac{d ​ \log (| σ |)}{d ​ σ} = σ^{- 1}

and if $‖ σ ‖$ is bounded above and below far from 0 (which satisfies because $σ$ is positive definite), then the map $σ \mapsto \log (| σ |)$ is Lipschitz; that is, there exists a constant $c_{σ}$ such that

| \log (| σ^{0} |) - \log (| σ |) | \leq c_{σ} ​ ‖ σ^{0} - σ ‖ .

Furthermore, we have $| σ | \geq σ_{\min}$ . Hence, for all $c_{β} > \frac{c_{σ}}{\log (σ_{\min})}$ , then we have

c_{β} ​ ϵ ​ \log (| σ^{0} |) \geq c_{σ} ​ ϵ \geq c_{σ} ​ ‖ σ - σ^{0} ‖ \geq | \log (| σ |) - \log (| σ^{0} |) | .

So that

(1 + c_{β} ​ ϵ) ​ \log (| σ^{0} |) \geq \log (| σ |) .

We want to choose $c_{α} > 0$ such that

[(1 + c_{β} ​ ϵ) ​ {(y - {(𝒂^{0})}^{⊤} ​ 𝒙 - b^{0})}^{⊤} ​ {(σ^{0})}^{- 1} ​ (y - {(𝒂^{0})}^{⊤} ​ 𝒙 - b^{0}) - {(y - 𝒂^{⊤} ​ 𝒙 - b)}^{⊤} ​ {(σ)}^{- 1} ​ (y - 𝒂^{⊤} ​ 𝒙 - b)] + c_{α} ​ ϵ \geq 0 .

Let $u := y - {(𝒂^{0})}^{⊤} 𝒙 - b^{0}, Δ u := {(𝒂^{0})}^{⊤} 𝒙 + b^{0} - [𝒂^{⊤} 𝒙 + b]$ , using the boundedness of $σ$ , there exist $c_{σ}$ such that

{(σ^{0})}^{- 1} \geq c_{σ} ​ σ^{- 1} .

Hence, we only need to prove

(1 + c_{β} ​ ϵ) ​ c_{σ} ​ u^{⊤} ​ σ^{- 1} ​ u - {(u + Δ ​ u)}^{⊤} ​ σ^{- 1} ​ (u + Δ ​ u) + c_{α} ​ ϵ \geq 0,

which is equivalent to

		$c_{β} ϵ c_{σ} u^{⊤} σ^{- 1} u - u^{⊤} σ^{- 1} Δ u - {(Δ u)}^{⊤} σ^{- 1} u - {(Δ u)}^{⊤} σ^{- 1} Δ u + c_{α} ϵ \geq 0$
	$\Leftrightarrow$	$ϵ c_{β} c_{σ} {(u - \frac{Δ u}{ϵ c_{β} c_{σ}})}^{⊤} σ^{- 1} (u - \frac{Δ u}{ϵ c_{β} c_{σ}}) + c_{α} ϵ \geq (1 + \frac{1}{ϵ c_{β} c_{σ}}) {(Δ u)}^{⊤} σ^{- 1} (Δ u) .$

We can bound the right-hand side of above equation as follow

(1 + \frac{1}{ϵ ​ c_{β} ​ c_{σ}}) ​ {(Δ ​ u)}^{⊤} ​ σ^{- 1} ​ (Δ ​ u) \leq (1 + \frac{1}{ϵ ​ c_{β} ​ c_{σ}}) ​ \frac{{‖ Δ ​ u ‖}^{2}}{σ_{\min}} \leq (1 + \frac{1}{ϵ ​ c_{β} ​ c_{σ}}) ​ \frac{ϵ^{2}}{σ_{\min}} .

Hence, it is sufficient to choose $c_{α}$ such that

c_{α} \geq (1 + \frac{1}{ϵ ​ c_{β} ​ c_{σ}}) ​ \frac{ϵ}{σ_{\min}} = \frac{ϵ}{σ_{\min}} + \frac{1}{c_{β} ​ c_{σ} ​ σ_{\min}} .

Then $(1 + c_{β} ϵ) {(y - {(𝒂^{0})}^{⊤} 𝒙 - b^{0})}^{⊤} {(σ^{0})}^{- 1} (y - {(𝒂^{0})}^{⊤} 𝒙 - b^{0}) - {(y - 𝒂^{⊤} 𝒙 - b)}^{⊤} {(σ)}^{- 1} (y - 𝒂^{⊤} 𝒙 - b) + c_{α} ϵ \geq 0$ .

Therefore, we complete the proof. ∎

F.5 Proof of Theorem 4

Define ${DSC}_{N}^{(κ)} = - (0 p t_{N}^{(κ)} + ϵ_{N} {\bar{ℓ}}_{N} (p_{{\hat{G}}_{N}^{(κ)}}))$ with $1 ≪ ϵ_{N} ≪ {(N / \log N)}^{1 / (2 \bar{r} ({\hat{G}}_{N}))}$ (e.g., $ϵ_{N} = \log N$ ). For $κ > K_{0}$ , $0 p t_{N}^{(κ)}$ shrinks at order ${(\log N / N)}^{1 / \bar{r} ({\hat{G}}_{N})}$ while the likelihood term cannot compensate at that scale given the chosen $ϵ_{N}$ , so ${DSC}_{N}^{(κ)}$ is suboptimal. For $κ < K_{0}$ , the (under-fit) likelihood gap dominates and ${DSC}_{N}^{(κ)}$ is worse than at $κ = K_{0}$ . Hence ${\hat{K}}_{N} = {\arg \min}_{κ} {DSC}_{N}^{(κ)} \to K_{0}$ in probability. We will give a more detailed proof below.

Proof of Theorem 4.

Note that entropy $H (p_{G_{0}}) = - ℒ (p_{G_{0}})$ . We have

0 ​ p ​ t_{N}^{(κ)} = {\begin{matrix} O ​ ({(\frac{\log N}{N})}^{1 / \bar{r} ​ ({\hat{G}}_{N})}), & if ​ κ > K_{0} \\ 0 ​ p ​ t_{0}^{(κ)} + O ​ ({(\frac{\log N}{N})}^{1 / 2}), & if ​ κ \leq K_{0} \end{matrix}

and in the proof of Theorem 3, we get

{\begin{matrix} {\bar{ℓ}}_{N}^{(κ)} \leq - H ​ (p_{G_{0}}) + O ​ ({(\frac{\log N}{N})}^{1 / 2 ​ \bar{r} ​ ({\hat{G}}_{N})}), & if ​ κ > K_{0} \\ {\bar{ℓ}}_{N}^{(κ)} = - H ​ (p_{G_{0}}) + O ​ ({(\frac{\log N}{N})}^{1 / 2 ​ \bar{r} ​ ({\hat{G}}_{N})}), & if ​ κ = K_{0} \\ {\bar{ℓ}}_{N}^{(κ)} = - H ​ (p_{G_{0}}) - KL ​ (p_{G_{0}} ∥ p_{G_{0}}^{(κ)}) + o ​ (1), & if ​ κ < K_{0} \end{matrix}

Then we have

{\begin{matrix} {DSC}_{N}^{(κ)} \geq ϵ_{N} ​ H ​ (p_{G_{0}}) + O ​ (ϵ_{N} ​ {(\frac{\log N}{N})}^{1 / 2 ​ \bar{r} ​ ({\hat{G}}_{N})}), & if ​ κ > K_{0} \\ {DSC}_{N}^{(κ)} = ϵ_{N} ​ H ​ (p_{G_{0}}) - 0 ​ p ​ t_{0}^{(κ)} + O ​ (ϵ_{N} ​ {(\frac{\log N}{N})}^{1 / 2 ​ \bar{r} ​ ({\hat{G}}_{N})}), & if ​ κ = K_{0} \\ {DSC}_{N}^{(κ)} = ϵ_{N} ​ H ​ (p_{G_{0}}) + ϵ_{N} ​ KL ​ (p_{G_{0}} ∥ p_{G_{0}}^{(κ)}) - 0 ​ p ​ t_{0}^{(κ)} + o ​ (ϵ_{N}), & if ​ κ < K_{0} \end{matrix}

Since $ϵ_{N} \to \infty, ϵ_{N} {(\log N / N)}^{1 / 2} \bar{r} ({\hat{G}}_{N}) \to 0$ and $KL (p_{G_{0}} ∥ p_{G_{0}}^{(κ)}) > 0$ , then as $N \to \infty, {DSC}_{N}^{K_{0}}$ is the smallest number. Hence, $ℙ_{p_{G_{0}}} ({\hat{K}}_{N} = K_{0}) \geq ℙ_{p_{G_{0}}} (A_{N}) \to 1$ as $N \to \infty$ , or ${\hat{K}}_{N} \to K_{0}$ in probability. ∎

[bib.bib130] H. Akaike (1974) A new look at the statistical model identification. IEEE Transactions on Automatic Control 19 (6), pp. 716–723. External Links: Document Cited by: §1. ↩ 1

[bib.bib492] A. Anandkumar, D. Hsu, and S. M. Kakade (2012) A method of moments for mixture models and hidden markov models. In COLT, Cited by: §1. ↩ 1

[bib.bib468] S. Balakrishnan, M. J. Wainwright, and B. Yu (2017) Statistical guarantees for the EM algorithm: from population to sample-based analysis. Annals of Statistics 45, pp. 77–120. Cited by: §1. ↩ 1

[bib.bib495] H. Bao, W. Wang, L. Dong, Q. Liu, O-K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei (2022) VLMo: unified vision-language pre-training with mixture-of-modality-experts. In Advances in Neural Information Processing Systems, Cited by: §1. ↩ 1

[bib.bib28] M. Berrettini, G. Galimberti, S. Ranciati, and T. B. Murphy (2024) Identifying Brexit voting patterns in the British house of commons: an analysis based on Bayesian mixture models with flexible concomitant covariate effects. Journal of the Royal Statistical Society Series C: Applied Statistics 73 (3), pp. 621–638. External Links: ISSN 0035-9254, Link, Document Cited by: §1. ↩ 1

[bib.bib85] C. Biernacki, G. Celeux, and G. Govaert (2000) Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence 22 (7), pp. 719–725. External Links: Document Cited by: §1. ↩ 1

[bib.bib5] M. Blein-Nicolas, E. Devijver, M. Gallopin, and E. Perthame (2024) Nonlinear network-based quantitative trait prediction from biological data. Journal of the Royal Statistical Society Series C: Applied Statistics 73 (3), pp. 796–815. External Links: ISSN 0035-9254, Document, Link, https://academic.oup.com/jrsssc/article-pdf/73/3/796/58185677/qlae012.pdf Cited by: Appendix A, Appendix A, Appendix A, §5, §5. ↩ 5 A

[bib.bib4] M. Blein-Nicolas, S. S. Negro, T. Balliau, C. Welcker, L. Cabrera-Bosquet, S. D. Nicolas, A. Charcosset, and M. Zivy (2020) A systems genetics approach reveals environment-dependent associations between snps, protein coexpression, and drought-related traits in maize. Genome Research 30 (11), pp. 1593–1604. Cited by: Appendix A, §5. ↩ 5 A

[bib.bib508] F. Chamroukhi, A. Samé, G. Govaert, and P. Aknin (2009) Time series modeling by a regression approach based on a latent process. Neural Networks 22 (5–6), pp. 593–602. External Links: ISSN 0893-6080, Link, Document Cited by: §4. ↩ 4

[bib.bib500] J. Chen (1995) Optimal Rate of Convergence for Finite Mixture Models. The Annals of Statistics 23 (1), pp. 221 – 233. Note: Publisher: Institute of Mathematical Statistics External Links: Link, Document Cited by: §1, §3.7. ↩ 1 3.7

[bib.bib62] Z. Chen, Y. Deng, Y. Wu, Q. Gu, and Y. Li (2022) Towards Understanding the Mixture-of-Experts Layer in Deep Learning. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: §1. ↩ 1

[bib.bib21] M. C. Chong, H. D. Nguyen, and TrungTin Nguyen (2024) Risk Bounds for Mixture Density Estimation on Compact Domains via the h-Lifted Kullback–Leibler Divergence. Transactions on Machine Learning Research. Note: Accept with minor revision External Links: Link Cited by: §1. ↩ 1

[bib.bib510] D. Dai, C. Deng, C. Zhao, R.x. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, Z. Xie, Y.k. Li, P. Huang, F. Luo, C. Ruan, Z. Sui, and W. Liang (2024) DeepSeekMoE: towards ultimate expert specialization in mixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 1280–1297. External Links: Link, Document Cited by: Appendix B. ↩ B

[bib.bib506] D. Do, L. Do, S. A. McKinley, J. Terhorst, and X. Nguyen (2024) Dendrogram of mixing measures: Learning latent hierarchy and model selection for finite mixture models. arXiv preprint arXiv:2403.01684. Cited by: Appendix B, §1, §6. ↩ 1 6 B

[bib.bib15] D. Do, L. Do, and X. Nguyen (2025) Strong identifiability and parameter learning in regression with heterogeneous response. Electronic Journal of Statistics 19 (1), pp. 131 – 203. Note: Publisher: Institute of Mathematical Statistics and Bernoulli Society External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib58] T. G. Do, H. K. Le, T. Nguyen, Q. Pham, B. T. Nguyen, T. Doan, C. Liu, S. Ramasamy, X. Li, and S. HOI (2023) HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore. Cited by: Appendix B, §1. ↩ 1 B

[bib.bib496] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, External Links: Link Cited by: §1. ↩ 1

[bib.bib230] N. Doss, Y. Wu, P. Yang, and H. H. Zhou (2023) Optimal estimation of high-dimensional Gaussian location mixtures. The Annals of Statistics 51 (1), pp. 62 – 95. Note: Publisher: Institute of Mathematical Statistics External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib470] R. Dwivedi, N. Ho, K. Khamaru, M. J. Wainwright, M. I. Jordan, and B. Yu (2020a) Sharp analysis of expectation-maximization for weakly identifiable models. AISTATS. Cited by: §1. ↩ 1

[bib.bib469] R. Dwivedi, N. Ho, K. Khamaru, M. J. Wainwright, M. I. Jordan, and B. Yu (2020b) Singularity, misspecification, and the convergence rate of EM. Annals of Statistics 44, pp. 2726–2755. Cited by: §1. ↩ 1

[bib.bib479] D. Eigen, M. Ranzato, and I. Sutskever (2014) Learning factored representations in a deep mixture of experts. In ICLR Workshops, Cited by: §1. ↩ 1

[bib.bib480] W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, pp. 1–39. Cited by: Appendix B, §1. ↩ 1 B

[bib.bib80] F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel (2022a) Mixture of expert posterior surrogates for approximate Bayesian computation. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique (SFdS), Lyon, France. Cited by: §1. ↩ 1

[bib.bib153] F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel (2022b) Summary statistics and discrepancy measures for approximate Bayesian computation via surrogate posteriors. Statistics and Computing 32 (5), pp. 85. External Links: ISSN 1573-1375, Link, Document Cited by: §1. ↩ 1

[bib.bib22] R. Foygel and M. Drton (2010) Extended Bayesian Information Criteria for Gaussian Graphical Models. In Advances in Neural Information Processing Systems, J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (Eds.), Vol. 23. External Links: Link Cited by: §1. ↩ 1

[bib.bib30] S. Frühwirth-Schnatter, C. Pamminger, A. Weber, and R. Winter-Ebmer (2012) Labor market entry and earnings dynamics: Bayesian inference using mixtures-of-experts Markov chain clustering. Journal of Applied Econometrics 27 (7), pp. 1116–1137. External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib29] S. Frühwirth-Schnatter, S. Pittner, A. Weber, and R. Winter-Ebmer (2018) Analysing plant closure effects using time-varying mixture-of-experts Markov chain clustering. The Annals of Applied Statistics 12 (3), pp. 1796 – 1830. Note: Publisher: Institute of Mathematical Statistics External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib27] S. Frühwirth-Schnatter (2019) Keeping the balance—Bridge sampling for marginal likelihood estimation in finite mixture, mixture of experts and Markov mixture models. Brazilian Journal of Probability and Statistics 33 (4), pp. 706 – 733. External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib142] C. R. Genovese and L. Wasserman (2000) Rates of convergence for the Gaussian mixture sieve. The Annals of Statistics 28 (4), pp. 1105 – 1127. External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib165] A. Guha, N. Ho, and X. Nguyen (2021) On posterior contraction of parameters and interpretability in Bayesian mixture modeling. Bernoulli 27 (4), pp. 2159 – 2188. External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib493] M. Hardt and E. Price (2015) Tight bounds for learning a mixture of two gaussians. In STOC, Cited by: §1. ↩ 1

[bib.bib472] P. Heinrich and J. Kahn (2018) Strong identifiability and optimal minimax rates for finite mixture estimation. The Annals of Statistics 46 (6), pp. 2844–2870. Cited by: §1. ↩ 1

[bib.bib52] B. H. Ho, L. N. Chi, T. Nguyen, V. H. Hoang, B. T. Nguyen, and C. Drovandi (2025) A Unified Framework for Variable Selection in Model-Based Clustering with Missing Not at Random. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1. ↩ 1

[bib.bib139] N. Ho and X. Nguyen (2016a) Convergence rates of parameter estimation for some weakly identifiable finite mixtures. The Annals of Statistics 44 (6), pp. 2726 – 2755. External Links: Link, Document Cited by: §F.4, §1, §1, Fact 7. ↩ 1 F.4

[bib.bib138] N. Ho and X. Nguyen (2016b) On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics 10 (1), pp. 271–307. Cited by: §1. ↩ 1

[bib.bib145] N. Ho, C. Yang, and M. I. Jordan (2022) Convergence Rates for Gaussian Mixtures of Experts. Journal of Machine Learning Research 23 (323), pp. 1–81. External Links: Link Cited by: §1, §1. ↩ 1

[bib.bib151] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton (1991) Adaptive mixtures of local experts. Neural computation 3 (1), pp. 79–87. Cited by: §1. ↩ 1

[bib.bib505] W. Jiang and M. A. Tanner (1999) On the identifiability of mixtures-of-experts. Neural Networks 12 (9), pp. 1253–1258. External Links: ISSN 0893-6080, Link, Document Cited by: §F.2. ↩ F.2

[bib.bib150] M. I. Jordan and R. A. Jacobs (1994) Hierarchical mixtures of experts and the EM algorithm. Neural computation 6 (2), pp. 181–214. Cited by: §1. ↩ 1

[bib.bib16] A. Khalili, A. Y. Yang, and X. Da (2024) Estimation and group-feature selection in sparse mixture-of-experts with diverging number of parameters. Journal of Statistical Planning and Inference, pp. 106250. External Links: ISSN 0378-3758, Link, Document Cited by: §1. ↩ 1

[bib.bib224] H. Liang, Z. Fan, R. Sarkar, Z. Jiang, T. Chen, K. Zou, Y. Cheng, C. Hao, and Z. Wang (2022) M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design. In NeurIPS, Cited by: §1. ↩ 1

[bib.bib486] T. Manole and N. Ho (2020) Uniform convergence rates for maximum likelihood estimation under two-component gaussian mixture models. arXiv preprint arXiv:2006.00704. Cited by: §1. ↩ 1

[bib.bib235] L. Montuelle and E. Le Pennec (2014) Mixture of Gaussian regressions model with logistic weights, a penalized maximum likelihood approach. Electronic Journal of Statistics 8 (1), pp. 1661–1695. Cited by: §1. ↩ 1

[bib.bib23] D. N. Nguyen and Z. Li (2024) Joint learning of Gaussian graphical models in heterogeneous dependencies of high-dimensional transcriptomic data. In The 16th Asian Conference on Machine Learning (Conference Track), External Links: Link Cited by: §1. ↩ 1

[bib.bib143] H. D. Nguyen, F. Chamroukhi, and F. Forbes (2019) Approximation results regarding the multiple-output Gaussian gated mixture of linear experts model. Neurocomputing 366, pp. 208–214. External Links: ISSN 0925-2312, Link, Document Cited by: §1. ↩ 1

[bib.bib132] H. D. Nguyen and F. Chamroukhi (2018) Practical and theoretical aspects of mixture-of-experts modeling: An overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8 (4), pp. e1246. Cited by: §1. ↩ 1

[bib.bib146] H. D. Nguyen, L. R. Lloyd-Jones, and G. J. McLachlan (2016) A universal approximation theorem for mixture-of-experts models. Neural computation 28 (12), pp. 2585–2593. Cited by: §1. ↩ 1

[bib.bib51] H. D. Nguyen, M. Gupta, J. Westerhout, and T. Nguyen (2025a) On the large-sample limits of some Bayesian model evaluation statistics. arXiv preprint arXiv:2502.03846. Cited by: §1. ↩ 1

[bib.bib207] H. D. Nguyen, T. Nguyen, F. Chamroukhi, and G. J. McLachlan (2021a) Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models. Journal of Statistical Distributions and Applications 8 (1), pp. 13. External Links: ISSN 2195-5832, Link, Document Cited by: §1. ↩ 1

[bib.bib8] H. D. Nguyen, T. Nguyen, J. Westerhout, and X. Guo (2025b) Approximation rates for finite mixtures of location-scale models. arXiv preprint arXiv:2508.10612. Cited by: §1. ↩ 1

[bib.bib7] H. D. Nguyen and T. Nguyen (2025) Modifications of the BIC for order selection in finite mixture models. 2506.20124. External Links: Link Cited by: §1. ↩ 1

[bib.bib57] H. Nguyen, P. Akbarian, T. Nguyen, and N. Ho (2024a) A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts. In Proceedings of The 41st International Conference on Machine Learning, Cited by: §1. ↩ 1

[bib.bib511] H. Nguyen, P. Akbarian, F. Yan, and N. Ho (2024b) Statistical perspective of top-k sparse softmax gating mixture of experts. External Links: 2309.13850, Link Cited by: Appendix B. ↩ B

[bib.bib507] H. Nguyen, T. Nguyen, and N. Ho (2023a) Demystifying Softmax Gating Function in Gaussian Mixture of Experts. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, pp. 4624–4652. Cited by: Appendix B, Appendix C, Appendix D, §E.2, §E.2, §F.2, §F.2, §F.4, §1, §1, §1, §2, §2, §3.7, §3.7, Fact 1, Fact 2, Fact 3, Fact 4, Fact 5. ↩ 1 2 3.7 B C D E.2 F.2 F.4

[bib.bib44] H. Nguyen, T. Nguyen, K. Nguyen, and N. Ho (2024c) Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, S. Dasgupta, S. Mandt, and Y. Li (Eds.), Proceedings of Machine Learning Research, Vol. 238, pp. 2683–2691. External Links: Link Cited by: §1. ↩ 1

[bib.bib135] T. Nguyen, F. Chamroukhi, H. D. Nguyen, and G. J. McLachlan (2023b) Approximation of probability density functions via location-scale finite mixtures in Lebesgue spaces. Communications in Statistics - Theory and Methods 52 (14), pp. 5048–5059. External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib82] T. Nguyen, F. Chamroukhi, H. D. Nguyen, and F. Forbes (2021b) Non-asymptotic model selection in block-diagonal mixture of polynomial experts models. Preprint. arXiv:2104.08959. External Links: Link Cited by: §1. ↩ 1

[bib.bib81] T. Nguyen, F. Chamroukhi, H. D. Nguyen, and F. Forbes (2022a) Model selection by penalization in mixture of experts models with a non-asymptotic approach. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique (SFdS), Lyon, France. Cited by: §1. ↩ 1

[bib.bib17] T. Nguyen, F. Forbes, J. Arbel, and H. Duy Nguyen (2024d) Bayesian nonparametric mixture of experts for inverse problems. Journal of Nonparametric Statistics, pp. 1–60. Note: doi: 10.1080/10485252.2024.2426091 External Links: ISSN 1048-5252, Link, Document Cited by: §1. ↩ 1

[bib.bib84] T. Nguyen, D. N. Nguyen, H. D. Nguyen, and F. Chamroukhi (2023c) A non-asymptotic theory for model selection in high-dimensional mixture of experts via joint rank and variable selection. In AJCAI Australasian Joint Conference on Artificial Intelligence 2023, External Links: Link Cited by: §1. ↩ 1

[bib.bib83] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and G. J. McLachlan (2023d) Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts. arXiv:2009.10622. External Links: Link Cited by: §1. ↩ 1

[bib.bib154] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and F. Forbes (2022b) A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models. Electronic Journal of Statistics 16 (2), pp. 4742 – 4822. External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib137] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and G. J. McLachlan (2020) Approximation by finite mixtures of continuous density functions that vanish at infinity. Cogent Mathematics & Statistics 7 (1), pp. 1750861. External Links: ISSN null, Link, Document Cited by: §1. ↩ 1

[bib.bib131] T. Nguyen (2021) Model Selection and Approximation in High-dimensional Mixtures of Experts Models: from Theory to Practice. PhD Thesis, Normandie Université, (en). External Links: Link Cited by: §1. ↩ 1

[bib.bib140] X. Nguyen (2013) Convergence of latent mixing measures in finite and infinite mixture models. The Annals of Statistics 41 (1), pp. 370–400. External Links: Link, Document Cited by: §1, §1. ↩ 1

[bib.bib147] A. Norets (2010) Approximation of conditional densities by smooth mixtures of regressions. The Annals of Statistics 38 (3), pp. 1733 – 1766. External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib221] F. Peng, R. A. Jacobs, and M. A. Tanner (1996) Bayesian Inference in Mixtures-of-Experts and Hierarchical Mixtures-of-Experts Models With an Application to Speech Recognition. Journal of the American Statistical Association 91 (435), pp. 953–960. External Links: ISSN 01621459 Cited by: §1. ↩ 1

[bib.bib18] Q. Pham, G. Do, H. Nguyen, T. Nguyen, C. Liu, M. Sartipi, B. T. Nguyen, S. Ramasamy, X. Li, S. Hoi, et al. (2024) CompeteSMoE–Effective Training of Sparse Mixture of Experts via Competition. arXiv preprint arXiv:2402.02526. Cited by: Appendix B, §1. ↩ 1 B

[bib.bib1] S. A. Prado, L. Cabrera-Bosquet, A. Grau, A. Coupel-Ledru, E. J. Millet, C. Welcker, and F. Tardieu (2018) Phenomics allows identification of genomic regions affecting maize stomatal conductance with conditional effects of water deficit and evaporative demand. Plant, Cell & Environment 41 (2), pp. 314–326. Cited by: Appendix A, §5. ↩ 5 A

[bib.bib141] A. Rakhlin, D. Panchenko, and S. Mukherjee (2005) Risk bounds for mixture density estimation. ESAIM: PS 9, pp. 220–229. External Links: Link, Document Cited by: §1. ↩ 1

[bib.bib129] G. Schwarz (1978) Estimating the dimension of a model. The Annals of Statistics 6 (2), pp. 461–464. Cited by: §1. ↩ 1

[bib.bib91] W. Shen, S. T. Tokdar, and S. Ghosal (2013) Adaptive Bayesian multivariate density estimation with Dirichlet mixtures. Biometrika 100 (3), pp. 623–640. Cited by: §1. ↩ 1

[bib.bib25] C. Sin and H. White (1996) Information criteria for selecting possibly misspecified parametric models. Journal of Econometrics 71 (1), pp. 207–225. External Links: ISSN 0304-4076, Link, Document Cited by: §1. ↩ 1

[bib.bib504] B. Sturmfels (2002) Solving systems of polynomial equations. CBMS Regional Conference Series in Mathematics, American Mathematical Soc.. Cited by: §2. ↩ 2

[bib.bib6] T. Thai, T. Nguyen, D. Do, N. Ho, and C. Drovandi (2025) Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures. arXiv preprint arXiv:2505.13052. Cited by: Appendix B, Appendix B, §1. ↩ 1 B

[bib.bib3] T. Tran, T. Nguyen, M. A. Bashar, N. Ho, R. Nayak, and C. Drovandi (2026a) Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models. Note: _eprint: 2602.07997 External Links: Link Cited by: §1. ↩ 1

[bib.bib2] T. Tran, T. Nguyen, G. Fort, T. Doan, H. D. Nguyen, B. T. Nguyen, F. Forbes, and C. Drovandi (2026b) Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts. arXiv preprint arXiv:2601.19811. Cited by: §1. ↩ 1

[bib.bib502] S. van de Geer (2000) Empirical Processes in M-estimation. Vol. 6, Cambridge university press. Cited by: §E.4, §3.7, Fact 6. ↩ 3.7 E.4 F.4

[bib.bib501] C. Villani (2003) Topics in optimal transportation. Graduate Studies in Mathematics, Vol. 58, American Mathematical Society. External Links: ISBN 0-8218-3312-X Cited by: §3.7. ↩ 3.7

[bib.bib161] C. Villani (2009) Optimal transport: old and new. Vol. 338, Springer. Cited by: §3.7. ↩ 3.7

[bib.bib33] J. Westerhout, T. Nguyen, X. Guo, and H. D. Nguyen (2024) On the Asymptotic Distribution of the Minimum Empirical Risk. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: §1. ↩ 1

[bib.bib491] Y. Wu and H. H. Zhou (2021) Randomly initialized EM algorithm for two-component Gaussian mixture achieves near optimality in $O (\sqrt{n})$ iterations. Mathematical Statistics and Learning 4, pp. 143–220. Cited by: §1. ↩ 1

[bib.bib471] Y. Wu and P. Yang (2020) Optimal estimation of Gaussian mixtures via denoised method of moments. The Annals of Statistics 48, pp. 1987–2007. Cited by: §1. ↩ 1

[bib.bib482] Z. You, S. Feng, D. Su, and D. Yu (2021) SpeechMoE: scaling to large acoustic models with dynamic routing mixture of experts. In Interspeech, Cited by: §1. ↩ 1

[bib.bib483] Z. You, S. Feng, D. Su, and D. Yu (2022) Speechmoe2: mixture-of-experts model with improved routing. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7217–7221. External Links: Document Cited by: §1. ↩ 1

[bib.bib134] S. E. Yuksel, J. N. Wilson, and P. D. Gader (2012) Twenty Years of Mixture of Experts. IEEE Transactions on Neural Networks and Learning Systems 23 (8), pp. 1177–1193. External Links: ISSN 2162-2388 VO - 23, Document Cited by: §1. ↩ 1

[bib.bib26] G. Zens (2019) Bayesian shrinkage in mixture-of-experts models: identifying robust determinants of class membership. Advances in Data Analysis and Classification 13 (4), pp. 1019–1051. External Links: ISSN 1862-5355, Link, Document Cited by: §1. ↩ 1

	$D_{O} (G, G_{0}) := D_{E} (G, G_{0})$
	$+ inf_{t_{0}, 𝒕_{1}} \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) (∥ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}, Δ b_{ℓ k}) ∥^{\bar{r} (\| 𝔸_{k} \|)}$
	$+ ∥ (Δ 𝒂_{ℓ k}, Δ σ_{ℓ k}) ∥^{\bar{r} (\| 𝔸_{k} \|) / 2}),$		(4)
	$D_{E} (G, G_{0}) := inf_{t_{0}, 𝒕_{1}} \sum_{k = 1}^{K_{0}} \| \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) - \exp (ω_{0 k}^{0} + t_{0}) \|$
	$+ \sum_{k : \| 𝔸_{k} \| = 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}) ‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}, Δ 𝒂_{ℓ k}, Δ b_{ℓ k}, Δ σ_{ℓ k}) ‖ .$

	$(\exp ω_{0 i} + \exp ω_{0 j}) {‖ (Δ_{𝒕_{1}} 𝝎_{1 * k}, Δ b_{* k}) ‖}^{\bar{r} (\| 𝔸_{k} \|)}$	$≲ \sum_{t \in {i, j}} \exp ω_{0 t} {‖ (Δ_{𝒕_{1}} 𝝎_{1 t k}, Δ b_{t k}) ‖}^{\bar{r} (\| 𝔸_{k} \|)},$
	$(\exp ω_{0 i} + \exp ω_{0 j}) {‖ (Δ 𝒂_{* k}, Δ σ_{* k}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2}$	$≲ \sum_{t \in {i, j}} \exp ω_{0 t} {‖ (Δ 𝒂_{t k}, Δ σ_{t k}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2} .$

	$𝒟 (G_{N}, G_{0}, t_{0}, 𝒕_{1}) := \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) ({‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}, Δ b_{ℓ k}^{N}) ‖}^{{\bar{r}}_{k}} + {‖ (Δ 𝒂_{ℓ k}^{N}, Δ σ_{ℓ k}^{N}) ‖}^{{\bar{r}}_{k} / 2})$
	$+ \sum_{k : \| 𝔸_{k} \| = 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) ‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}, Δ 𝒂_{ℓ k}^{N}, Δ b_{ℓ k}^{N}, Δ σ_{ℓ k}^{N}) ‖ + \sum_{k = 1}^{K_{0}} \| \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) - \exp (ω_{0 k}^{0} + t_{0}) \|$
	$+ \sum_{k : \| 𝔸_{k} \| > 1} (∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ b_{ℓ k}^{N}) ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [{(Δ b_{ℓ k}^{N})}^{2} + (Δ σ_{ℓ k}^{N})] ∥$
	$+ ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) (Δ b_{ℓ k}^{N}) + (Δ 𝒂_{ℓ k}^{N})] ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{⊤} ∥),$

	$\| \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) - \exp (ω_{0 k}^{0} + t_{0}) \|$	$= \| \sum_{ℓ \in 𝔸_{k}, ℓ \notin {ℓ_{1}, ℓ_{2}}} \exp (ω_{0 ℓ}^{N}) + \exp (ω_{0, *}^{N}) - \exp (ω_{0 k}^{0} + t_{0}) \|,$
	$\exp (ω_{0, }^{N}) Δ b_{ k}^{N}$	$= \exp (ω_{0, }^{N}) (b_{}^{N} - b_{k}^{0})$
		$= \exp (ω_{0, }^{N}) (\exp (ω_{0 ℓ_{1}}^{N} - ω_{0, }^{N}) b_{ℓ_{1}}^{N} + \exp (ω_{0 ℓ_{2}}^{N} - ω_{0, *}^{N}) b_{ℓ_{2}}^{N} - b_{k}^{0})$
		$= \exp (ω_{0 ℓ_{1}}^{N}) b_{ℓ_{1}}^{N} + \exp (ω_{0 ℓ_{2}}^{N}) b_{ℓ_{2}}^{N} - \exp (ω_{0, *}^{N}) b_{k}^{0}$
		$= \exp (ω_{0 ℓ_{1}}^{N}) (b_{ℓ_{1}}^{N} - b_{k}^{0}) + \exp (ω_{0 ℓ_{2}}^{N}) (b_{ℓ_{2}}^{N} - b_{k}^{0})$
		$= \exp (ω_{0 ℓ_{1}}^{N}) Δ b_{ℓ_{1} k}^{N} + \exp (ω_{0 ℓ_{2}}^{N}) Δ b_{ℓ_{2} k}^{N} .$

	$D_{V} (G_{N}, G_{0}) := \sum_{k : \| 𝔸_{k} \| > 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) ({‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}, Δ b_{ℓ k}^{N}) ‖}^{\bar{r} (\| 𝔸_{k} \|)} + {‖ (Δ 𝒂_{ℓ k}^{N}, Δ σ_{ℓ k}^{N}) ‖}^{\bar{r} (\| 𝔸_{k} \|) / 2})$
	$+ \sum_{k : \| 𝔸_{k} \| = 1} \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) ‖ (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}, Δ 𝒂_{ℓ k}^{N}, Δ b_{ℓ k}^{N}, Δ σ_{ℓ k}^{N}) ‖ + \sum_{k = 1}^{K_{0}} \| \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) - \exp (ω_{0 k}^{0} + t_{0}) \|$
	$+ \sum_{k : \| 𝔸_{k} \| > 1} (∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ b_{ℓ k}^{N}) ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [{(Δ b_{ℓ k}^{N})}^{2} + (Δ σ_{ℓ k}^{N})] ∥$
	$+ ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) [(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) (Δ b_{ℓ k}^{N}) + (Δ 𝒂_{ℓ k}^{N})] ∥ + ∥ \sum_{ℓ \in 𝔸_{k}} \exp (ω_{0 ℓ}^{N}) (Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N}) {(Δ_{𝒕_{1}} 𝝎_{1 ℓ k}^{N})}^{⊤} ∥),$

Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps

Abstract

1 INTRODUCTION

2 PRELIMINARIES

Fact 1 (Nguyen et al., 2023a, Lemma 1).

Fact 2 (Nguyen et al., 2023a, Theorems 1 and 2).

3 FAST-RATE-AWARE EXPERT AGGREGATION IN SGMOE

3.1 Why Merge Experts? The Rate Gap

3.2 A Fast-Rate-Aware Voronoi Distance

3.3 A Merge Operator Tailored to SGMoE

3.4 The Hierarchical View of Aggregation Path

Lemma 1.

Theorem 1 (Fast convergence rates along the path).

3.5 Heights and Likelihood Along the Path

Theorem 2 (Height control).

Theorem 3 (Likelihood concentration on the path).

3.6 Choosing the Number of Experts via a Height-Likelihood Rule

Theorem 4 (Consistency of model selection).

3.7 Proof Sketches

4 SIMULATION STUDIES

5 REAL DATA APPLICATION

6 CONCLUSION

Acknowledgments

References

Appendix A ADDITIONAL DETAILS ON THE MAIZE DROUGHT-RESPONSE DATA

Biological motivation and data provenance.

Why this dataset is relevant here.

Raw variables used in the illustration.

Preprocessing strategy.

Model fitting for the SGMoE path.

Interpretation of the fitted path.

Why the real-data example is informative for our methodology.

Appendix B OVERVIEW OF MIXTURE AND MOE GEOMETRY

Difference from existing dendrogram approaches.

Compare to GGMoE.

Appendix C ILLUSTRATION OF VORONOI CELLS AND MERGE STEPS FOR SGMoE

Appendix D THEORETICAL CHALLENGES: MORE DETAILS

Appendix E PROOF SKETCHES

Why the DFRA loss in eq. 6?

E.1 Proof Sketch of Lemma 1

E.2 Proof Sketch of Theorem 1

(A) Inverse bound.

(B) Applying density rates.

E.3 Proof Sketch of Theorem 2

E.4 Proof Sketch of Theorem 3

Appendix F PROOF OF MAIN RESULTS

F.1 Proof of Lemma 1

F.2 Proof of Theorem 1

Fact 3 (Nguyen et al., 2023a, Proposition 1).

Fact 4 (Nguyen et al., 2023a, Proposition 2).

Theorem 5.

Proof of Theorem 5.

Fact 5 (Nguyen et al., 2023a, Lemma 2).

Proof of Theorem 1.

F.3 Proof of Theorem 2

F.4 Proof of Theorem 3

Preliminary on Empirical Process Theory.

Fact 6 (van de Geer, 2000, Theorem 5.11).

Connection between the Hellinger distance and the Wasserstein metric.

Fact 7 (Ho and Nguyen, 2016a, Lemma 1).

Proof of Theorem 3.

Checking condition K.

Lemma 2.

Proof of Lemma 2.

F.5 Proof of Theorem 4

Proof of Theorem 4.

Cite this paper

Related work

Why the $D_{FRA}$ loss in eq. 6?