Towards convergence rates for parameter estimation in Gaussian-gated mixture of experts

Huy Nguyen; TrungTin Nguyen; Khai Nguyen; Nhat Ho

Abstract

Originally introduced as a neural network for ensemble learning, mixture of experts (MoE) has recently become a fundamental building block of highly successful modern deep neural networks for heterogeneous data analysis in several applications of machine learning and statistics. Despite its popularity in practice, a satisfactory level of theoretical understanding of the MoE model is far from complete. To shed new light on this problem, we provide a convergence analysis for maximum likelihood estimation (MLE) in the Gaussian-gated MoE model. The main challenge of that analysis comes from the inclusion of covariates in the Gaussian gating functions and expert networks, which leads to their intrinsic interaction via some partial differential equations with respect to their parameters. We tackle these issues by designing novel Voronoi loss functions among parameters to accurately capture the heterogeneity of parameter estimation rates. Our findings reveal that the MLE has distinct behaviors under two complement settings of location parameters of the Gaussian gating functions, namely when all these parameters are non-zero versus when at least one among them vanishes. Notably, these behaviors can be characterized by the solvability of two different systems of polynomial equations. Finally, we conduct a simulation study to empirically verify our theoretical results.

1 INTRODUCTION

Mixture of experts (MoE) [19, 23] is a popular statistical machine learning model where experts are either regression functions or classifiers, while the input-dependent weights (also called gating functions) softly partition the input space into different regions and define which regions each expert is responsible for (see [56, 32, 8] for further details). In regression analysis with heterogeneous data, softmax-gated MoE [19, 23] and Gaussian-gated MoE (GMoE) [54] models are the most popular choices. One of the main drawbacks of the softmax-gated MoE models is the difficulty of applying an expectation-maximization (EM) algorithm [5], which requires an internal iterative numerical optimization procedure, e.g., Newton-Raphson algorithm, to update the softmax parameters in the maximization step. On the other hand, parameters of the GMoE models can be updated analytically, which helps reduce the computational complexity of the estimation routine. For those reasons, GMoE has become a fundamental component of modern deep neural networks in various fields, including speech recognition [12, 55], computer vision [29, 50], natural language processing [52, 9, 35, 7, 49], medical images [14], robot dynamics [51, 34], remote sensing [4, 25, 10, 11], and econometrics [48, 47, 6]. However, there is a paucity of work aiming at theoretically understanding the density estimation and parameter estimation in the GMoE models, which has remained poorly understood in the literature to the best of our knowledge.

Related literature. In the GMoE setting, early classical research focused on identifiability issues [22] and parameter estimation in the exact-fitted setting, assuming the true number of components $k_{0}$ is known [20]. For most applications, it is a too strong presumption as the true number of components is seldom known. To deal with this problem, there are three common practical approaches. The first approach is based on model selection, most importantly the Bayesian information criterion from asymptotic theory [11, 3, 24] and the slope heuristic [1, 2] in a non-asymptotic framework [45, 44, 46, 42]. In particular, the bias term can be substantially reduced with a sufficiently large model collection w.r.t. the number of mixture components $k$ by well-studied universal approximations theorems [41, 33, 21]. However, since we have to search for the optimal $k$ over all possible values, this approach is computationally expensive. The second approach is to design a tractable Bayesian nonparametric GMoE model. For example, [43] avoided any commitment to an arbitrary $k$ with posterior consistency guarantee thanks to the merge-truncate-merge post-processing in [13]. However, this approach still depends on a tuning parameter, which prevents the direct application of this approach to real data sets. The last approach is to use prior knowledge to over-specify the true model, i.e. specifying more mixture components than necessary, where most existing work is limited to its particular case, including mixture models [15, 16, 17, 13, 30] and mixture of experts [18, 40, 37, 38, 36]. It is worth noting that the convergence behavior of parameter estimations in the GMoE model has remained an open question, which we aim to answer in this paper. Before going into further details, we first formally introduce an affine instance of the GMoE model. This is a simplified but standard setting where we use linear functions for Gaussian mean experts.

GMoE setting. GMoE models are used to capture the non-linear and heterogeneous relationship between the response $Y \in 𝒴 \subset ℝ$ and the set of covariates $X \in 𝒳 \subset ℝ^{d}$ , $d \in ℕ$ . In the affine GMoE model, the response $Y$ is approximated by a $k_{0}$ local affine:

Y = \sum_{j = 1}^{k_{0}} 𝕀 ​ (Z = j) ​ [{(a_{j}^{0})}^{⊤} ​ X + b_{j}^{0} + e_{j}^{0}] .

(1)

Here $𝕀$ is an indicator function and $Z$ is a latent variable that captures a cluster relationship, such that $Z = j$ if $Y$ comes from cluster $j \in [k_{0}] := {1, 2, \dots, k_{0}}$ . Vectors $a_{j}^{0} \in ℝ^{d}$ and scalars $b_{j}^{0} \in ℝ$ define cluster-specific affine transformations. In addition, $e_{j}^{0}$ are error terms that capture both the reconstruction error (due to the local affine approximations) and the observation noise in $ℝ$ . Let $ℱ_{d} := {f (\cdot | ψ, Σ) : ψ \in ℝ^{d}, Σ \in 𝒮_{d}^{+}}$ be the family of $d$ -dimensional Gaussian density functions with mean $ψ$ and positive-definite covariance matrix $Σ$ , where $𝒮_{d}^{+}$ indicates the set of all symmetric positive-definite matrices on $ℝ^{d \times d}$ . Following the usual assumption that $e_{j}^{0}$ is a zero-mean Gaussian variable with variance $ν_{j}^{0} \in ℝ_{+}$ , it follows that

p ​ (Y | X, Z = j) = f_{𝒟} ​ (Y | {(a_{j}^{0})}^{⊤} ​ X + b_{j}^{0}, ν_{j}^{0}), f_{𝒟} \in ℱ_{1} .

To enforce the affine transformations to be local, $X$ is defined as a mixture of $k_{0}$ Gaussian components:

p ​ (X | Z = j) = f_{ℒ} ​ (X | c_{j}^{0}, Γ_{j}^{0}), p ​ (Z = j) = π_{j}^{0},

(2)

where $f_{ℒ} \in ℱ_{d}$ . Here, we refer to $f_{𝒟}$ and $f_{ℒ}$ as the data density and the local density, respectively. Additionally, $π_{j}^{0} > 0$ are called mixing proportions (or weights), satisfying $\sum_{j = 1}^{k_{0}} π_{j}^{0} = 1$ . Via the law of total probability, we obtain the GMoE model of order $k_{0}$ whose joint density function $p_{G_{0}} (X, Y)$ is given by:

\sum_{j = 1}^{k_{0}} π_{j}^{0} ​ f_{ℒ} ​ (X | c_{j}^{0}, Γ_{j}^{0}) \cdot f_{𝒟} ​ (Y | {(a_{j}^{0})}^{⊤} ​ X + b_{j}^{0}, ν_{j}^{0}) .

(3)

Here, $G_{0} := \sum_{j = 1}^{k_{0}} π_{j}^{0} δ_{(c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0})}$ denotes a true but unknown probability mixing measure, where $δ$ is the Dirac measure and for $j \in [k_{0}]$ , $(c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0}) \in Θ \subset ℝ^{d} \times 𝒮_{d}^{+} \times ℝ^{d} \times ℝ \times ℝ_{+}$ are called components of $G_{0}$ . We assume that ${(X_{i}, Y_{i})}_{i \in [n]}$ are i.i.d. samples of random variable $(X, Y)$ , coming from the GMoE model of order $k_{0}$ . To facilitate our theoretical guarantee, we assume that $Θ$ is compact and $𝒳$ is bounded.

Maximum likelihood estimation. We propose a general theoretical framework for analyzing the statistical performance of maximum likelihood estimation (MLE) for parameters under the setting of the GMoE model. Since the true order $k_{0}$ is generally unknown in practice, it is necessary to over-specify the number of components of mixing measures to at most $k$ , where $k > k_{0}$ . In particular, we consider

{\hat{G}}_{n} \in \underset{G \in 𝒪_{k} ​ (Θ)}{\arg ​ \max} ​ \sum_{i = 1}^{n} \log (p_{G} ​ (X_{i}, Y_{i})),

(4)

where $𝒪_{k} (Θ) := {G = \sum_{i = 1}^{k^{'}} π_{i} δ_{(c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i})} : 1 \leq k^{'} \leq k, \sum_{i = 1}^{k} π_{i} = 1, (c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i}) \in Θ}$ denotes the set of all mixing measures with at most $k$ components.

Theoretical challenges. For the purpose of deriving parameter estimation rates in the GMoE model, we first use the Taylor expansion to decompose the term $p_{{\hat{G}}_{n}} (X, Y) - p_{G_{0}} (X, Y)$ into a linear combination of elements which belong to a linearly independent set and associate with coefficients involving the discrepancies between parameter estimations and true parameters. By doing so, when the density estimation $p_{{\hat{G}}_{n}}$ converges to the true density $p_{G_{0}}$ , those parameter discrepancies also go to zero and we then obtain our desired parameter estimation rates. Nevertheless, the density decomposition is challenging due to a number of linearly dependent derivative terms in the Taylor expansion. In particular, we find out two interactions among the parameters of either function $f_{𝒟}$ or $f_{ℒ}$ via the following partial differential equations (PDEs):

\frac{\partial^{2} f_{𝒟}}{\partial b^{2}} = 2 ​ \frac{\partial f_{𝒟}}{\partial ν}, \frac{\partial^{2} f_{ℒ}}{\partial c ​ \partial c^{⊤}} = 2 ​ \frac{\partial f_{ℒ}}{\partial Γ} .

(5)

We refer to those interactions as interior interactions since each of them involves either parameters $b, ν$ of function $f_{𝒟}$ or parameters $c, Γ$ of function $f_{ℒ}$ . Furthermore, we also figure out an interaction between the parameters of functions $f_{𝒟}$ and $f_{ℒ}$ . More specifically, let us denote $F (X, Y | θ) := f_{ℒ} (X | c, Γ) f_{𝒟} (Y | a^{⊤} X + b, ν)$ where $θ := (c, Γ, a, b, ν)$ . Then, by taking the derivatives of $F$ with respect to its parameters as follows:

	$\frac{\partial^{2} F}{\partial c \partial b} (X, Y \| θ_{j}^{0})$	$= Γ^{- 1} (X - c_{j}^{0}) \cdot f_{ℒ} \cdot \frac{\partial f_{𝒟}}{\partial b};$
	$\frac{\partial F}{\partial a} (X, Y \| θ_{j}^{0})$	$= X \cdot f_{ℒ} \cdot \frac{\partial f_{𝒟}}{\partial b},$

it can be seen that the following PDE holds true when the location parameter of $f_{ℒ}$ vanishes, i.e. $c_{j}^{0} = 0$ :

\frac{\partial^{2} F}{\partial c ​ \partial b} ​ (X, Y | θ_{j}^{0}) = Γ^{- 1} \cdot \frac{\partial F}{\partial a} ​ (X, Y | θ_{j}^{0}) .

(6)

We refer to the interaction among parameters $c, b, a$ in equation (6) as the exterior interaction. Back to the density decomposition, it is necessary to aggregate linearly dependent derivative terms in equations (5) and (6) by taking the summation of their associated coefficients. As a result, we achieve our desired linear combination of linearly independent terms. However, the structure of associated coefficients in that combination becomes complex owing to the previous aggregation. Thus, when those coefficients converge to zero, we have to cope with two complex systems of polynomial equations given in equations (9) and (12).

Overall contributions. In this paper, we characterize the convergence behavior of maximum likelihood estimation in the GMoE model. Firstly, we demonstrate that the density estimation $p_{{\hat{G}}_{n}}$ converges to the true density $p_{G_{0}}$ under the Total Variation distance $V$ at the parametric rate $V (p_{{\hat{G}}_{n}}, p_{G_{0}}) = 𝒪 (n^{- 1 / 2})$ . Regarding the parameter estimation problem, given the above challenge discussion, we consider two complement settings of the location parameters $c_{1}^{0}, c_{2}^{0}, \dots, c_{k_{0}}^{0}$ based on the validity of the PDE in equation (6) as follows (see also Table 1):

1. Type I setting: all the values of $c_{1}^{0}, c_{2}^{0}, \dots, c_{k_{0}}^{0}$ are different from zero. Since the PDE (6) does not hold under this setting, we have to deal with only the interior interactions in equation (5). Thus, we propose a novel Voronoi loss function $\bar{D} (G, G_{0})$ defined in equation (10) to capture those interactions, and then establish the Total Variation lower bound $\bar{D} ({\hat{G}}_{n}, G_{0}) ≲ V (p_{{\hat{G}}_{n}}, p_{G_{0}}) = 𝒪 (n^{- 1 / 2})$ . This result together with the formulation of $\bar{D} ({\hat{G}}_{n}, G_{0})$ indicate that exact-fitted parameters $c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0}$ , which are approximated by exactly one component, share the same estimation rate of order $𝒪 (n^{- 1 / 2})$ . By contrast, the rates for estimating over-fitted parameters $c_{j}^{0}, Γ_{j}^{0}, b_{j}^{0}, ν_{j}^{0}$ , which are fitted by at least two components, depend on the solvability of the system of polynomial equations (9) and become no faster than $𝒪 (n^{- 1 / 4})$ . These slow rates are due to the interior interactions among those parameters in equation (5). As over-fitted parameters $a_{j}^{0}$ are not involved in those interactions, their estimation rates keep unchanged of order $𝒪 (n^{- 1 / 4})$ .

2. Type II setting: at least one among the values of $c_{1}^{0}, c_{2}^{0}, \dots, c_{k_{0}}^{0}$ is equal to zero. Without loss of generality, we assume that $c_{1}^{0}, c_{2}^{0}, \dots, c_{\tilde{k}}^{0}$ equal zero, where $1 \leq \tilde{k} \leq k_{0}$ , while other $c_{j}$ ’s are non-zero. Since the PDE (6) holds true under this setting, we have to confront both interior and exterior interactions among parameters. For that purpose, we construct another novel Voronoi loss function $\tilde{D} (G, G_{0})$ in equation (3.2) to handle those interactions, and then derive the Total Variation lower bound $\tilde{D} ({\hat{G}}_{n}, G_{0}) ≲ V (p_{{\hat{G}}_{n}}, p_{G_{0}}) = 𝒪 (n^{- 1 / 2})$ . Due to the occurrence of both interior and exterior interactions, the rates for estimating over-fitted parameters $c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0}$ are now determined by the solvability of both systems of polynomial equations (9) and (12). Meanwhile, the estimation rates for their exact-fitted counterparts remain the same of order $𝒪 (n^{- 1 / 2})$ .

Setting	Exact-fitted $c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0}$	Over-fitted $a_{j}^{0}$		Over-fitted $c_{j}^{0}, b_{j}^{0}$		Over-fitted $Γ_{j}^{0}, ν_{j}^{0}$
Setting		$j \in [\tilde{k}]$	$j \in [k_{0}] ∖ [\tilde{k}]$	$j \in [\tilde{k}]$	$j \in [k_{0}] ∖ [\tilde{k}]$	$j \in [\tilde{k}]$	$j \in [k_{0}] ∖ [\tilde{k}]$
Type I	$𝒪 (n^{- 1 / 2})$	$𝒪 (n^{- 1 / 4})$		$𝒪 (n^{- 1 / 2 {\bar{r}}_{j}})$		$𝒪 (n^{- 1 / {\bar{r}}_{j}})$
Type II	$𝒪 (n^{- 1 / 2})$	$𝒪 (n^{- 1 / {\tilde{r}}_{j}})$	$𝒪 (n^{- 1 / 4})$	$𝒪 (n^{- 1 / 2 {\tilde{r}}_{j}})$	$𝒪 (n^{- 1 / 2 {\bar{r}}_{j}})$	$𝒪 (n^{- 1 / {\tilde{r}}_{j}})$	$𝒪 (n^{- 1 / {\bar{r}}_{j}})$

Table 1: Summary of parameter estimation rates in the GMoE model under the Type I and Type II settings. Recall that the cardinality of Voronoi cells

𝒜_{j}

(see Section 2) generated by true components

(c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0})

indicates the number of components fitting them. When

| 𝒜_{j} | = 1

, we call them exact-fitted parameters, but when

| 𝒜_{j} | > 1

, they are referred to as over-fitted parameters. Additionally, the notations

{\bar{r}}_{j} := \bar{r} ​ (| 𝒜_{j} |)

and

{\tilde{r}}_{j} := \tilde{r} ​ (| 𝒜_{j} |)

stand for the solvability of two polynomial equation systems (9) and (12), respectively. For example, if

| 𝒜_{j} | = 2

, then we have

{\bar{r}}_{j} = {\tilde{r}}_{j} = 4

. Meanwhile, we get

{\bar{r}}_{j} = {\tilde{r}}_{j} = 6

if

| 𝒜_{j} | = 3

.

Practical implication. In practice, the parameters specific to each mixing component may carry useful information about the heterogeneity of the underlying (latent) subpopulations. Since in reality there is a tendency to “over-fit” the mixture generously by adding many more mixing components, our theory warns against this because, as we have shown, the convergence rate via standard methods such as MLE for subpopulation-specific parameters deteriorates rapidly with the number of redundant components. Hopefully, the theoretical results will suggest practical ways to identify benign scenarios and impose helpful constraints when GMoE models have favourable convergence rates, and detect pathological scenarios that practitioners would do well to avoid. In particular, practitioners can consistently estimate the true number of components based on our important threshold on the convergence rates of the MLE using the merge-truncate-merge procedure [13] or Group-Sort-Fuse [31].

Paper organization.

The rest of this paper proceeds as follows. In Section 2, we begin with providing some background on the identifiability of the GMoE model and the rate for estimating the joint density function under that model. Next, in Section 3, we establish the convergence rates of parameter estimation under both Type I and Type II settings, which are then empirically verified by simulation studies in Section 4. Finally, we conclude the paper in Section 5 and defer proofs of all theoretical results to the supplementary material.

Notation.

Throughout the paper, ${1, 2, \dots, n}$ is abbreviated as $[n]$ for any $n \in ℕ$ . Given any two sequences of positive real numbers ${a_{n}}_{n = 1}^{\infty}$ and ${b_{n}}_{n = 1}^{\infty}$ , we write $a_{n} = 𝒪 (b_{n})$ or $a_{n} ≲ b_{n}$ to indicate that there exists a constant $C > 0$ such that $a_{n} \leq C b_{n}$ for all $n \geq 1$ . Next, for any vector $v \in ℝ^{d}$ , we denote $| v | := v_{1} + v_{2} + \dots + v_{d}$ , whereas ${‖ v ‖}_{p}$ stands for its $p$ -norm with a note that $‖ v ‖$ implicitly indicates the $2$ -norm unless stating otherwise. By abuse of notation, we also denote by $‖ A ‖$ the Frobenius norm of any matrix $A \in ℝ^{d \times d}$ . Additionally, the notation $| S |$ represents for the cardinality of any set $S$ . Finally, given two probability density functions $p, q$ with respect to the Lebesgue measure $μ$ , we define $V (p, q) := \frac{1}{2} \int | p - q | 𝑑 μ$ as their Total Variation distance, while $h^{2} (p, q) := \frac{1}{2} \int {(\sqrt{p} - \sqrt{q})}^{2} 𝑑 μ$ denotes the squared Hellinger distance between them.

2 PRELIMINARIES

In this section, we first verify the identifiability of the GMoE model, and then establish the density estimation rate under that model. Lastly, we introduce a notion of a Voronoi cells, which will be used to build Voronoi loss functions in Section 3.

Firstly, we demonstrate in the following proposition that the GMoE model is identifiable:

Proposition 1 (Identifiability of the GMoE model).

Let $G$ and $G^{'}$ be two mixing measures in $𝒪_{k} (Θ)$ . If the equation $p_{G} (X, Y) = p_{G^{'}} (X, Y)$ holds true for almost surely $(X, Y) \in 𝒳 \times 𝒴$ , then we obtain that $G \equiv G^{'}$ .

The proof of Proposition 1 is deferred to Appendix C.1. Given the above result, we know that two mixing measures $G$ and $G^{'}$ are equivalent if and only if they share the same joint density function.

Next, we characterize the convergence rate of the joint density estimation $p_{{\hat{G}}_{n}}$ to its true counterpart $p_{G_{0}}$ under the Total Variation distance.

Proposition 2 (Joint Density Estimation Rate).

With the MLE ${\hat{G}}_{n}$ defined in equation (4), the following bound indicates that the density estimation $p_{{\hat{G}}_{n}}$ converges to the true density $p_{G_{0}}$ under the Total Variation distance at the parametric rate of order $𝒪 (n^{- 1 / 2})$ (up to a logarithmic term):

ℙ ​ (V ​ (p_{{\hat{G}}_{n}}, p_{G_{0}}) > C_{1} ​ \sqrt{\log (n) / n}) ≲ n^{- C_{2}},

where $C_{1}$ and $C_{2}$ are universal constants.

The proof of Proposition 2 can be found in Appendix C.2. This result is a key ingredient to study the parameter estimation problem in the GMoE model in subsequent sections. In particular, if we are able to establish the lower bound of the Total Variation distance in terms of some loss function $D$ between two mixing measures, i.e., $V (p_{G}, p_{G_{0}}) ≳ D (G, G_{0})$ for any mixing measure $G \in 𝒪_{k} (Θ)$ , then the MLE ${\hat{G}}_{n}$ also converges to the true mixing measure $G_{0}$ at the parametric rate of $𝒪 (n^{- 1 / 2})$ . Based on this result, we then achieve the parameter estimation rates through the formulation of the loss function $D ({\hat{G}}_{n}, G_{0})$ . For that purpose, let us introduce a notion of Voronoi cells which are essential to construct Voronoi loss functions later in Section 3.

Voronoi cells. In general, true parameters which are fitted by exactly one component should enjoy faster estimation rates than those approximated by more than one component. Therefore, in order to capture the convergence behavior of parameter estimations accurately, we define $k_{0}$ different index sets called Voronoi cells to control the number of fitted components approaching each of the $k_{0}$ true components. More formally, for any $G \in 𝒪_{k} (Θ)$ , the Voronoi cell $𝒜_{j} := 𝒜_{j} (G)$ generated by $θ_{j}^{0} := (c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0})$ is defined as

𝒜_{j}

:= {i \in [k] : ‖ θ_{i} - θ_{j}^{0} ‖ \leq ‖ θ_{i} - θ_{ℓ}^{0} ‖, \forall ℓ \neq j},

(7)

for any $j \in [k_{0}]$ , where $θ_{i} := (c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i})$ . An illustration of Voronoi cells is given in Appendix A. Notably, the cardinality of each Voronoi cell $𝒜_{j}$ is exactly the number of fitted components approximating the true component $θ_{j}^{0}$ .

3 PARAMETER ESTIMATION RATES

In this section, we conduct the convergence analysis for parameter estimation in the GMoE model under the Type I and Type II settings in Section 3.1 and Section 3.2, respectively. Then, we sketch the proof for main results in both settings in Section 3.3.

3.1 Type I Setting

Let us recall that under this setting, all the values of $c_{1}^{0}, c_{2}^{0}, \dots, c_{k_{0}}^{0}$ are non-zero. Although the exterior interaction between the parameters of two functions $f_{ℒ}$ and $f_{𝒟}$ mentioned in equation (6) does not hold in this scenario, we encounter two interior interactions among parameters $b, ν$ and $c, Γ$ via the following PDEs:

\frac{\partial^{2} f_{𝒟}}{\partial b^{2}} = 2 ​ \frac{\partial f_{𝒟}}{\partial ν}, \frac{\partial^{2} f_{ℒ}}{\partial c ​ \partial c^{⊤}} = 2 ​ \frac{\partial f_{ℒ}}{\partial Γ} .

(8)

System of polynomial equations. To precisely characterize the estimation rates for those parameters, we need to consider the solvability of a system of polynomial equations which was previously studied in [15]. In particular, for each $m \geq 2$ , let $\bar{r} (m)$ be the smallest positive integer $r$ such that the system:

\sum_{l = 1}^{m} \sum_{\begin{matrix} n_{1}, n_{2} \in ℕ : \\ n_{1} + 2 ​ n_{2} = s \end{matrix}} \frac{p_{l}^{2} ​ q_{1 ​ l}^{n_{1}} ​ q_{2 ​ l}^{n_{2}}}{n_{1}! ​ n_{2}!} = 0, s = 1, 2, \dots, r,

(9)

does not have any non-trivial solutions for the unknown variables ${p_{l}, q_{1 l}, q_{2 l}}_{l = 1}^{m}$ . Here, a solution is called non-trivial if all the values of $p_{l}$ are different from zero, whereas at least one among $q_{1 l}$ is non-zero. The following lemma gives us the values of $\bar{r} (m)$ at some specific points $m$ .

Lemma 1 (Proposition 2.1, [15]).

When $m = 2$ , we have that $\bar{r} (m) = 4$ , while for $m = 3$ , we get $\bar{r} (m) = 6$ . If $m \geq 4$ , then $\bar{r} (m) \geq 7$ .

Proof of Lemma 1 is in [15]. Now, we are ready to introduce a Voronoi loss function used for this setting.

Voronoi loss function. For simplicity, we denote $Δ c_{i j} := c_{i} - c_{j}^{0}$ , $Δ Γ_{i j} := Γ_{i} - Γ_{j}^{0}$ , $Δ a_{i j} := a_{i} - a_{j}^{0}$ , $Δ b_{i j} := b_{i} - b_{j}^{0}$ , $Δ ν_{i j} := ν_{i} - ν_{j}^{0}$ and ${\bar{r}}_{j} := \bar{r} (| 𝒜_{j} |)$ . Additionally, we also define mappings $K_{i j} : ℕ^{5} \to ℝ$ such that $K_{i j} (κ_{1}, κ_{2}, κ_{3}, κ_{4}, κ_{5}) := {‖ Δ c_{i j} ‖}^{κ_{1}} + {‖ Δ Γ_{i j} ‖}^{κ_{2}} + {‖ Δ a_{i j} ‖}^{κ_{3}} + {| Δ b_{i j} |}^{κ_{4}} + {| Δ ν_{i j} |}^{κ_{5}}$ , for any $j \in [k_{0}]$ and $i \in 𝒜_{j}$ . Then, the Voronoi loss function $\bar{D} (G, G_{0})$ of interest in this setting is given by:

		$\bar{D} (G, G_{0}) := \sum_{\begin{matrix} j : \| 𝒜_{j} \| > 1, \\ i \in 𝒜_{j} \end{matrix}} π_{i} K_{i j} ({\bar{r}}_{j}, \frac{{\bar{r}}_{j}}{2}, 2, {\bar{r}}_{j}, \frac{{\bar{r}}_{j}}{2}) +$
		$\sum_{\begin{matrix} j : \| 𝒜_{j} \| = 1, \\ i \in 𝒜_{j} \end{matrix}} π_{i} K_{i j} (1, 1, 1, 1, 1) + \sum_{j = 1}^{k_{0}} \| \sum_{i \in 𝒜_{j}} π_{i} - π_{j}^{0} \| .$		(10)

Given this loss function, we capture parameter estimation rates in the GMoE model in the following theorem.

Theorem 1.

Under the Type I setting, the Total Variation lower bound $V (p_{G}, p_{G_{0}}) ≳ \bar{D} (G, G_{0})$ holds for any $G \in 𝒪_{k} (Θ)$ , which implies that there exists a universal constant $C_{3} > 0$ depending on $G_{0}$ and $Θ$ satisfying

ℙ ​ (\bar{D} ​ ({\hat{G}}_{n}, G_{0}) > C_{3} ​ \sqrt{\log (n) / n}) ≲ n^{- C_{4}},

where $C_{4} > 0$ is a constant that depends only on $Θ$ .

Proof of Theorem 1 is in Appendix B.1. It follows from Theorem 1 that the discrepancy $\bar{D} ({\hat{G}}_{n}, G_{0})$ vanishes at a rate of order $𝒪 (n^{- 1 / 2})$ up to a logarithmic constant, which leads to following observations: (i) True parameters $c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0}$ , which are fitted by exactly one component, share the same estimation rate of order $𝒪 (n^{- 1 / 2})$ ; (ii) By contrast, the rates for estimating parameters fitted by more than one element are significantly slower. In particular, the estimation rates for $c_{j}^{0}, b_{j}^{0}$ are of order $𝒪 (n^{- 1 / 2 \bar{r} (| 𝒜_{j}^{n} |)})$ , whereas those for $Γ_{j}^{0}, ν_{j}^{0}$ are of order $𝒪 (n^{- 1 / \bar{r} (| 𝒜_{j}^{n} |)})$ in which $𝒜_{j}^{n} := 𝒜_{j} ({\hat{G}}_{n})$ . For instance, if we have $| 𝒜_{j}^{n} | = 3$ , then Lemma 1 indicates that the previous two rates become $𝒪 (n^{- 1 / 12})$ and $𝒪 (n^{- 1 / 6})$ , respectively. These slow rates are owing to the interior interactions among those parameters in equation (8). Meanwhile, $a_{j}^{0}$ admits a much faster rate of order $𝒪 (n^{- 1 / 4})$ as it does not interact with other parameters.

3.2 Type II Setting

Next, we consider the Type II setting, namely when at least one among $c_{1}^{0}, c_{2}^{0}, \dots, c_{k_{0}}^{0}$ is equal to vector $𝟎_{d}$ . Without loss of generality, we assume that $c_{1}^{0} = c_{2}^{0} = \dots = c_{\tilde{k}}^{0} = 𝟎_{d}$ , while $c_{\tilde{k} + 1}^{0}, c_{\tilde{k} + 2}^{0}, \dots, c_{k_{0}}^{0}$ are different from $𝟎_{d}$ . Under this setting, we encounter not only the two interior interactions in equation (8) but also the exterior interaction expressed by the following PDE:

\frac{\partial^{2} F}{\partial c ​ \partial b} ​ (X, Y | θ_{j}^{0}) = Γ^{- 1} \cdot \frac{\partial F}{\partial a} ​ (X, Y | θ_{j}^{0}),

(11)

where $F (X, Y | θ) := f_{ℒ} (X | c, Γ) f_{𝒟} (Y | a^{⊤} X + b, ν)$ and $θ := (c, Γ, a, b, ν)$ . This phenomenon poses a lot of challenges in the parameter estimation problem. Therefore, we will only present the results when $d = 1$ for simplicity, while those for the setting $d > 1$ can be argued in a similar fashion but with more complex notations.

System of polynomial equations. Due to the emergence of the exterior interaction, we need to control the solvability of a totally new system of polynomial equations, which is given by

\sum_{l = 1}^{m} \sum_{α \in 𝒥_{ℓ_{1}, ℓ_{2}}} \frac{p_{l}^{2} ​ q_{1 ​ l}^{α_{1}} ​ q_{2 ​ l}^{α_{2}} ​ q_{3 ​ l}^{α_{3}} ​ q_{4 ​ l}^{α_{4}} ​ q_{5 ​ l}^{α_{5}}}{α_{1}! ​ α_{2}! ​ α_{3}! ​ α_{4}! ​ α_{5}!} = 0,

(12)

for all $ℓ_{1}, ℓ_{2} \geq 0$ satisfying $1 \leq ℓ_{1} + ℓ_{2} \leq r$ , where $𝒥_{ℓ_{1}, ℓ_{2}} := {α = {(α_{l})}_{l = 1}^{5} \in ℕ^{5} : α_{1} + 2 α_{2} + α_{3} = ℓ_{1}, α_{3} + α_{4} + 2 α_{5} = ℓ_{2}}$ . Now, we define $\tilde{r} (m)$ as the smallest natural number $r$ such that the system in equation (12) does not have any non-trivial solutions for the unknown variables ${p_{l}, q_{1 l}, q_{2 l}, q_{3 l}, q_{4 l}, q_{5 l}}_{l = 1}^{m}$ , namely, all of $p_{l}$ are non-zero, whereas at least one among $q_{4 l}$ is different from zero. The following lemma establishes a connection between $\tilde{r} (m)$ and $\bar{r} (m)$ as well as provides the values of $\tilde{r} (m)$ given some specific choices of $m$ .

Lemma 2.

In general, we have $\tilde{r} (m) \leq \bar{r} (m)$ for all $m \in ℕ$ . Furthermore, the equality occurs when $m = 2$ and $m = 3$ , meaning that $\tilde{r} (2) = 4$ and $\tilde{r} (3) = 6$ .

Proof of Lemma 2 is in Appendix C.3. Next, we introduce a Voronoi loss function tailored to this setting.

Voronoi loss function. Firstly, let us reformulate the mappings $K_{i j}$ defined in Section 3.1 for $d = 1$ as $K_{i j} (κ_{1}, κ_{2}, κ_{3}, κ_{4}, κ_{5}) := {| Δ c_{i j} |}^{κ_{1}} + {| Δ Γ_{i j} |}^{κ_{2}} + {| Δ a_{i j} |}^{κ_{3}} + {| Δ b_{i j} |}^{κ_{4}} + {| Δ ν_{i j} |}^{κ_{5}}$ . In addition, we denote ${\tilde{r}}_{j} := \tilde{r} (| 𝒜_{j} |)$ and ${\bar{r}}_{j} := \bar{r} (| 𝒜_{j} |)$ , for any $j \in [k_{0}]$ . Then, the Voronoi loss of interest $\tilde{D} (G, G_{0})$ is defined as follows:

		$\tilde{D} (G, G_{0}) := \sum_{\begin{matrix} j \in [\tilde{k}] : \| 𝒜_{j} \| > 1, \\ i \in 𝒜_{j} \end{matrix}} π_{i} K_{i j} ({\tilde{r}}_{j}, \frac{{\tilde{r}}_{j}}{2}, \frac{{\tilde{r}}_{j}}{2}, {\tilde{r}}_{j}, \frac{{\tilde{r}}_{j}}{2})$
		$+ \sum_{\begin{matrix} j \in [k_{0}] ∖ [\tilde{k}] : \| 𝒜_{j} \| > 1, \\ i \in 𝒜_{j} \end{matrix}} π_{i} K_{i j} ({\bar{r}}_{j}, \frac{{\bar{r}}_{j}}{2}, 2, {\bar{r}}_{j}, \frac{{\bar{r}}_{j}}{2})$
		$+ \sum_{\begin{matrix} j \in [k_{0}] : \| 𝒜_{j} \| = 1, \\ i \in 𝒜_{j} \end{matrix}} π_{i} K_{i j} (1, 1, 1, 1, 1) + \sum_{j = 1}^{k_{0}} \| \sum_{i \in 𝒜_{j}} π_{i} - π_{j}^{0} \| .$		(13)

Given the above loss function, we derive the rates for estimating parameters under the Type II setting in the following theorem.

Theorem 2.

Under the Type II setting, the Total Variation lower bound $V (p_{G}, p_{G_{0}}) ≳ \tilde{D} (G, G_{0})$ holds for any $G \in 𝒪_{k} (Θ)$ , which indicates that we can find a constant $C_{5} > 0$ depending on $G_{0}, Θ$ such that

ℙ ​ (\tilde{D} ​ ({\hat{G}}_{n}, G_{0}) > C_{5} ​ \sqrt{\log (n) / n}) ≲ n^{- C_{6}},

where $C_{6} > 0$ is a constant that depends only on $Θ$ .

Proof of Theorem 2 is in Appendix B.2. Similar to Theorem 1, the Voronoi loss $\tilde{D} ({\hat{G}}_{n}, G_{0})$ also converges to zero at a rate of order $𝒪 (n^{- 1 / 2})$ (up to a logarithmic term) under the Type II setting. Moreover, true parameters $c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0}$ enjoy the same estimation rates as their counterparts in Section 3.1 for any $j \in [k_{0}] : | 𝒜_{j}^{n} | = 1$ and $j \in [k_{0}] ∖ [\tilde{k}] : | 𝒜_{j}^{n} | > 1$ . However, the difference in the convergence behavior occurs when $j \in [\tilde{k}] : | 𝒜_{j}^{n} | > 1$ . In particular, the rates for estimating parameters $a_{j}^{0}$ now drop substantially to $𝒪 (n^{- 1 / \tilde{r} (| 𝒜_{j} |)})$ in comparison with $𝒪 (n^{- 1 / 4})$ under the Type I settings. This phenomenon happens due to the interaction of $a_{j}^{0}$ with parameters $c_{j}^{0}, b_{j}^{0}$ via the PDE in equation (11).

3.3 Proof Sketch

Since arguments used for the proof of Theorem 1 are included in that of Theorem 2, we will present the former proof sketch implicitly inside the latter. In particular, we focus on establishing the bound ${inf}_{G \in 𝒪_{k} (Θ)} V (p_{G}, p_{G_{0}}) / \tilde{D} (G, G_{0}) > 0$ under the Type II setting when $d = 1$ . For that purpose, we will respectively demonstrate its local and global versions by contradiction as follows:

Local bound: We wish to prove that

lim_{ε > 0} inf_{G \in 𝒪_{k} ​ (Θ), \tilde{D} ​ (G, G_{0}) \leq ε} V ​ (p_{G}, p_{G_{0}}) / \tilde{D} ​ (G, G_{0}) > 0 .

Assume that this bound does not hold, then we can find a sequence $G_{n} = \sum_{i = 1}^{k_{n}} π_{i}^{n} δ_{θ_{i}^{n}} \in 𝒪_{k} (Θ)$ , where $θ_{i}^{n} := (c_{i}^{n}, Γ_{i}^{n}, a_{i}^{n}, b_{i}^{n}, ν_{i}^{n})$ , such that $V (p_{G_{n}}, p_{G_{0}}) / \tilde{D} (G_{n}, G_{0})$ and $\tilde{D} (G_{n}, G_{0})$ both vanish as $n \to \infty$ . Now, we decompose $Ξ_{n} := p_{G_{n}} (X, Y) - p_{G_{0}} (X, Y)$ as

	$Ξ_{n}$	$= \sum_{j = 1}^{k_{0}} \sum_{i \in 𝒜_{j}} π_{i}^{n} [F (X, Y \| θ_{i}^{n}) - F (X, Y \| θ_{j}^{0})]$
		$+ \sum_{j = 1}^{k_{0}} (\sum_{i \in 𝒜_{j}} π_{i}^{n} - π_{j}^{0}) F (X, Y \| θ_{j}^{0}),$

where $θ_{j}^{0} := (c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0})$ . Let us denote $h_{1} (X, a, b) = a^{⊤} X + b$ for any $a \in ℝ^{d}, b \in ℝ$ . Then, for $i \in 𝒜_{j}$ and $i^{'} \in 𝒜_{j^{'}}$ where $j \in [\tilde{k}]$ and $j^{'} \in [k_{0}] ∖ [\tilde{k}]$ , we invoke the Taylor expansion up to some orders $r_{1 j}$ and $r_{2 j^{'}}$ (we will choose later) for $F (X, Y | θ_{i}^{n})$ and $F (X, Y | θ_{i^{'}}^{n})$ , respectively, as follows:

	$F (X, Y \| θ_{i}^{n}) - F (X, Y \| θ_{j}^{0})$
	$= \sum_{ℓ_{1} + ℓ_{2} = 1}^{2 r_{1 j}} Q_{ℓ_{1}, ℓ_{2}}^{n} (j) \cdot X^{ℓ_{1}} f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0})$
	$\times \frac{\partial^{ℓ_{2}} f_{𝒟}}{\partial h_{1}^{ℓ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) + R_{1 i j} (X, Y),$
	$F (X, Y \| θ_{i^{'}}^{n}) - F (X, Y \| θ_{j^{'}}^{0})$
	$= R_{2 i^{'} j^{'}} (X, Y) + \sum_{α_{3} = 0}^{r_{2 j^{'}}} \sum_{τ_{1} + τ_{2} = 0}^{2 (r_{2 j^{'}} - α_{3})} T_{α_{3}, τ_{1}, τ_{2}}^{n} (j^{'}) \cdot X^{α_{3}}$
	$\times \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j^{'}}^{0}, Γ_{j^{'}}^{0}) \frac{\partial^{α_{3} + τ_{2}} f_{𝒟}}{\partial h_{1}^{α_{3} + τ_{2}}} (Y \| a_{j^{'}}^{0} X + b_{j^{'}}^{0}, ν_{j^{'}}^{0}) .$

Here $R_{1 i j} (X, Y)$ and $R_{2 i^{'} j^{'}} (X, Y)$ are Taylor remainders such that their ratios to $\tilde{D} (G_{n}, G_{0})$ vanishes as $n \to \infty$ . Thus, we can treat $Ξ_{n} / \tilde{D} (G_{n}, G_{0})$ as a linear combination of linearly independent terms

X^{ℓ_{1}} \cdot f_{ℒ} ​ (X | c_{j}^{0}, Γ_{j}^{0}) \cdot \frac{\partial^{ℓ_{2}} f_{𝒟}}{\partial h_{1}^{ℓ_{2}}} ​ (Y | a_{j}^{0} ​ X + b_{j}^{0}, ν_{j}^{0}),

X^{α_{3}} \cdot \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} ​ (X | c_{j^{'}}^{0}, Γ_{j^{'}}^{0}) \cdot \frac{\partial^{α_{3} + τ_{2}} f_{𝒟}}{\partial h_{1}^{α_{3} + τ_{2}}} ​ (Y | a_{j^{'}}^{0} ​ X + b_{j^{'}}^{0}, ν_{j^{'}}^{0})

associated with coefficients $Q_{ℓ_{1}, ℓ_{2}}^{n} (j)$ and $T_{α_{3}, τ_{1}, τ_{2}}^{n} (j^{'})$ , respectively. Moreover, it follows from Fatou’s lemma that $Ξ_{n} / \tilde{D} (G_{n}, G_{0})$ approaches zero when $n \to \infty$ . Consequently, all the coefficients in the representation of $Ξ_{n} / \tilde{D} (G_{n}, G_{0})$ , i.e. $Q_{ℓ_{1}, ℓ_{2}}^{n} (j) / \tilde{D} (G_{n}, G_{0})$ and $T_{α_{3}, τ_{1}, τ_{2}}^{n} (j^{'}) / \tilde{D} (G_{n}, G_{0})$ , go to zero as $n \to \infty$ . Therefore, in order to point out a contradiction, we need to choose the values of $r_{1 j}$ and $r_{2 j^{'}}$ such that at least one among these coefficients does not vanish. As a result, we achieve the aforementioned local bound. Now, we will show how to determine such values of $r_{1 j}$ and $r_{2 j^{'}}$ . It is worth noting that if we set $\tilde{k} = 0$ , then Type II settings reduces to Type I settings and we only need to deal with $r_{2 j^{'}}$ as follows:

Type I setting: We will specify an appropriate of $r_{2 j^{'}}$ during proving by contradiction that not all the coefficients $T_{α_{3}, τ_{1}, τ_{2}}^{n} (j^{'}) / \tilde{D} (G_{n}, G_{0})$ tend to zero. Assume that these coefficients all vanish, then we extract some useful limits among them for our arguments and end up with the following system of polynomial equations:

\sum_{i^{'} \in 𝒜_{j^{'}}} \sum_{n_{1} + 2 ​ n_{2} = s} \frac{p_{l}^{2} ​ q_{1 ​ l}^{n_{1}} ​ q_{2 ​ l}^{n_{2}}}{n_{1}! ​ n_{2}!} = 0, s = 1, 2, \dots, r_{2 ​ j^{'}} .

By construction, this system must have at least one non-trivial solution. Thus, to contradict this condition, we set $r_{2 j^{'}} = \bar{r} (| 𝒜_{j^{'}} |)$ , which makes the above system has no non-trivial solutions.

Type II setting: When $\tilde{k} > 0$ , i.e. there exist some zero-valued parameter $c_{j}$ , we will keep $r_{2 j^{'}} = \bar{r} (| 𝒜_{j^{'}} |)$ for all $j^{'} \in [k_{0}] ∖ [\tilde{k}]$ and find the desired values of $r_{1 j}$ for $j \in [\tilde{k}]$ by showing by contradiction that not all the coefficients $Q_{ℓ_{1}, ℓ_{2}}^{n} (j) / \tilde{D} (G_{n}, G_{0})$ go to zero. En route to pointing out a contradiction to the hypothesis, we come across a more complex system of polynomial equations than its counterpart in the previous setting, specifically

\sum_{i \in 𝒜_{j}} \sum_{α \in 𝒥_{ℓ_{1}, ℓ_{2}}} \frac{p_{i}^{2} ​ q_{1 ​ i}^{α_{1}} ​ q_{2 ​ i}^{α_{2}} ​ q_{3 ​ i}^{α_{3}} ​ q_{4 ​ i}^{α_{4}} ​ q_{5 ​ i}^{α_{5}}}{α_{1}! ​ α_{2}! ​ α_{3}! ​ α_{4}! ​ α_{5}!} = 0,

for all $ℓ_{1}, ℓ_{2} \geq 0$ such that $1 \leq ℓ_{1} + ℓ_{2} \leq r_{1 j}$ , where $𝒥_{ℓ_{1}, ℓ_{2}} := {α = {(α_{i})}_{i = 1}^{5} \in ℕ^{5} : α_{1} + 2 α_{2} + α_{3} = ℓ_{1}, α_{3} + α_{4} + 2 α_{5} = ℓ_{2}}$ . Since this system necessarily has a non-trivial solution, we choose $r_{1 j} = \tilde{r} (| 𝒜_{j} |)$ so that it admits only trivial solutions, which contradicts the previous claim. Consequently, we can find a constant $ε^{'} > 0$ such that

inf_{G \in 𝒪_{k} ​ (Θ), \tilde{D} ​ (G, G_{0}) \leq ε^{'}} V ​ (p_{G}, p_{G_{0}}) / \tilde{D} ​ (G, G_{0}) > 0 .

Global bound: Thus, to complete the proof, it is sufficient to demonstrate the global bound

inf_{G \in 𝒪_{k} ​ (Θ), \tilde{D} ​ (G, G_{0}) > ε^{'}} V ​ (p_{G}, p_{G_{0}}) / \tilde{D} ​ (G, G_{0}) > 0 .

If this bound did not hold, there would be a mixing measure $G^{'} \in 𝒪_{k} (Θ)$ that satisfies $p_{G^{'}} (X, Y) = p_{G_{0}} (X, Y)$ for almost surely $(X, Y)$ , which leads to $G^{'} \equiv G_{0}$ by Proposition 1. As a result, we obtain $\tilde{D} (G^{'}, G_{0}) = 0$ , which contradicts the constraint that $\tilde{D} (G^{'}, G_{0}) > ε^{'}$ . Hence, the proof sketch is completed.

4 EXPERIMENTS

In this section, we empirically validate the convergence rates of parameter estimation in four GMoE models which satisfy the assumptions of Type I and Type II settings, respectively, when $k_{0} = 3$ . Note that for simplicity, we only perform a simulation study to illustrate the convergence rates of Theorems 1 and 2 for the GMoE model when $X$ lies in one- and two-dimensional space with unknown location and scale parameters. All code to reproduce our simulation study is publicly available¹¹1https://github.com/Trung-TinNGUYEN/CRPE-GMoE and all simulations below were performed in Python 3.9.13 on a standard Unix machine.

Numerical schemes. In Model I, we set $G_{0}$ as follows:

	$\sum_{j = 1}^{3} π_{j}^{0} δ_{(c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0})} = 0.3 δ_{(- 0.1, 0.04, 0.40, 0.34, 0.01)}$
	$+ 0.4 δ_{(0.1, 0.02, - 0.71, - 0.33, 0.03)} + 0.3 δ_{(0.5, 0.01, 0, 0.2, 0.02)} .$

For Model II, we consider the same setting as in Model I but with $c_{1}^{0} = 0$ and $b_{1}^{0} = 0.3$ . To demonstrate the claim that the empirical convergence rates of parameter estimation under the Type I (Model III) and Type II (Model IV) settings also hold in higher dimensions, we conduct a numerical simulation for $d = 2$ and $k_{0} = 3$ . In Model III, we set $G_{0}$ as

	$\sum_{j = 1}^{3} π_{j}^{0} δ_{(c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0})} = 0.3 δ_{(- 0.1 \cdot 𝟏_{d}, 0.04 \cdot 𝐈_{d}, 0.4 \cdot 𝟏_{d}, 0.34, 0.01)}$
	$+ 0.4 δ_{(0.1 \cdot 𝟏_{d}, 0.02 \cdot 𝐈_{d}, - 0.71 \cdot 𝟏_{d}, - 0.33, 0.03)}$
	$+ 0.3 δ_{(0.5 \cdot 𝟏_{d}, 0.01 \cdot 𝐈_{d}, 𝟎_{d}, 0.2, 0.02)},$

where $𝟏_{d} = (1, 1)$ , $𝟎_{d} = (0, 0)$ and $𝐈_{d}$ is the identity matrix of size $d$ . In Model IV, we consider the same setting of $G_{0}$ as in Model III but with $c_{1}^{0} = 𝟎_{d}$ and $b_{1}^{0} = 0.3$ .

Numerical details. In accordance with the hierarchical GMoE setting of (2), we generate 20 samples ${(X_{i}, Y_{i})}_{i \in [n]}$ of size $n$ for each setting, given $100$ different choices of sample size $n$ between $10^{2}$ and $10^{5}$ . Then, we compute the MLE ${\hat{G}}_{n}$ w.r.t. a number of components $k$ for each sample. For both of these settings, we choose $k \in {k_{0} + 1, k_{0} + 2}$ with corresponding $\bar{r}, \tilde{r} \in {4, 6}$ using Lemmas 1 and 2. Here we implement the MLE using the EM algorithm for GMoE. This is a simplification of a general hybrid GMoE-EM from [4, Section 5]. We choose the convergence criteria $ϵ = 10^{- 5}$ and $2000$ maximum EM iterations. Our goal is to illustrate the theoretical properties of the estimator ${\hat{G}}_{n}$ . Therefore, we have initialized the EM algorithm in a favourable way. More specifically, we first randomly partitioned the set ${1, \dots, k}$ into $k_{0}$ index sets $J_{1}, \dots, J_{k_{0}}$ , each containing at least one point, for any given $k$ and $k_{0}$ and for each replication. Finally, we sampled $c_{j}^{0}$ (resp. $Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0}$ ) from a unique Gaussian distribution centered on $c_{t}^{0}$ (resp. $Γ_{t}^{0}, a_{t}^{0}, b_{t}^{0}, ν_{t}^{0}$ ), with vanishing covariance so that $j \in J_{t}$ .

Empirical convergence rates. The empirical mean of discrepancies $\bar{D}$ and $\tilde{D}$ between ${\hat{G}}_{n}$ and $G_{0}$ , and the choice of $k$ for Models I-II are reported in Figure 1. It can be observed from Figure 1 that those average discrepancies vanish at a rate of order $𝒪 (n^{- 1 / 2})$ , which matches the results of Theorems 1 and 2, where the only theoretical assumption that can be violated is the global convergence of the MLE. Note that the use of the joint density function allows the GMoE to be linked to a hierarchical mixture model, which guarantees global convergence for parameter estimation for arbitrary dimensions, see recent advances, e.g., [27, 26, 28]. We can therefore guarantee that the rates in Theorems 1 and 2 also hold in higher dimensions. Indeed, it can be observed from Figure 2 that the average discrepancies $\bar{D} ({\hat{G}}_{n}, G_{0})$ and $\tilde{D} ({\hat{G}}_{n}, G_{0})$ also approach zero at the rate of order $𝒪 (n^{- 1 / 2})$ for $d = 2$ , confirming the empirical behaviour of Theorems 1 and 2 under the high dimensional settings.

5 CONCLUSION

In this paper, we conduct a convergence analysis for density estimation and parameter estimation in the Gaussian-gated mixture of experts (GMoE) under two complement settings of location parameters of the gating function. We demonstrate that the density estimation rate remains parametric on the sample size under both settings. On the other hand, due to several challenges induced by the interior and exterior interactions among parameters arising in those settings, we have to solve two complex systems of polynomial equations and then propose two corresponding novel Voronoi loss functions among parameters. We show that these Voronoi losses are able to capture the dependence of parameter estimation rates on the number of fitted components, which are more accurate than those characterized by the generalized Wasserstein loss used in previous works. We believe that our current techniques can be extended to the GMoE model with general experts in [18] and to the hierarchical MoE for exponential family models in [20]. In addition, understanding the convergence behavior of least squares estimation under the deterministic MoE model [39] with Gaussian gate is also a potential direction. However, we leave such non-trivial developments for future work.

Acknowledgements

NH acknowledges support from the NSF IFML 2019844 and the NSF AI Institute for Foundations of Machine Learning.

References

[1] J.-P. Baudry, C. Maugis, and B. Michel. Slope heuristics: overview and implementation. Statistics and Computing, 22(2):455–470, 2012. ↩ a
[2] L. Birgé and P. Massart. Minimal penalties for Gaussian model selection. Probability Theory and Related Fields, 138(1):33–73, 2007. Publisher: Springer. ↩ a
[3] F. Chamroukhi and B.-T. Huynh. Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models. Journal de la Société Française de Statistique, 160(1):57–85, 2019. ↩ a
[4] A. Deleforge, F. Forbes, and R. Horaud. High-dimensional regression with gaussian mixtures and partially-latent response variables. Statistics and Computing, 25(5):893–911, 2015. ↩ a b
[5] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–22, Sept. 1977. Publisher: John Wiley & Sons, Ltd. ↩ a
[6] C. Diani, G. Galimberti, and G. Soffritti. Multivariate cluster-weighted models based on seemingly unrelated linear regression. Computational Statistics & Data Analysis, page 107451, Feb. 2022. ↩ a
[7] T. G. Do, H. K. Le, T. Nguyen, Q. Pham, B. T. Nguyen, T.-N. Doan, C. Liu, S. Ramasamy, X. Li, and S. HOI. HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, Dec. 2023. Association for Computational Linguistics. ↩ a
[8] W. Fedus, J. Dean, and B. Zoph. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667, 2022. ↩ a
[9] W. Fedus, B. Zoph, and N. Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. ↩ a
[10] F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel. Mixture of expert posterior surrogates for approximate Bayesian computation. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique (SFdS), Lyon, France, June 2022. ↩ a
[11] F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel. Summary statistics and discrepancy measures for approximate Bayesian computation via surrogate posteriors. Statistics and Computing, 32(5):85, Oct. 2022. ↩ a b
[12] J. Fritsch, M. Finke, and A. Waibel. Adaptively growing hierarchical mixtures of experts. In Advances in Neural Information Processing Systems, volume 9, 1996. ↩ a
[13] A. Guha, N. Ho, and X. Nguyen. On posterior contraction of parameters and interpretability in Bayesian mixture modeling. Bernoulli, 27(4):2159 – 2188, 2021. Publisher: Bernoulli Society for Mathematical Statistics and Probability. ↩ a b c
[14] X. Han, H. Nguyen, C. Harris, N. Ho, and S. Saria. Fusemoe: Mixture-of-experts transformers for fleximodal fusion. arXiv preprint arXiv:2402.03226, 2024. ↩ a
[15] N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. The Annals of Statistics, 44(6):2726 – 2755, 2016. Publisher: Institute of Mathematical Statistics and Bernoulli Society. ↩ a b c d
[16] N. Ho and X. Nguyen. On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics, 10(1):271–307, 2016. Publisher: The Institute of Mathematical Statistics and the Bernoulli Society. ↩ a
[17] N. Ho and X. Nguyen. Singularity Structures and Impacts on Parameter Estimation in Finite Mixtures of Distributions. SIAM Journal on Mathematics of Data Science, 1(4):730–758, Jan. 2019. Publisher: Society for Industrial and Applied Mathematics. ↩ a
[18] N. Ho, C.-Y. Yang, and M. I. Jordan. Convergence Rates for Gaussian Mixtures of Experts. Journal of Machine Learning Research, 23(323):1–81, 2022. ↩ a b
[19] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991. Publisher: MIT Press. ↩ a b
[20] W. Jiang and M. A. Tanner. Hierarchical mixtures-of-experts for exponential family regression models: approximation and maximum likelihood estimation. Annals of Statistics, pages 987–1011, 1999. ↩ a b
[21] W. Jiang and M. A. Tanner. Hierarchical Mixtures-of-Experts for Generalized Linear Models: Some Results on Denseness and Consistency. In D. Heckerman and J. Whittaker, editors, Proceedings of the Seventh International Workshop on Artificial Intelligence and Statistics, volume R2 of Proceedings of Machine Learning Research. PMLR, Jan. 1999. ↩ a
[22] W. Jiang and M. A. Tanner. On the identifiability of mixtures-of-experts. Neural Networks, 12(9):1253–1258, 1999. ↩ a
[23] M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural computation, 6(2):181–214, 1994. Publisher: MIT Press. ↩ a b
[24] A. Khalili. New estimation and feature selection methods in mixture-of-experts models. Canadian Journal of Statistics, 38(4):519–539, 2010. Publisher: Wiley Online Library. ↩ a
[25] B. Kugler, F. Forbes, and S. Douté. Fast Bayesian inversion for high dimensional inverse problems. Statistics and Computing, 32(2):31, Mar. 2022. ↩ a
[26] J. Kwon and C. Caramanis. EM Converges for a Mixture of Many Linear Regressions. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1727–1736. PMLR, Aug. 2020. ↩ a
[27] J. Kwon, N. Ho, and C. Caramanis. On the Minimax Optimality of the EM Algorithm for Learning Two-Component Mixed Linear Regression. In A. Banerjee and K. Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1405–1413. PMLR, Apr. 2021. ↩ a
[28] J. Kwon, W. Qian, C. Caramanis, Y. Chen, and D. Davis. Global Convergence of the EM Algorithm for Mixtures of Two Component Linear Regression. In A. Beygelzimer and D. Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 2055–2110. PMLR, June 2019. ↩ a
[29] S. Lathuilière, R. Juge, P. Mesejo, R. Muñoz-Salinas, and R. Horaud. Deep mixture of linear inverse regressions applied to head-pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4817–4825, 2017. ↩ a
[30] T. Manole and N. Ho. Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Models. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14979–15006. PMLR, July 2022. ↩ a
[31] T. Manole and A. Khalili. Estimating the number of components in finite mixture models via the Group-Sort-Fuse procedure. The Annals of Statistics, 49(6):3043 – 3069, 2021. Publisher: Institute of Mathematical Statistics. ↩ a
[32] S. Masoudnia and R. Ebrahimpour. Mixture of experts: a literature survey. Artificial Intelligence Review, 42(2):275–293, 2014. ↩ a
[33] E. F. Mendes and W. Jiang. On convergence rates of mixtures of polynomial experts. Neural computation, 24(11):3025–3051, 2012. Publisher: MIT Press. ↩ a
[34] J. Moody and C. J. Darken. Fast Learning in Networks of Locally-Tuned Processing Units. Neural Computation, 1(2):281–294, 1989. ↩ a
[35] B. Mustafa, C. R. Ruiz, J. Puigcerver, R. Jenatton, and N. Houlsby. Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. ↩ a
[36] H. Nguyen, P. Akbarian, and N. Ho. Is temperature sample efficient for softmax Gaussian mixture of experts? arXiv preprint arXiv:2401.13875, 2024. ↩ a
[37] H. Nguyen, P. Akbarian, T. Nguyen, and N. Ho. A general theory for softmax gating multinomial logistic mixture of experts. arXiv preprint arXiv:2310.14188, 2023. ↩ a
[38] H. Nguyen, P. Akbarian, F. Yan, and N. Ho. Statistical perspective of top-k sparse softmax gating mixture of experts. In International Conference on Learning Representations, 2024. ↩ a
[39] H. Nguyen, N. Ho, and A. Rinaldo. On least squares estimation in softmax gating mixture of experts. arXiv preprint arXiv:2402.02952, 2024. ↩ a
[40] H. Nguyen, T. Nguyen, and N. Ho. Demystifying softmax gating function in Gaussian mixture of experts. In Advances in Neural Information Processing Systems, 2023. ↩ a
[41] H. D. Nguyen, T. Nguyen, F. Chamroukhi, and G. J. McLachlan. Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models. Journal of Statistical Distributions and Applications, 8(1):13, Aug. 2021. ↩ a
[42] T. Nguyen, F. Chamroukhi, H. D. Nguyen, and F. Forbes. Model selection by penalization in mixture of experts models with a non-asymptotic approach. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique (SFdS), Lyon, France, June 2022. ↩ a
[43] T. Nguyen, F. Forbes, J. Arbel, and H. D. Nguyen. Bayesian nonparametric mixture of experts for high-dimensional inverse problems. hal-04015203, Mar. 2023. ↩ a
[44] T. Nguyen, D. N. Nguyen, H. D. Nguyen, and F. Chamroukhi. A non-asymptotic theory for model selection in high-dimensional mixture of experts via joint rank and variable selection. Preprint hal-03984011, Feb. 2023. ↩ a
[45] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and F. Forbes. A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models. Electronic Journal of Statistics, 16(2):4742 – 4822, 2022. Publisher: Institute of Mathematical Statistics and Bernoulli Society. ↩ a
[46] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and G. J. McLachlan. Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts. arXiv:2009.10622, Feb. 2023. ↩ a
[47] A. Norets and D. Pati. Adaptive Bayesian estimation of conditional densities. Econometric Theory, 33(4):980–1012, 2017. Publisher: Cambridge University Press. ↩ a
[48] A. Norets and J. Pelenis. Adaptive Bayesian estimation of conditional discrete-continuous distributions with an application to stock market trading activity. Journal of Econometrics, 2021. ↩ a
[49] Q. Pham, G. Do, H. Nguyen, T. Nguyen, C. Liu, M. Sartipi, B. T. Nguyen, S. Ramasamy, X. Li, S. Hoi, and N. Ho. Competesmoe – effective training of sparse mixture of experts via competition, 2024. ↩ a
[50] J. Puigcerver, C. R. Ruiz, B. Mustafa, C. Renggli, A. S. Pinto, S. Gelly, D. Keysers, and N. Houlsby. Scalable Transfer Learning with Expert Models. In International Conference on Learning Representations, 2021. ↩ a
[51] M. Sato and S. Ishii. On-line EM algorithm for the normalized gaussian network. Neural computation, 12(2):407–432, feb 2000. ↩ a
[52] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations, 2017. ↩ a
[53] S. van de Geer. Empirical Processes in M-estimation. Cambridge University Press, 2000. ↩ a b c d e f
[54] L. Xu, M. Jordan, and G. E. Hinton. An Alternative Model for Mixtures of Experts. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7. MIT Press, 1995. ↩ a
[55] Z. You, S. Feng, D. Su, and D. Yu. Speechmoe2: Mixture-of-Experts Model with Improved Routing. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7217–7221, 2022. ↩ a
[56] S. E. Yuksel, J. N. Wilson, and P. D. Gader. Twenty Years of Mixture of Experts. IEEE Transactions on Neural Networks and Learning Systems, 23(8):1177–1193, 2012. ↩ a

In this supplementary material, we first include an illustration of Voronoi cells in Appendix A to help the readers understand this concept better. Then, we provide the proof of Theorem 1 and Theorem 2 in Appendix B. Finally, proofs for the remaining results are presented in Appendix C.

Appendix A ILLUSTRATION OF VORONOI CELLS

In this appendix, we aim to illustrate the Voronoi cells defined in Section 2. For that purpose, let us recall the definition of that concept here. In particular, for any mixing measure $G \in 𝒪_{k} (Θ)$ , the Voronoi cell $𝒜_{j} := 𝒜_{j} (G)$ generated by a true component $θ_{j}^{0} := (c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0})$ of $G_{0}$ is given by

𝒜_{j} := {i \in [k] : ‖ θ_{i} - θ_{j}^{0} ‖ \leq ‖ θ_{i} - θ_{ℓ}^{0} ‖, \forall ℓ \neq j},

(14)

for any $j \in [k_{0}]$ , where $θ_{i} := (c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i})$ is a component of $G$ . Now, we provide an illustration of the above Voronoi cells under the setting when $k_{0} = 6$ and $k = 10$ in Figure 3.

Connection to Theorem 1. Under the Type I setting, parameters of the true components $(c_{j}^{0}, Γ_{j}^{0}, a_{j}^{0}, b_{j}^{0}, ν_{j}^{0})$ in cells 3, 5 and 6, which are fitted by one component, enjoy a parametric estimation rate of order $𝒪 (n^{- 1 / 2})$ . Next, the rates for estimating parameters $c_{1}^{0}, b_{1}^{0}$ of the true component in cell 1, which are approximated by three components, stand at order $𝒪 (n^{- 1 / 2 \bar{r} (3)}) = 𝒪 (n^{- 1 / 12})$ , while those for $Γ_{1}^{0}, ν_{1}^{0}$ are of order $𝒪 (n^{- 1 / \bar{r} (3)}) = 𝒪 (n^{- 1 / 6})$ . Meanwhile, the estimation rate for $a_{1}^{0}$ is independent of the cardinality of its corresponding Voronoi cell and remains stable at order $𝒪 (n^{- 1 / 4})$ .

Connection to Theorem 2. Parameter estimation rates under the Type II setting share the same behavior as those in Theorem 1 except for the rates of estimating $a_{j}^{0}$ . More specifically, if $c_{1}^{0} = 0$ , the estimation rate for $a_{1}^{0}$ now depends on the cardinality of cell 1, and experiences a drop to order $𝒪 (n^{- 1 / \tilde{r} (3)}) = 𝒪 (n^{- 1 / 6})$ .

Appendix B PROOF OF MAIN RESULTS

Before going to the proofs for Theorems 1 and 2 in Appendices B.1 and B.2, respectively, let us define some necessary notations used throughout this appendix. Firstly, for any vector $v \in ℝ^{d}$ , either $v_{i}$ or $v^{(i)}$ represents the $i$ -th entry of $v$ , while the sum of its entries is abbreviated as $| v | := v_{1} + v_{2} + \dots + v_{d}$ . Next, for any vector $p \in ℕ^{d}$ , we denote $v^{p} := v_{1}^{p_{1}} v_{2}^{p_{2}} \dots v_{d}^{p_{d}}$ and $p! := p_{1}! p_{2}! \dots p_{d}!$ . Additionally, we sometimes use the notation $h_{1}$ and $h_{2}$ to denote the expert functions considered in this work. In particular, we define $h_{1} (X, a, b) = a^{⊤} X + b$ as the mean expert function for any $X \in 𝒳 \subset ℝ^{d}$ , $a \in ℝ^{d}$ and $b \in ℝ$ , whereas $h_{2} (X, ν) = ν$ stands for the variance expert function for any $ν \in ℝ_{+}$ . Finally, since parameters in the proofs for Theorem 1 and Theorem 2 belong to various high-dimensional spaces, we summarize their domains in Table 2 and Table 3, respectively, to help readers keep track of them.

	$c$	$Γ$	$a$	$b$	$ν$	$τ_{1}$	$τ_{2}$	$α_{1}$	$α_{2}$	$α_{3}$	$α_{4}$	$α_{5}$	$ℓ_{1}$	$ℓ_{2}$
Thm 1	$ℝ^{d}$	$𝒮_{d}^{+}$	$ℝ^{d}$	$ℝ$	$ℝ_{+}$	$ℕ^{d}$	$ℕ$	$ℕ^{d}$	$ℕ^{d \times d}$	$ℕ^{d}$	$ℕ$	$ℕ^{d}$	N/A	N/A

Table 2: Domains for parameters used in the proof of Theorem 1

	$c$	$Γ$	$a$	$b$	$ν$	$τ_{1}$	$τ_{2}$	$α_{1}$	$α_{2}$	$α_{3}$	$α_{4}$	$α_{5}$	$ℓ_{1}$	$ℓ_{2}$
Thm 2	$ℝ$	$ℝ_{+}$	$ℝ$	$ℝ$	$ℝ_{+}$	$ℕ$	$ℕ$	$ℕ$	$ℕ$	$ℕ$	$ℕ$	$ℕ$	$ℕ$	$ℕ$

Table 3: Domains for parameters used in the proof of Theorem 2

B.1 Proof of Theorem 1

Our goal is to show the following inequality:

inf_{G \in 𝒪_{k, β} ​ (Θ)} V ​ (p_{G}, p_{G_{0}}) / \bar{D} ​ (G, G_{0}) > 0,

(15)

which implies the desired Total Variation lower bound $V (p_{{\hat{G}}_{n}}, p_{G_{0}}) ≳ \bar{D} ({\hat{G}}_{n}, G_{0})$ . Given this bound, the joint density estimation rate in Proposition 2 then leads to the convergence rate of the MLE ${\hat{G}}_{n}$ to $G_{0}$ under the loss $\bar{D}$ as follows:

ℙ ​ (\bar{D} ​ ({\hat{G}}_{n}, G_{0}) > C_{3} ​ \sqrt{\log (n) / n}) ≲ n^{- C_{4}},

for some universal constants $C_{3}$ and $C_{4}$ . Note that the infimum in equation (15) is subject to all the mixing measures in the set $𝒪_{k, β} (Θ) := {G = \sum_{i = 1}^{k^{'}} π_{i} δ_{(c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i})} : 1 \leq k^{'} \leq k, \sum_{i = 1}^{k} π_{i} = 1, π_{i} \geq β, (c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i}) \in Θ}$ , for some positive constant $β$ . Now, we divide the proof of inequality (15) into two parts which we refer to as local bound and global bound.

Local bound: Firstly, we will prove the local version of inequality (15):

lim_{ε \to 0} inf_{\begin{matrix} G \in 𝒪_{k, β} ​ (Θ) \\ \bar{D} ​ (G, G_{0}) \leq ε \end{matrix}} V ​ (p_{G}, p_{G_{0}}) / \bar{D} ​ (G, G_{0}) > 0 .

(16)

Assume by contrary that the claim in equation (16) does not hold. Then, there exists a sequence of mixing measures $G_{n} = \sum_{i = 1}^{k_{n}} π_{i}^{n} δ_{(c_{i}^{n}, Γ_{i}^{n}, a_{i}^{n}, b_{i}^{n}, ν_{i}^{n})} \in 𝒪_{k, β} (Θ)$ such that $\bar{D} (G_{n}, G_{0}) \to 0$ and $V (p_{G_{n}}, p_{G_{0}}) / \bar{D} (G_{n}, G_{0}) \to 0$ as $n \to \infty$ . Moreover, since $k_{n} \leq k$ for all $n \in ℕ$ , we can replace $(G_{n})$ by its subsequence that admits a fixed number of atoms $k_{n} = k^{'} \leq k$ . Additionally, $𝒜_{j} = 𝒜_{j}^{n}$ does not change with $n$ for all $j \in [k_{0}]$ .

Step 1 - Taylor expansion for density decomposition: Now, we consider the quantity

		$p_{G_{n}} (X, Y) - p_{G_{0}} (X, Y)$
	$=$	$\sum_{j : \| 𝒜_{j} \| > 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} [f_{ℒ} (X \| c_{i}^{n}, Γ_{i}^{n}) f_{𝒟} (Y \| {(a_{i}^{n})}^{⊤} X + b_{i}^{n}, ν_{i}^{n}) - f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) f_{𝒟} (Y \| {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0})]$
	$+$	$\sum_{j : \| 𝒜_{j} \| = 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} [f_{ℒ} (X \| c_{i}^{n}, Γ_{i}^{n}) f_{𝒟} (Y \| {(a_{i}^{n})}^{⊤} X + b_{i}^{n}, ν_{i}^{n}) - f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) f_{𝒟} (Y \| {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0})]$
	$+$	$\sum_{j = 1}^{k_{0}} (\sum_{i \in 𝒜_{j}} π_{i}^{n} - π_{j}^{0}) f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) f_{𝒟} (Y \| {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0})$
	$:$	$= A_{n} + B_{n} + E_{n} .$

For each $j \in [k_{0}] : | 𝒜_{j} | > 1$ , we perform a Taylor expansion up to the $\bar{r} (| 𝒜_{j} |)$ -th order, and then rewrite $A_{n}$ with a note that $α = (α_{1}, α_{2}, α_{3}, α_{4}, α_{5}) \in ℕ^{d} \times ℕ^{d \times d} \times ℕ^{d} \times ℕ \times ℕ$ as follows:

	$A_{n}$	$= \sum_{j : \| 𝒜_{j} \| > 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} \sum_{\| α \| = 1}^{\bar{r} (\| 𝒜_{j} \|)} \frac{1}{α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}} {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}}$
		$\times \frac{\partial^{\| α_{1} \| + \| α_{2} \|} f_{ℒ}}{\partial c^{α_{1}} \partial Γ^{α_{2}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \cdot \frac{\partial^{\| α_{3} \| + α_{4} + α_{5}} f_{𝒟}}{\partial a^{α_{3}} \partial b^{α_{4}} \partial ν^{α_{5}}} (Y \| {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0}) + R_{1} (X, Y)$

where $R_{1} (X, Y)$ is a remainder term such that $R_{1} (X, Y) / \bar{D} (G_{n}, G_{0}) \to 0$ as $n \to \infty$ , which is due to the uniform Holder continuity of a location-scale Gaussian family. Since $f_{ℒ}$ $d$ -dimensional Gaussian density functions, we have the following partial differential equation (PDE):

\frac{\partial^{| α_{1} | + | α_{2} |} f_{ℒ}}{\partial c^{α_{1}} ​ \partial Γ^{α_{2}}} ​ (X | c_{j}^{0}, Γ_{j}^{0}) = \frac{1}{2^{| α_{2} |}} \cdot \frac{\partial^{| α_{1} | + 2 ​ | α_{2} |} f_{ℒ}}{\partial c^{τ ​ (α_{1}, α_{2})}} ​ (X | c_{j}^{0}, Γ_{j}^{0}),

where $τ (α_{1}, α_{2}) := {(α_{1}^{(v)} + \sum_{u = 1}^{d} (α_{2}^{(u v)} + α_{2}^{(v u)}))}_{v = 1}^{d} = {(α_{1}^{(v)} + 2 \sum_{u = 1}^{d} α_{2}^{(u v)})}_{v = 1}^{d} \in ℕ^{d}$ . Similarly, as $f_{ℒ}$ is an univariate Gaussian density function, then

\frac{\partial^{| α_{3} | + α_{4} + α_{5}} f_{𝒟}}{\partial a^{α_{3}} ​ \partial b^{α_{4}} ​ \partial ν^{α_{5}}} ​ (Y | {(a_{j}^{0})}^{⊤} ​ X + b_{j}^{0}, ν_{j}^{0})

= \frac{X^{α_{3}}}{2^{α_{5}}} \cdot \frac{\partial^{| α_{3} | + α_{4} + 2 ​ α_{5}} f_{𝒟}}{\partial h_{1}^{| α_{3} | + α_{4} + 2 ​ α_{5}}} ​ (Y | {(a_{j}^{0})}^{⊤} ​ X + b_{j}^{0}, ν_{j}^{0}),

where $h_{1} (X, a, b) = a^{⊤} X + b$ is the mean expert function. Combine these results together, $A_{n}$ can be represented as follows:

	$A_{n}$	$= \sum_{j : \| 𝒜_{j} \| > 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} \sum_{\| α \| = 1}^{\bar{r} (\| 𝒜_{j} \|)} \frac{1}{α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}} {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}}$
		$\times \frac{1}{2^{\| α_{2} \|}} \frac{\partial^{\| α_{1} \| + 2 \| α_{2} \|} f_{ℒ}}{\partial c^{τ (α_{1}, α_{2})}} (X \| c_{j}^{0}, Γ_{j}^{0}) \cdot \frac{X^{α_{3}}}{2^{α_{5}}} \frac{\partial^{\| α_{3} \| + α_{4} + 2 α_{5}} f_{𝒟}}{\partial h_{1}^{\| α_{3} \| + α_{4} + 2 α_{5}}} (Y \| {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0}) + R_{1} (X, Y),$

Let $τ_{1} = τ (α_{1}, α_{2}) \in ℕ^{d}$ and $τ_{2} = α_{4} + 2 α_{5} \in ℕ$ , we can rewrite $A_{n}$ as

	$A_{n}$	$= \sum_{j : \| 𝒜_{j} \| > 1} \sum_{\| α_{3} \| = 0}^{\bar{r} (\| 𝒜_{j} \|)} \sum_{\| τ_{1} \| + τ_{2} = 0}^{2 (\bar{r} (\| 𝒜_{j} \|) - \| α_{3} \|)} \sum_{\begin{matrix} τ (α_{1}, α_{2}) = τ_{1} \\ α_{4} + 2 α_{5} = τ_{2} \end{matrix}} \sum_{i \in 𝒜_{j}} \frac{π_{i}^{n}}{2^{\| α_{2} \| + α_{5}} α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}}$
		$\times {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}} \cdot X^{α_{3}} \frac{\partial^{\| τ_{1} \|} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \frac{\partial^{\| α_{3} \| + τ_{2}} f_{𝒟}}{\partial h_{1}^{\| α_{3} \| + τ_{2}}} (Y \| {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0}) + R_{1} (X, Y),$

Analogously, for each $j \in [k_{0}] : | 𝒜_{j} | = 1$ , by means of Taylor expansion up to the first order, $B_{n}$ is rewritten as follows:

	$B_{n}$	$= \sum_{j : \| 𝒜_{j} \| = 1} \sum_{\| α_{3} \| = 0}^{1} \sum_{\| τ_{1} \| + τ_{2} = 0}^{2 (1 - \| α_{3} \|)} \sum_{\begin{matrix} τ (α_{1}, α_{2}) = τ_{1} \\ α_{4} + 2 α_{5} = τ_{2} \end{matrix}} \sum_{i \in 𝒜_{j}} \frac{π_{i}^{n}}{2^{\| α_{2} \| + α_{5}} α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}}$
		$\times {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}} \cdot X^{α_{3}} \frac{\partial^{\| τ_{1} \|} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \frac{\partial^{\| α_{3} \| + τ_{2}} f_{𝒟}}{\partial h_{1}^{\| α_{3} \| + τ_{2}}} (Y \| {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0}) + R_{2} (X, Y),$		(17)

where $R_{2} (X, Y)$ is a remainder such that $R_{2} (X, Y) / \bar{D} (G_{n}, G_{0}) \to 0$ as $n \to \infty$ .

It is worth noting that $A_{n}$ , $B_{n}$ and $E_{n}$ can be treated as linear combinations of elements of the following set:

	$ℱ := {X^{α_{3}} \frac{\partial^{\| τ_{1} \|} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \frac{\partial^{\| α_{3} \| + τ_{2}} f_{𝒟}}{\partial h_{1}^{\| α_{3} \| + τ_{2}}} (Y \|$	${(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0}) : j \in [k_{0}], 0 \leq \| α_{3} \| \leq \bar{r} (\| 𝒜_{j} \|),$
		$0 \leq \| τ_{1} \| + τ_{2} \leq 2 (\bar{r} (\| 𝒜_{j} \|) - \| α_{3} \|)} .$		(18)

Let $T_{α_{3}, τ_{1}, τ_{2}}^{n} (j)$ be the coefficients of

X^{α_{3}} ​ \frac{\partial^{| τ_{1} |} f_{ℒ}}{\partial c^{τ_{1}}} ​ (X | c_{j}^{0}, Γ_{j}^{0}) ​ \frac{\partial^{| α_{3} | + τ_{2}} f_{𝒟}}{\partial h_{1}^{| α_{3} | + τ_{2}}} ​ (Y | {(a_{j}^{0})}^{⊤} ​ X + b_{j}^{0}, ν_{j}^{0})

in the representations of $A_{n}$ , $B_{n}$ and $E_{n}$ .

Step 2 - Proof of non-vanishing coefficients by contradiction: Assume that all the coefficients in the representations of $A_{n} / \bar{D} (G_{n}, G_{0})$ , $B_{n} / \bar{D} (G_{n}, G_{0})$ and $E_{n} / \bar{D} (G_{n}, G_{0})$ go to 0 as $n \to \infty$ . Then, by taking the summation of the absolute values of coefficients in $E_{n} / \bar{D} (G_{n}, G_{0})$ , which are $| T_{𝟎_{d}, 𝟎_{d}, 0} (j) | / \bar{D} (G_{n}, G_{0})$ for all $j \in [k_{0}]$ , we get that

\frac{1}{\bar{D} ​ (G_{n}, G_{0})} \cdot \sum_{j = 1}^{k_{0}} | \sum_{i \in 𝒜_{j}} π_{i}^{n} - π_{j}^{0} | \to 0 .

(19)

Subsequently, from the formulation of $B_{n}$ in equation (B.1), we have

\frac{1}{\bar{D} ​ (G_{n}, G_{0})} \cdot \sum_{j : | 𝒜_{j} | = 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} ​ ({‖ Δ ​ c_{i ​ j}^{n} ‖}_{1} + {‖ Δ ​ Γ_{i ​ j}^{n} ‖}_{1} + {‖ Δ ​ a_{i ​ j}^{n} ‖}_{1} + | Δ ​ b_{i ​ j}^{n} | + | Δ ​ ν_{i ​ j}^{n} |) \to 0 .

It follows from the topological equivalence of $1$ -norm and $2$ -norm that

\frac{1}{\bar{D} ​ (G_{n}, G_{0})} \cdot \sum_{j : | 𝒜_{j} | = 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} ​ (‖ Δ ​ c_{i ​ j}^{n} ‖ + ‖ Δ ​ Γ_{i ​ j}^{n} ‖ + ‖ Δ ​ a_{i ​ j}^{n} ‖ + | Δ ​ b_{i ​ j}^{n} | + | Δ ​ ν_{i ​ j}^{n} |) \to 0 .

(20)

Next, from the formulation of $A_{n}$ , by combining all terms of the form $| T_{α_{3}, 𝟎_{d}, 0} (j) | / \bar{D} (G_{n}, G_{0})$ where $j \in [k_{0}] : | 𝒜_{j} | > 1$ and $α_{3} \in {2 e_{1}, 2 e_{2}, \dots, 2 e_{d}}$ with $e_{u} := (0, \dots, 0, \underset{u-th}{\underset{⏟}{1}}, 0, \dots, 0)$ being a one-hot vector in $ℝ^{d}$ for all $u \in [d]$ , we obtain that

\frac{1}{\bar{D} ​ (G_{n}, G_{0})} \cdot \sum_{j : | 𝒜_{j} | > 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} ​ {‖ Δ ​ a_{i ​ j}^{n} ‖}^{2} \to 0 .

(21)

Putting the results in equations (19), (20) and (21) together with the formulation of $\bar{D} (G_{n}, G_{0})$ in equation (10), we deduce that

\frac{\sum_{j : | 𝒜_{j} | > 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} ​ ({‖ Δ ​ c_{i ​ j}^{n} ‖}^{\bar{r} ​ (| 𝒜_{j} |)} + {‖ Δ ​ Γ_{i ​ j}^{n} ‖}^{\bar{r} ​ (| 𝒜_{j} |) / 2} + {| Δ ​ b_{i ​ j}^{n} |}^{\bar{r} ​ (| 𝒜_{j} |)} + {| Δ ​ ν_{i ​ j}^{n} |}^{\bar{r} ​ (| 𝒜_{j} |) / 2})}{\bar{D} ​ (G_{n}, G_{0})} \to 1 .

As a result, we can find an index $j^{*} \in [k_{0}]$ such that $| 𝒜_{j} | > 1$ and

\frac{\sum_{i \in 𝒜_{j^{*}}} π_{i}^{n} ​ ({‖ Δ ​ c_{i ​ j^{*}}^{n} ‖}^{\bar{r} ​ (𝒜_{j^{*}})} + {‖ Δ ​ Γ_{i ​ j^{*}}^{n} ‖}^{\bar{r} ​ (𝒜_{j^{*}}) / 2} + {| Δ ​ b_{i ​ j^{*}}^{n} |}^{\bar{r} ​ (𝒜_{j^{*}})} + {| Δ ​ ν_{i ​ j^{*}}^{n} |}^{\bar{r} ​ (𝒜_{j^{*}}) / 2})}{\bar{D} ​ (G_{n}, G_{0})} \to̸ 0 .

(22)

Without loss of generality (WLOG), we may assume that $j^{*} = 1$ . Now, we divide our arguments into two main cases as follows:

Case 1: $\frac{1}{\bar{D} (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{1}} π_{i}^{n} ({‖ Δ c_{i 1}^{n} ‖}^{\bar{r} (| 𝒜_{1} |)} + {‖ Δ Γ_{i 1}^{n} ‖}^{\bar{r} (| 𝒜_{1} |) / 2}) \to̸ 0 .$

Here, we continue to split this case into two possibilities:

Case 1.1: $\frac{1}{\bar{D} (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{1}} π_{i}^{n} ({‖ Δ c_{i 1}^{n} ‖}^{\bar{r} (| 𝒜_{1} |)} + {‖ {({(Δ Γ_{i 1}^{n})}^{(u u)})}_{u = 1}^{d} ‖}^{\bar{r} (| 𝒜_{1} |) / 2}) \to̸ 0 .$

In this case, it must hold for some index $u^{*} \in [d]$ that

\frac{1}{\bar{D} ​ (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{1}} π_{i}^{n} ​ ({| {(Δ ​ c_{i ​ 1}^{n})}^{(u^{*})} |}^{\bar{r} ​ (| 𝒜_{1} |)} + {| {(Δ ​ Γ_{i ​ 1}^{n})}^{(u^{*} ​ u^{*})} |}^{\bar{r} ​ (| 𝒜_{1} |) / 2}) \to̸ 0 .

(23)

WLOG, we assume that $u^{*} = 1$ throughout case 1.1. In the representation of $A_{n}$ , we consider the following coefficient:

T_{𝟎_{d}, τ_{1}, 0} ​ (1) = \sum_{i \in 𝒜_{1}} \sum_{\begin{matrix} α_{1}, α_{2} : \\ τ ​ (α_{1}, α_{2}) = τ_{1} \end{matrix}} \frac{π_{i}^{n}}{2^{| α_{2} |} ​ α_{1}! ​ α_{2}!} ​ {(Δ ​ c_{i ​ 1}^{n})}^{α_{1}} ​ {(Δ ​ Γ_{i ​ 1}^{n})}^{α_{2}},

(24)

where $τ_{1} \in ℕ^{d}$ such that $τ_{1}^{(u)} = 0$ for all $u = 2, \dots, d$ . Thus, the constraint $τ (α_{1}, α_{2}) = τ_{1}$ holds if and only if $α_{1}^{(u)} = α_{2}^{(u 1)} = α_{2}^{(1 v)} = α_{2}^{(u v)} = 0$ for all $u, v = 2, \dots, d$ . Therefore, by assumption, we have

\frac{T_{𝟎_{d}, τ_{1}, 0} ​ (1)}{\bar{D} ​ (G_{n}, G_{0})} = \frac{1}{\bar{D} ​ (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{1}} \sum_{\begin{matrix} α_{1}^{(1)} + 2 ​ α_{2}^{(11)} = τ_{1}^{(1)} \end{matrix}} \frac{π_{i}^{n}}{2^{α_{2}^{(11)}} ​ α_{1}^{(1)}! ​ α_{2}^{(11)}!} ​ {(Δ ​ c_{i ​ 1}^{n})}^{α_{1}^{(1)}} ​ {(Δ ​ Γ_{i ​ 1}^{n})}^{α_{2}^{(11)}} \to 0 .

(25)

Collect results in equations (23) and (25), we obtain that

\frac{\sum_{i \in 𝒜_{1}} \sum_{\begin{matrix} α_{1}^{(1)} + 2 ​ α_{2}^{(11)} = τ_{1}^{(1)} \end{matrix}} \frac{π_{i}^{n}}{2^{α_{2}^{(11)}} ​ α_{1}^{(1)}! ​ α_{2}^{(11)}!} ​ {(Δ ​ c_{i ​ 1}^{n})}^{α_{1}^{(1)}} ​ {(Δ ​ Γ_{i ​ 1}^{n})}^{α_{2}^{(11)}}}{\sum_{i \in 𝒜_{1}} π_{i}^{n} ​ ({| {(Δ ​ c_{i ​ 1}^{n})}^{(1)} |}^{\bar{r} ​ (| 𝒜_{1} |)} + {| {(Δ ​ Γ_{i ​ 1}^{n})}^{(11)} |}^{\bar{r} ​ (| 𝒜_{1} |) / 2})} \to 0 .

(26)

Next, we define ${\bar{M}}_{n} = \max {| {(Δ c_{i 1}^{n})}^{(1)} |, {| {(Δ Γ_{i 1}^{n})}^{(11)} |}^{1 / 2} : i \in 𝒜_{1}}$ and ${\bar{π}}_{n} = \max_{i \in 𝒜_{1}} π_{i}^{n}$ . For any $i \in 𝒜_{1}$ , it is clear that the sequence of positive real numbers $(π_{i}^{n} / {\bar{π}}_{n})$ is bounded, therefore, we can replace it by its subsequence that admits a non-negative limit denoted by $p_{i}^{2} = {lim}_{n \to \infty} π_{i}^{n} / {\bar{π}}_{n}$ . In addition, let us denote ${(Δ c_{i 1}^{n})}^{(1)} / {\bar{M}}_{n} \to η_{i}$ and ${(Δ Γ_{i 1}^{n})}^{(11)} / 2 {\bar{M}}_{n}^{2} \to γ_{i}$ . From the formulation of $𝒪_{k, β} (Θ)$ , since $π_{i}^{n} \geq β$ , the real numbers $p_{i}$ will not vanish, and at least one of them is equal to 1. Analogously, at least one of the $η_{i}$ and $γ_{i}$ is equal to either 1 or $- 1$ .

Note that $\sum_{i \in 𝒜_{1}} π_{i}^{n} ({| {(Δ c_{i 1}^{n})}^{(1)} |}^{\bar{r} (| 𝒜_{1} |)} + {| {(Δ Γ_{i 1}^{n})}^{(11)} |}^{\bar{r} (| 𝒜_{1} |) / 2}) / ({\bar{π}}_{n} {\bar{M}}_{n}^{τ_{1}^{(1)}}) \to̸ 0$ for all $τ_{1}^{(1)} \in [\bar{r} (| 𝒜_{1} |)]$ . Thus, we are able to divide both the numerator and the denominator in equation (26) by ${\bar{π}}_{n} {\bar{M}}_{n}^{τ_{1}^{(1)}}$ and let $n \to \infty$ in order to achieve the following system of polynomial equations:

\sum_{i \in 𝒜_{1}} \sum_{α_{1}^{(1)} + 2 ​ α_{2}^{(11)} = τ_{1}^{(1)}} \frac{p_{i}^{2} ​ η_{i}^{α_{1}^{(1)}} ​ γ_{i}^{α_{2}^{(11)}}}{α_{1}^{(1)}! ​ α_{2}^{(11)}!} = 0, τ_{1}^{(1)} \in [\bar{r} ​ (| 𝒜_{1} |)] .

However, by the definition of $\bar{r} (| 𝒜_{1} |)$ , the above system cannot admit any non-trivial solutions, which is a contradiction. Thus, case 1.1 cannot happen.

Case 1.2: $\frac{1}{\bar{D} (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{1}} π_{i}^{n} ({‖ {({(Δ Γ_{i 1}^{n})}^{(u v)})}_{1 \leq u \neq v \leq d} ‖}^{\bar{r} (| 𝒜_{1} |) / 2}) \to̸ 0 .$

In this case, it must hold for some indices $u^{*} \neq v^{*}$ that

\frac{1}{\bar{D} ​ (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{1}} π_{i}^{n} ​ {| {(Δ ​ Γ_{i ​ 1}^{n})}^{(u^{*} ​ v^{*})} |}^{\bar{r} ​ (| 𝒜_{1} |) / 2} \to̸ 0 .

Recall that $| 𝒜_{1} | > 1$ , or equivalently, $| 𝒜_{1} | \geq 2$ , we have that $\bar{r} (| 𝒜_{1} |) \geq 4$ . Therefore, the above equation leads to

\frac{1}{\bar{D} ​ (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{1}} π_{i}^{n} ​ {| {(Δ ​ Γ_{i ​ 1}^{n})}^{(u^{*} ​ v^{*})} |}^{2} \to̸ 0 .

(27)

WLOG, we assume that $u^{*} = 1$ and $v^{*} = 2$ throughout case 1.2. We continue to consider the coefficient $T_{𝟎_{d}, τ_{1}, 0}$ in equation (24) with $τ_{1} = (2, 2, 0, \dots, 0) \in ℕ^{d}$ . By assumption, we have $T_{𝟎_{d}, τ_{1}, 0} / \bar{D} (G_{n}, G_{0}) \to 0$ , which together with equation (27) imply that

\frac{\sum_{i \in 𝒜_{1}} \sum_{\begin{matrix} α_{1}, α_{2} : \\ τ ​ (α_{1}, α_{2}) = τ_{1} \end{matrix}} \frac{π_{i}^{n}}{2^{| α_{2} |} ​ α_{1}! ​ α_{2}!} ​ {(Δ ​ c_{i ​ 1}^{n})}^{α_{1}} ​ {(Δ ​ Γ_{i ​ 1}^{n})}^{α_{2}}}{\sum_{i \in 𝒜_{1}} π_{i}^{n} ​ {| {(Δ ​ Γ_{i ​ 1}^{n})}^{(12)} |}^{2}} \to 0 .

(28)

Similarly, by combining the fact that case 1.1 does not hold and the result in equation (27), we get

\frac{\sum_{i \in 𝒜_{1}} π_{i}^{n} ​ ({‖ Δ ​ c_{i ​ 1}^{n} ‖}^{\bar{r} ​ (| 𝒜_{1} |)} + {‖ {({(Δ ​ Γ_{i ​ 1}^{n})}^{(u ​ u)})}_{u = 1}^{d} ‖}^{\bar{r} ​ (| 𝒜_{1} |) / 2})}{\sum_{i \in 𝒜_{1}} π_{i}^{n} ​ {| {(Δ ​ Γ_{i ​ 1}^{n})}^{(12)} |}^{2}} \to 0 .

Since $\bar{r} (| 𝒜_{1} |) \geq 4$ , the above limit indicates that any terms in equation (28) with $α_{1}^{(u)} > 0$ and $α_{2}^{(u u)} > 0$ for $u \in {1, 2}$ will vanish. Consequently, we deduce from equation (28) that

1 = \frac{\sum_{i \in 𝒜_{1}} π_{i}^{n} ​ {| {(Δ ​ Γ_{i ​ 1}^{n})}^{(12)} |}^{2}}{\sum_{i \in 𝒜_{1}} π_{i}^{n} ​ {| {(Δ ​ Γ_{i ​ 1}^{n})}^{(12)} |}^{2}} \to 0,

which is a contradiction. Thus, case 1.2 cannot happen.

Case 2: $\frac{1}{\bar{D} (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{1}} π_{i}^{n} ({| Δ b_{i 1}^{n} |}^{\bar{r} (| 𝒜_{1} |)} + {| Δ ν_{i 1}^{n} |}^{\bar{r} (| 𝒜_{1} |) / 2}) \to̸ 0 .$

In this case, we consider the coefficient $T_{𝟎_{d}, 𝟎_{d}, 0} (1)$ in the formulation of $A_{n}$ . By assumption,

\frac{T_{𝟎_{d}, 𝟎_{d}, 0} ​ (1)}{\bar{D} ​ (G_{n}, G_{0})} = \frac{1}{\bar{D} ​ (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{1}} π_{i}^{n} ​ \sum_{\begin{matrix} α_{4}, α_{5} : \\ α_{4} + 2 ​ α_{5} = τ_{2} \end{matrix}} \frac{{(Δ ​ b_{i ​ 1}^{n})}^{α_{4}} ​ {(Δ ​ ν_{i ​ 1}^{n})}^{α_{5}}}{2^{α_{5}} ​ α_{4}! ​ α_{5}!} \to 0 .

Consequently, we obtain that

\frac{\sum_{i \in 𝒜_{1}} π_{i}^{n} ​ \sum_{\begin{matrix} α_{4}, α_{5} : \\ α_{4} + 2 ​ α_{5} = τ_{2} \end{matrix}} \frac{{(Δ ​ b_{i ​ 1}^{n})}^{α_{4}} ​ {(Δ ​ ν_{i ​ 1}^{n})}^{α_{5}}}{2^{α_{5}} ​ α_{4}! ​ α_{5}!}}{\sum_{i \in 𝒜_{1}} π_{i}^{n} ​ ({| Δ ​ b_{i ​ 1}^{n} |}^{\bar{r} ​ (| 𝒜_{1} |)} + {| Δ ​ ν_{i ​ 1}^{n} |}^{\bar{r} ​ (| 𝒜_{1} |) / 2})} \to 0 .

By employing the same arguments for showing that the equation (26) does not hold in case 1.1, we obtain that the above limit does not hold, either. Thus, case 2 cannot happen.

From the above results of the two main cases, we conclude that not all the coefficients in the representations of $A_{n} / \bar{D} (G_{n}, G_{0})$ , $B_{n} / \bar{D} (G_{n}, G_{0})$ and $E_{n} / \bar{D} (G_{n}, G_{0})$ vanish as $n \to \infty$ .

Step 3 - Application of Fatou’s lemma: Subsequently, we denote by $m_{n}$ the maximum of the absolute values of the coefficients in the representations of $A_{n} / \bar{D} (G_{n}, G_{0})$ , $B_{n} / \bar{D} (G_{n}, G_{0})$ and $E_{n} / \bar{D} (G_{n}, G_{0})$ , that is,

m_{n} := \max_{(α_{3}, τ_{1}, τ_{2}, j) \in 𝒮} | T_{α_{3}, τ_{1}, τ_{2}}^{n} ​ (j) | / \bar{D} ​ (G_{n}, G_{0}),

where the constraint set $𝒮$ is defined as

𝒮 := {(α_{3}, τ_{1}, τ_{2}, j) \in ℕ^{d} \times ℕ^{d} \times ℕ \times [k_{0}] : 0 \leq | α_{3} | \leq \bar{r} ​ (| 𝒜_{j} |), 0 \leq | τ_{1} |, τ_{2} \leq 2 ​ (\bar{r} ​ (| 𝒜_{j} |) - | α_{3} |)} .

Additionally, we define $T_{α_{3}, τ_{1}, τ_{2}}^{n} (j) / m_{n} \to ξ_{α_{3}, τ_{1}, τ_{2}} (j)$ as $n \to \infty$ for all $(α_{3}, τ_{1}, τ_{2}, j) \in 𝒮$ . Since not all the coefficients in the representations of $A_{n} / \bar{D} (G_{n}, G_{0})$ , $B_{n} / \bar{D} (G_{n}, G_{0})$ and $E_{n} / \bar{D} (G_{n}, G_{0})$ vanish as $n \to \infty$ , at least one among $ξ_{α_{3}, τ_{1}, τ_{2}} (j)$ is different from zero and $m_{n} \to̸ 0$ . Then, by applying the Fatou’s lemma, we get that

0 = lim_{n \to \infty} \frac{1}{m_{n}} \cdot \frac{2 ​ V ​ (p_{G_{n}}, p_{G})}{\bar{D} ​ (G_{n}, G_{0})} \geq \int \underset{n \to \infty}{lim inf} \frac{1}{m_{n}} \cdot \frac{| p_{G_{n}} ​ (X, Y) - p_{G} ​ (X, Y) |}{\bar{D} ​ (G_{n}, G_{0})} ​ d ​ (X, Y) \geq 0 .

Moreover, by definition, we have

	$\frac{1}{m_{n}} \cdot$	$\frac{p_{G_{n}} (X, Y) - p_{G} (X, Y)}{\bar{D} (G_{n}, G_{0})}$
		$\to \sum_{(α_{3}, τ_{1}, τ_{2}, j) \in 𝒮} ξ_{α_{3}, τ_{1}, τ_{2}} (j) X^{α_{3}} \frac{\partial^{\| τ_{1} \|} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \frac{\partial^{\| α_{3} \| + τ_{2}} f_{𝒟}}{\partial h_{1}^{\| α_{3} \| + τ_{2}}} (Y \| {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0}) .$

As a consequence, we achieve that

\sum_{(α_{3}, τ_{1}, τ_{2}, j) \in 𝒮} ξ_{α_{3}, τ_{1}, τ_{2}} ​ (j) ​ X^{α_{3}} ​ \frac{\partial^{| τ_{1} |} f_{ℒ}}{\partial c^{τ_{1}}} ​ (X | c_{j}^{0}, Γ_{j}^{0}) ​ \frac{\partial^{| α_{3} | + τ_{2}} f_{𝒟}}{\partial h_{1}^{| α_{3} | + τ_{2}}} ​ (Y | {(a_{j}^{0})}^{⊤} ​ X + b_{j}^{0}, ν_{j}^{0}) = 0,

for almost surely $(X, Y)$ . Since elements of the set $ℱ$ defined in equation (18) are linearly independent (proof of this claim is deferred to the end of this proof), the above equation implies that $ξ_{α_{3}, τ_{1}, τ_{2}} (j) = 0$ for all $(α_{3}, τ_{1}, τ_{2}, j) \in 𝒮$ , which contradicts the fact that at least one among $ξ_{α_{3}, τ_{1}, τ_{2}} (j)$ is different from zero. Hence, we reach the conclusion in equation (16), which indicates that there exists some $ε_{0} > 0$ such that

inf_{\begin{matrix} G \in 𝒪_{k, β} ​ (Θ) \\ \bar{D} ​ (G, G_{0}) \leq ε_{0} \end{matrix}} V ​ (p_{G}, p_{G_{0}}) / \bar{D} ​ (G, G_{0}) > 0 .

Global bound: Given the above result, in order to achieve the inequality in equation (15), we only need to prove its following global version:

inf_{\begin{matrix} G \in 𝒪_{k, β} ​ (Θ) \\ \bar{D} ​ (G, G_{0}) > ε_{0} \end{matrix}} V ​ (p_{G}, p_{G_{0}}) / \bar{D} ​ (G, G_{0}) > 0 .

Assume by contrary that the above claim is not true. Then, there exists a sequence $G_{n}^{'} \in 𝒪_{k, β} (Θ)$ such that $V (p_{G_{n}^{'}}, p_{G_{0}}) / \bar{D} (G_{n}^{'}, G_{0}) \to 0$ and $\bar{D} (G_{n}^{'}, G_{0}) > ε_{0}$ for all $n \in ℕ$ . Since the set $Θ$ is compact, we can replace $G_{n}^{'}$ by its subsequence that converges to some mixing measure $G^{'} \in 𝒪_{k, β} (Θ)$ . Consequently, we deduce that $\bar{D} (G^{'}, G_{0}) = {lim}_{n \to \infty} \bar{D} (G_{n}^{'}, G_{0}) \geq ε_{0}$ . This result together with the fact that $V (p_{G_{n}^{'}}, p_{G_{0}}) / \bar{D} (G_{n}^{'}, G_{0}) \to 0$ lead to the limit $V (p_{G_{n}^{'}}, p_{G_{0}}) \to 0$ as $n \to \infty$ . Again, by applying the Fatou’s lemma, we obtain that

	$0 = lim_{n \to \infty} 2 V (p_{G_{n}^{'}}, p_{G_{0}})$	$\geq \int \underset{n \to \infty}{lim inf} \| p_{G_{n}^{'}} (X, Y) - p_{G_{0}} (X, Y) \| d (X, Y)$
		$= \int \| p_{G^{'}} (X, Y) - p_{G_{0}} (X, Y) \| d (X, Y) \geq 0 .$

As a consequence, we have that $p_{G^{'}} (X, Y) = p_{G_{0}} (X, Y)$ for almost surely $(X, Y)$ . Due to the identifiability of the model, this equality leads to $G^{'} \equiv G_{0}$ , which contradicts the bound $\bar{D} (G^{'}, G_{0}) \geq ε_{0} > 0$ . Hence, we achieve the conclusion in equation (15).

Linear independence of elements in $ℱ$ : For completion, we will demonstrate elements of the set $ℱ$ defined in equation (18) are linearly independent by definition. In particular, assume that there exist real numbers $ξ_{α_{3}, τ_{1}, τ_{2}} (j)$ , where $(α_{3}, τ_{1}, τ_{2}, j) \in 𝒮$ , such that the following equation holds for almost surely $(X, Y)$ :

\sum_{(α_{3}, τ_{1}, τ_{2}, j) \in 𝒮} ξ_{α_{3}, τ_{1}, τ_{2}} ​ (j) ​ X^{α_{3}} ​ \frac{\partial^{| τ_{1} |} f_{ℒ}}{\partial c^{τ_{1}}} ​ (X | c_{j}^{0}, Γ_{j}^{0}) ​ \frac{\partial^{| α_{3} | + τ_{2}} f_{𝒟}}{\partial h_{1}^{| α_{3} | + τ_{2}}} ​ (Y | {(a_{j}^{0})}^{⊤} ​ X + b_{j}^{0}, ν_{j}^{0}) = 0 .

Now, we rewrite the above equation as follows:

	$\sum_{j = 1}^{k_{0}} \sum_{ω = 0}^{2 \bar{r} (\| 𝒜_{j} \|)} (\sum_{\| α_{3} \| + τ_{2} = ω} \sum_{\| τ_{1} \| = 0}^{2 (\bar{r} (\| 𝒜_{j} \|) - α_{3}) - τ_{2}}$	$ξ_{α_{3}, τ_{1}, τ_{2}} (j) X^{α_{3}} \cdot \frac{\partial^{\| τ_{1} \|} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}))$
		$\times \frac{\partial^{ω} f_{𝒟}}{\partial h_{1}^{ω}} (Y \| {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0}) = 0,$		(29)

for almost surely $(X, Y)$ . As $(a_{j}^{0}, b_{j}^{0}, ν_{j}^{0})$ for $j \in [k_{0}]$ are $k_{0}$ distinct tuples, we deduce that $({(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0})$ for $j \in [k_{0}]$ are also $k_{0}$ distinct tuples for almost surely $X$ . Thus, for almost surely $X$ , one has $\frac{\partial^{ω} f_{𝒟}}{\partial h_{1}^{ω}} (Y | {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0})$ for $j \in [k_{0}]$ and $0 \leq ω \leq 2 \bar{r} (| 𝒜_{j} |)$ are linearly independent with respect to $Y$ . Given that result, the equation (B.1) indicates that for almost surely $X$ ,

\sum_{| α_{3} | + τ_{2} = ω} \sum_{| τ_{1} | = 0}^{2 ​ (\bar{r} ​ (| 𝒜_{j} |) - ω)} ξ_{α_{3}, τ_{1}, τ_{2}} ​ (j) ​ X^{α_{3}} \cdot \frac{\partial^{| τ_{1} |} f_{ℒ}}{\partial c^{τ_{1}}} ​ (X | c_{j}^{0}, Γ_{j}^{0}) = 0,

for all $j \in [k_{0}]$ and $0 \leq ω \leq 2 \bar{r} (| 𝒜_{j} |)$ . Note that for each $j \in [k_{0}]$ and $0 \leq ω \leq \bar{r} (| 𝒜_{j} |)$ , the left hand side of the above equation can be viewed as a high-dimensional polynomial of two random vectors $X$ and $X - c_{j}^{0}$ ( $c_{j}^{0} \neq 𝟎_{d}$ ) in $𝒳$ , which is a compact set in $ℝ^{d}$ . As a result, the above equation holds when $ξ_{α_{3}, τ_{1}, τ_{2}} (j) = 0$ for all $j \in [k_{0}]$ , $0 \leq ω \leq 2 \bar{r} (| 𝒜_{j} |)$ , $| α_{3} | + τ_{2} = ω$ and $| τ_{1} | \leq 2 (\bar{r} (| 𝒜_{j} |) - α_{3}) - τ_{2}$ . This is equivalent to $ξ_{α_{3}, τ_{1}, τ_{2}} (j) = 0$ for all $(α_{3}, τ_{1}, τ_{2}, j) \in 𝒮$ .

Hence, we conclude that the elements of $ℱ$ are linearly independent.

B.2 Proof of Theorem 2

In order to reach the conclusion in Theorem 2, we only need to demonstrate the following inequality:

inf_{G \in 𝒪_{k, β} ​ (Θ)} V ​ (p_{G}, p_{G_{0}}) / \tilde{D} ​ (G, G_{0}) > 0 .

(30)

In this proof, we will only prove the following local version of inequality (30) while the global version can be argued in the same fashion as in Appendix (B.1):

lim_{ε \to 0} inf_{\begin{matrix} G \in 𝒪_{k, β} ​ (Θ) \\ \tilde{D} ​ (G, G_{0}) \leq ε \end{matrix}} V ​ (p_{G}, p_{G_{0}}) / \tilde{D} ​ (G, G_{0}) > 0 .

(31)

Assume that the claim in equation (31) is not true. This indicates that we can find a sequence of mixing measures $G_{n} = \sum_{i = 1}^{k_{n}} π_{i}^{n} δ_{(c_{i}, Γ_{i}^{n}, a_{i}^{n}, b_{i}^{n}, ν_{i}^{n})} \in 𝒪_{k, β} (Θ)$ that satisfies: $\tilde{D} (G_{n}, G_{0}) \to 0$ and $V (p_{G_{n}}, p_{G_{0}}) / \tilde{D} (G_{n}, G_{0}) \to 0$ as $n \to \infty$ . Additionally, since $k_{n} \leq k$ for all $n \in ℕ$ , we are able to replace $(G_{n})$ by its subsequence which admits a fixed number of atoms $k_{n} \leq k^{'} \leq k$ and $𝒜_{j} = 𝒜_{j}^{n}$ is independent of $n$ for all $j \in [k_{0}]$ .

Step 1 - Taylor expansion for density decomposition: Next, we take into account the quantity

	$p_{G_{n}} (X, Y) - p_{G_{0}} (X, Y)$
	$= \sum_{j : \| 𝒜_{j} \| > 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} [f_{ℒ} (X \| c_{i}^{n}, Γ_{i}^{n}) f_{𝒟} (Y \| a_{i}^{n} X + b_{i}^{n}, ν_{i}^{n}) - f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) f_{𝒟} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0})]$
	$+ \sum_{j : \| 𝒜_{j} \| = 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} [f_{ℒ} (X \| c_{i}^{n}, Γ_{i}^{n}) f_{𝒟} (Y \| a_{i}^{n} X + b_{i}^{n}, ν_{i}^{n}) - f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) f_{𝒟} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0})]$
	$+ \sum_{j = 1}^{k_{0}} (\sum_{i \in 𝒜_{j}} π_{i}^{n} - π_{j}^{0}) f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) f_{𝒟} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0})$
	$:= A_{n} + B_{n} + E_{n} .$

For each $j \in [k_{0}] : | 𝒜_{j} | > 1$ , by means of Taylor expansion up to the $\tilde{r} (| 𝒜_{j} |)$ -th order, $A_{n}$ can be rewritten as follows with a note that $α = (α_{1}, α_{2}, α_{3}, α_{4}, α_{5}) \in ℕ^{5}$ :

	$A_{n}$	$= \sum_{j : \| 𝒜_{j} \| > 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} \sum_{\| α \| = 1}^{\tilde{r} (\| 𝒜_{j} \|)} \frac{1}{α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}} {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}}$
		$\times \frac{\partial^{α_{1} + α_{2}} f_{ℒ}}{\partial c^{α_{1}} \partial Γ^{α_{2}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \cdot \frac{\partial^{α_{3} + α_{4} + α_{5}} f}{\partial a^{α_{3}} \partial b^{α_{4}} \partial ν^{α_{5}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) + R_{1} (X, Y)$
		$= \sum_{j : \| 𝒜_{j} \| > 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} \sum_{\| α \| = 1}^{\tilde{r} (\| 𝒜_{j} \|)} \frac{1}{α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}} {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}}$
		$\times \frac{1}{2^{α_{2}}} \frac{\partial^{α_{1} + 2 α_{2}} f_{ℒ}}{\partial c^{α_{1} + 2 α_{2}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \cdot \frac{X^{α_{3}}}{2^{α_{5}}} \frac{\partial^{α_{3} + α_{4} + 2 α_{5}} f_{𝒟}}{\partial h_{1}^{α_{3} + α_{4} + 2 α_{5}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) + R_{3} (X, Y),$

where $R_{3} (X, Y)$ is Taylor remainder such that $R_{3} (X, Y) / \tilde{D} (G_{n}, G_{0}) \to 0$ . Since $c_{j}^{0}$ is equal to zero when $j \in [\tilde{k}]$ and different from zero otherwise, the formulation of $\frac{\partial^{α_{1} + 2 α_{2}} f_{ℒ}}{\partial c^{α_{1} + 2 α_{2}}} (X | c_{j}^{0}, Γ_{j}^{0})$ will vary when $j \in [\tilde{k}]$ compared to $\tilde{k} + 1 \leq j \leq k_{0}$ . Thus, we will consider these two cases of $j$ separately.

For $j \in [\tilde{k}]$ , when $α_{1}$ is an even integer, we have

\frac{\partial^{α_{1} + 2 ​ α_{2}} f_{ℒ}}{\partial c^{α_{1} + 2 ​ α_{2}}} ​ (X | c_{j}^{0}, Γ_{j}^{0}) = {\begin{matrix} \sum_{w = 0}^{α_{1} / 2 + α_{2}} t_{2 ​ w, α_{1} + 2 ​ α_{2}} ​ X^{2 ​ w}, j \in [\tilde{k}] \\ \sum_{w = 0}^{α_{1} / 2 + α_{2}} s_{2 ​ w, α_{1} + 2 ​ α_{2}} ​ {(X - c_{j}^{0})}^{2 ​ w}, \tilde{k} + 1 \leq j \leq k_{0} . \end{matrix}

On the other hand, when $α_{1}$ is an odd integer, we get

\frac{\partial^{α_{1} + 2 ​ α_{2}} f_{ℒ}}{\partial c^{α_{1} + 2 ​ α_{2}}} ​ (X | c_{j}^{0}, Γ_{j}^{0}) = {\begin{matrix} \sum_{w = 0}^{(α_{1} - 1) / 2 + α_{2}} t_{2 ​ w + 1, α_{1} + 2 ​ α_{2}} ​ X^{2 ​ w + 1}, j \in [\tilde{k}] \\ \sum_{w = 0}^{(α_{1} - 1) / 2 + α_{2}} s_{2 ​ w + 1, α_{1} + 2 ​ α_{2}} ​ {(X - c_{j}^{0})}^{2 ​ w + 1}, \tilde{k} + 1 \leq j \leq k_{0} . \end{matrix}

By combining both cases, we rewrite $A_{n}$ as follows:

$A_{n}$	$= \sum_{\begin{matrix} j : \| 𝒜_{j} \| > 1, \\ j \in [\tilde{k}] \end{matrix}} \sum_{i \in 𝒜_{j}} \sum_{ℓ_{1} + ℓ_{2} = 1}^{2 \tilde{r} (\| 𝒜_{j} \|)} \sum_{α \in ℐ_{ℓ_{1}, ℓ_{2}}} \frac{π_{i}^{n}}{2^{α_{2} + α_{5}} α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}}$
	$\times {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}} t_{ℓ_{1} - α_{3}, α_{1} + 2 α_{2}} X^{ℓ_{1}} \frac{\partial^{ℓ_{2}} f_{𝒟}}{\partial h_{1}^{ℓ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0})$
$+$	$\sum_{\begin{matrix} j : \| 𝒜_{j} \| > 1 \\ \tilde{k} + 1 \leq j \leq k_{0} \end{matrix}} \sum_{i \in 𝒜_{j}} \sum_{α_{3} = 0}^{\tilde{r} (\| 𝒜_{j} \|)} \sum_{τ_{1} + τ_{2} = 0}^{2 (\tilde{r} (\| 𝒜_{j} \|) - α_{3})} \sum_{\begin{matrix} α_{1} + 2 α_{2} = τ_{1} \\ α_{4} + 2 α_{5} = τ_{2} \end{matrix}} \frac{π_{i}^{n}}{2^{α_{2} + α_{5}} α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}}$
	$\times {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}} \times X^{α_{3}} \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \frac{\partial^{α_{3} + τ_{2}} f_{𝒟}}{\partial h_{1}^{α_{3} + τ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) + R_{3} (X, Y),$	(32)

where for any $0 \leq ℓ_{1} \leq 2 \bar{r} (| 𝒜_{j} |)$ and $0 \leq ℓ_{2} \leq 2 \bar{r} (| 𝒜_{j} |) - ℓ_{1}$ , we define

	$ℐ_{ℓ_{1}, ℓ_{2}} := {α = {(α_{i})}_{i = 1}^{5} \in ℕ^{5} :$	$α_{1} + 2 α_{2} + α_{3} \geq ℓ_{1}, α_{3} + α_{4} + 2 α_{5} = ℓ_{2},$
		$1 \leq α_{1} + α_{2} + \dots + α_{5} \leq \tilde{r} (\| 𝒜_{j} \|)} .$

Regarding the formulation of $B_{n}$ , for each $j \in [k_{0}] : | 𝒜_{j} | = 1$ , we perform a Taylor expansion up to the first order and obtain that

$B_{n}$	$= \sum_{\begin{matrix} j : \| 𝒜_{j} \| = 1, \\ j \in [\tilde{k}] \end{matrix}} \sum_{i \in 𝒜_{j}} \sum_{ℓ_{1} + ℓ_{2} = 1}^{2} \sum_{α \in ℐ_{ℓ_{1}, ℓ_{2}}} \frac{π_{i}^{n}}{2^{α_{2} + α_{5}} α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}}$
	$\times {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}} t_{ℓ_{1} - α_{3}, α_{1} + 2 α_{2}} X^{ℓ_{1}} \frac{\partial^{ℓ_{2}} f_{𝒟}}{\partial h_{1}^{ℓ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0})$
	$+ \sum_{\begin{matrix} j : \| 𝒜_{j} \| = 1 \\ \tilde{k} + 1 \leq j \leq k_{0} \end{matrix}} \sum_{i \in 𝒜_{j}} \sum_{α_{3} = 0}^{2} \sum_{τ_{1} + τ_{2} = 0}^{2 (1 - α_{3})} \sum_{\begin{matrix} α_{1}, α_{2} : \\ α_{1} + 2 α_{2} = τ_{1} \end{matrix}} \sum_{\begin{matrix} α_{4}, α_{5} : \\ α_{4} + 2 α_{5} = τ_{2} \end{matrix}} \frac{π_{i}^{n}}{2^{α_{2} + α_{5}} α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}}$
	$\times {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}} \times X^{α_{3}} \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \frac{\partial^{α_{3} + τ_{2}} f_{𝒟}}{\partial h_{1}^{α_{3} + τ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) + R_{4} (X, Y),$	(33)

where $R_{4} (X, Y)$ is a Taylor remainder such that $R_{4} (X, Y) / \tilde{D} (G_{n}, G_{0}) \to 0$ as $n \to \infty$ .

From equations (32) and (B.2), we can treat $A_{n} / \tilde{D} (G_{n}, G_{0})$ , $B_{n} / \tilde{D} (G_{n}, G_{0})$ and $E_{n} / \tilde{D} (G_{n}, G_{0})$ as linear combinations of elements of the following set:

$ℋ :$	$= {X^{ℓ_{1}} \frac{\partial^{ℓ_{2}} f_{𝒟}}{\partial h_{1}^{ℓ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) : j \in [\tilde{k}], 0 \leq ℓ_{1} + ℓ_{2} \leq 2 \tilde{r} (\| 𝒜_{j} \|)}$
	$\cup {X^{α_{3}} \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \frac{\partial^{α_{3} + τ_{2}} f_{𝒟}}{\partial h_{1}^{α_{3} + τ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) : \tilde{k} + 1 \leq j \leq k_{0}, 0 \leq α_{3} \leq \tilde{r} (\| 𝒜_{j} \|),$
	$0 \leq τ_{1} + τ_{2} \leq 2 (\tilde{r} (\| 𝒜_{j} \|) - α_{3})} .$	(34)

For any $(j, ℓ_{1}, ℓ_{2}) \in 𝒬 := {(j, ℓ_{1}, ℓ_{2}) \in ℕ^{3} : j \in [\tilde{k}], 0 \leq ℓ_{1} + ℓ_{2} \leq 2 \tilde{r} (| 𝒜_{j} |)}$ , let $Q_{ℓ_{1}, ℓ_{2}}^{n} (j)$ be the coefficient of

X^{ℓ_{1}} ​ \frac{\partial^{ℓ_{2}} f_{𝒟}}{\partial h_{1}^{ℓ_{2}}} ​ (Y | a_{j}^{0} ​ X + b_{j}^{0}, ν_{j}^{0}) ​ f_{ℒ} ​ (X | c_{j}^{0}, Γ_{j}^{0})

in the representations of $A_{n}$ , $B_{n}$ and $E_{n}$ . It follows from equations (32) and (B.2) that $Q_{ℓ_{1}, ℓ_{2}}^{n} (j)$ is given by

Q_{ℓ_{1}, ℓ_{2}}^{n} ​ (j) = {\begin{matrix} \sum_{α \in ℐ_{ℓ_{1}, ℓ_{2}}} \sum_{i \in 𝒜_{j}} \frac{π_{i}^{n} \cdot t_{ℓ_{1} - α_{3}, α_{1} + 2 ​ α_{2}}}{2^{α_{2} + α_{5}} ​ α!} ​ {(Δ ​ c_{i ​ j}^{n})}^{α_{1}} ​ {(Δ ​ Γ_{i ​ j}^{n})}^{α_{2}} ​ {(Δ ​ a_{i ​ j}^{n})}^{α_{3}} ​ {(Δ ​ b_{i ​ j}^{n})}^{α_{4}} ​ {(Δ ​ ν_{i ​ j}^{n})}^{α_{5}} \\ (ℓ_{1}, ℓ_{2}) \neq (0, 0), \\ \sum_{i \in 𝒜_{j}} π_{i}^{n} - π_{j}^{0}, (ℓ_{1}, ℓ_{2}) = (0, 0) . \end{matrix}

Meanwhile, we denote by $T_{α_{3}, τ_{1}, τ_{2}}^{n} (j)$ the coefficient of

X^{α_{3}} ​ \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} ​ (X | c_{j}^{0}, Γ_{j}^{0}) ​ \frac{\partial^{α_{3} + τ_{2}} f_{𝒟}}{\partial h_{1}^{α_{3} + τ_{2}}} ​ (Y | a_{j}^{0} ​ X + b_{j}^{0}, ν_{j}^{0}),

for all $(j, α_{3}, τ_{1}, τ_{2}) \in 𝒯 := {(j, α_{3}, τ_{1}, τ_{2}) \in ℕ^{3} : \tilde{k} + 1 \leq j \leq k_{0}, 0 \leq α_{3} \leq \tilde{r} (| 𝒜_{j} |), 0 \leq τ_{1} + τ_{2} \leq 2 (\tilde{r} (| 𝒜_{j} |) - α_{3})}$ . Thus, $T_{α_{3}, τ_{1}, τ_{2}}^{n} (j)$ is represented as

T_{α_{3}, τ_{1}, τ_{2}}^{n} ​ (j) = {\begin{matrix} \sum_{\begin{matrix} α_{1} + 2 ​ α_{2} = τ_{1}, \\ α_{4} + 2 ​ α_{5} = τ_{2} \end{matrix}} \sum_{i \in 𝒜_{j}} \frac{π_{i}^{n}}{2^{α_{2} + α_{5}} ​ α!} ​ {(Δ ​ c_{i ​ j}^{n})}^{α_{1}} ​ {(Δ ​ Γ_{i ​ j}^{n})}^{α_{2}} ​ {(Δ ​ a_{i ​ j}^{n})}^{α_{3}} ​ {(Δ ​ b_{i ​ j}^{n})}^{α_{4}} ​ {(Δ ​ ν_{i ​ j}^{n})}^{α_{5}} \\ (α_{3}, τ_{1}, τ_{2}) \neq (0, 0, 0), \\ \sum_{i \in 𝒜_{j}} π_{i}^{n} - π_{j}^{0}, (α_{3}, τ_{1}, τ_{2}) = (0, 0, 0) . \end{matrix}

Step 2 - Proof of non-vanishing coefficients by contradiction: Assume by contrary that all the coefficients of elements in the set $ℋ$ in the representations of $A_{n} / \tilde{D} (G_{n}, G_{0})$ , $B_{n} / \tilde{D} (G_{n}, G_{0})$ and $E_{n} / \tilde{D} (G_{n}, G_{0})$ vanish when $n$ tends to infinity. It is worth noting that for $(ℓ_{1}, ℓ_{2}) \neq (0, 0)$ , we have $ℐ_{ℓ_{1} + 1, ℓ_{2}} \subseteq ℐ_{ℓ_{1}, ℓ_{2}}$ and

	$𝒥_{ℓ_{1}, ℓ_{2}} := ℐ_{ℓ_{1}, ℓ_{2}} ∖ ℐ_{ℓ_{1} + 1, ℓ_{2}} = {(α_{1}, \dots, α_{5}) \in ℕ^{5} :$	$α_{1} + 2 α_{2} + α_{3} = ℓ_{1}, α_{3} + α_{4} + 2 α_{5} = ℓ_{2},$
		$1 \leq α_{1} + α_{2} + \dots + α_{5} \leq \tilde{r} (\| 𝒜_{j} \|)} .$

Since $Q_{ℓ_{1}, ℓ_{2}}^{n} (j) / \tilde{D} (G_{n}, G_{0}) \to 0$ for all tuples $(j, ℓ_{1}, ℓ_{2}) \in 𝒬$ , we achieve that

	$\frac{S_{ℓ_{1}, ℓ_{2}}^{n} (j)}{\tilde{D} (G_{n}, G_{0})} := \frac{Q_{ℓ_{1}, ℓ_{2}}^{n} (j) - Q_{ℓ_{1} + 1, ℓ_{2}} (j)}{\tilde{D} (G_{n}, G_{0})}$
	$= \frac{\sum_{α \in 𝒥_{ℓ_{1}, ℓ_{2}}} \sum_{i \in 𝒜_{j}} \frac{π_{i}^{n} \cdot t_{ℓ_{1} - α_{3}, α_{1} + 2 α_{2}}}{2^{α_{2} + α_{5}} α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}} {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}}}{\tilde{D} (G_{n}, G_{0})}$
	$= \frac{\sum_{α \in 𝒥_{ℓ_{1}, ℓ_{2}}} \sum_{i \in 𝒜_{j}} \frac{π_{i}^{n}}{{(Γ_{j}^{0})}^{α_{1} + 2 α_{2}} 2^{α_{2} + α_{5}} α!} {(Δ c_{i j}^{n})}^{α_{1}} {(Δ Γ_{i j}^{n})}^{α_{2}} {(Δ a_{i j}^{n})}^{α_{3}} {(Δ b_{i j}^{n})}^{α_{4}} {(Δ ν_{i j}^{n})}^{α_{5}}}{\tilde{D} (G_{n}, G_{0})}$
	$\to 0,$

where the third inequality follows from the fact that $t_{ℓ_{1} - α_{3}, α_{1} + 2 α_{2}} = t_{α_{1} + 2 α_{2}, α_{1} + 2 α_{2}} = {(Γ_{j}^{0})}^{- (α_{1} + 2 α_{2})}$ . Additionally, we also let $S_{0, 0} (j) := Q_{0, 0} (j)$ for all $j \in [\tilde{k}]$ .

By assumption, $| S_{0, 0} (j) | / \tilde{D} (G_{n}, G_{0}) \to 0$ for all $j \in [\tilde{k}]$ and $| T_{0, 0, 0} (j) | / \tilde{D} (G_{n}, G_{0}) \to 0$ for all $\tilde{k} + 1 \leq j \leq k_{0}$ as $n \to \infty$ . By taking the summation of all such terms, we get that

\frac{1}{\tilde{D} ​ (G_{n}, G_{0})} \cdot \sum_{j = 1}^{k_{0}} | \sum_{i \in 𝒜_{j}} π_{i}^{n} - π_{j}^{0} | \to 0 .

(35)

Next, we consider indices $j \in [k_{0}] : | 𝒜_{j} | = 1$ , i.e. those in the formulation of $B_{n}$ . For $j \in [\tilde{k}]$ , since $| S_{ℓ_{1}, ℓ_{2}}^{n} (j) | / \tilde{D} (G_{n}, G_{0}) \to 0$ for all $(ℓ_{1}, ℓ_{2}) \in {(1, 0), (0, 1), (1, 1), (2, 0), (0, 2)}$ , we get that

\frac{1}{\tilde{D} ​ (G_{n}, G_{0})} \cdot \sum_{\begin{matrix} j : | 𝒜_{j} | = 1 \\ j \in [\tilde{k}] \end{matrix}} \sum_{i \in 𝒜_{j}} π_{i}^{n} ​ (| Δ ​ c_{i ​ j}^{n} | + | Δ ​ Γ_{i ​ j}^{n} | + | Δ ​ a_{i ​ j}^{n} | + | Δ ​ b_{i ​ j}^{n} | + | Δ ​ ν_{i ​ j}^{n} |) \to 0 .

(36)

Moreover, for $\tilde{k} + 1 \leq j \leq k_{0}$ , as $| T_{α_{3}, τ_{1}, τ_{2}}^{n} (j) | / \tilde{D} (G_{n}, G_{0}) \to 0$ for all $(α_{3}, τ_{1}, τ_{2}) \in {(0, 1, 0), (0, 2, 0), (1, 0, 0), (0, 0, 1), (0, 0, 2)}$ , we deduce that

\frac{1}{\tilde{D} ​ (G_{n}, G_{0})} \cdot \sum_{\begin{matrix} j : | 𝒜_{j} | = 1 \\ \tilde{k} + 1 \leq j \leq k_{0} \end{matrix}} \sum_{i \in 𝒜_{j}} π_{i}^{n} ​ (| Δ ​ c_{i ​ j}^{n} | + | Δ ​ Γ_{i ​ j}^{n} | + | Δ ​ a_{i ​ j}^{n} | + | Δ ​ b_{i ​ j}^{n} | + | Δ ​ ν_{i ​ j}^{n} |) \to 0 .

(37)

Let us denote

K_{i ​ j}^{n} ​ (κ_{1}, κ_{2}, κ_{3}, κ_{4}, κ_{5}) := {| Δ ​ c_{i ​ j}^{n} |}^{κ_{1}} + {| Δ ​ Γ_{i ​ j}^{n} |}^{κ_{2}} + {| Δ ​ a_{i ​ j}^{n} |}^{κ_{3}} + {| Δ ​ b_{i ​ j}^{n} |}^{κ_{4}} + {| Δ ​ ν_{i ​ j}^{n} |}^{κ_{5}} .

Then, equations (36) and (37) indicates that

\frac{1}{\tilde{D} ​ (G_{n}, G_{0})} \cdot \sum_{j : | 𝒜_{j} | = 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} ​ K_{i ​ j}^{n} ​ (1, 1, 1, 1, 1) \to 0 .

(38)

Additionally, since $| T_{2, 0, 0} (j) | / \tilde{D} (G_{n}, G_{0}) \to 0$ for all $\tilde{k} + 1 \leq j \leq k_{0}$ , we have that

\frac{1}{\tilde{D} ​ (G_{n}, G_{0})} \cdot \sum_{j : | 𝒜_{j} | > 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} ​ {| Δ ​ a_{i ​ j}^{n} |}^{2} \to 0 .

(39)

Putting the results in equations (35), (38) and (39) together with the formulation of $\tilde{D} (G_{n}, G_{0})$ in equation (3.2), we obtain that

	$\frac{1}{\tilde{D} (G_{n}, G_{0})}$	$[\sum_{\begin{matrix} j : \| 𝒜_{j} \| > 1 \\ j \in [\tilde{k}] \end{matrix}} \sum_{i \in 𝒜_{j}} π_{i}^{n} K_{i j}^{n} (\tilde{r} (\| 𝒜_{j} \|), \frac{\tilde{r} (\| 𝒜_{j} \|)}{2}, \frac{\tilde{r} (\| 𝒜_{j} \|)}{2}, \tilde{r} (\| 𝒜_{j} \|), \frac{\tilde{r} (\| 𝒜_{j} \|)}{2})$
		$+ \sum_{\begin{matrix} j : \| 𝒜_{j} \| > 1 \\ \tilde{k} + 1 \leq j \leq k_{0} \end{matrix}} \sum_{i \in 𝒜_{j}} π_{i}^{n} K_{- 3, i j}^{n} (\tilde{r} (\| 𝒜_{j} \|), \frac{\tilde{r} (\| 𝒜_{j} \|)}{2}, \tilde{r} (\| 𝒜_{j} \|), \frac{\tilde{r} (\| 𝒜_{j} \|)}{2})] \to 1,$		(40)

where $K_{- 3, i j}^{n} (κ_{1}, κ_{2}, κ_{3}, κ_{4}, κ_{5}) := {| Δ c_{i j}^{n} |}^{κ_{1}} + {| Δ Γ_{i j}^{n} |}^{κ_{2}} + {| Δ b_{i j}^{n} |}^{κ_{4}} + {| Δ ν_{i j}^{n} |}^{κ_{5}}$ . Now, we will divide our arguments into two main scenarios based on the above limit:

Case 1: $\frac{\sum_{\begin{matrix} j : | 𝒜_{j} | > 1 \\ j \in [\tilde{k}] \end{matrix}} \sum_{i \in 𝒜_{j}} π_{i}^{n} K_{i j}^{n} (\tilde{r} (| 𝒜_{j} |), \frac{\tilde{r} (| 𝒜_{j} |)}{2}, \frac{\tilde{r} (| 𝒜_{j} |)}{2}, \tilde{r} (| 𝒜_{j} |), \frac{\tilde{r} (| 𝒜_{j} |)}{2})}{\tilde{D} (G_{n}, G_{0})} \to̸ 0$ .

This assumption indicates that we can find an index $j^{*} \in [\tilde{k}] : | 𝒜_{j^{*}} | > 1$ such that

\frac{1}{\tilde{D} ​ (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{j^{*}}} π_{i}^{n} ​ K_{i ​ j^{*}}^{n} ​ (\tilde{r} ​ (| 𝒜_{j^{*}} |), \frac{\tilde{r} ​ (| 𝒜_{j^{*}} |)}{2}, \frac{\tilde{r} ​ (| 𝒜_{j^{*}} |)}{2}, \tilde{r} ​ (| 𝒜_{j^{*}} |), \frac{\tilde{r} ​ (| 𝒜_{j^{*}} |)}{2}) \to̸ 0 .

WLOG, we may assume that $j^{*} = 1$ throughout this case. Recall that $S_{ℓ_{1}, ℓ_{2}}^{n} (1) / \tilde{D} (G_{n}, G_{0}) \to 0$ for all pairs $(ℓ_{1}, ℓ_{2})$ such that $0 \leq ℓ_{1} + ℓ_{2} \leq 2 \tilde{r} (| 𝒜_{1} |)$ . Combine this result with the assumption of case 1, we obtain

\frac{S_{ℓ_{1}, ℓ_{2}}^{n} ​ (1)}{D_{1} ​ (G_{n}, G_{0})} = \frac{S_{ℓ_{1}, ℓ_{2}}^{n} ​ (1)}{\tilde{D} ​ (G_{n}, G_{0})} \cdot \frac{\tilde{D} ​ (G_{n}, G_{0})}{D_{1} ​ (G_{n}, G_{0})} \to 0,

where $D_{1} (G_{n}, G_{0}) := \sum_{i \in 𝒜_{1}} π_{i}^{n} K_{i 1}^{n} (\tilde{r} (| 𝒜_{1} |), \frac{\tilde{r} (| 𝒜_{1} |)}{2}, \frac{\tilde{r} (| 𝒜_{1} |)}{2}, \tilde{r} (| 𝒜_{1} |), \frac{\tilde{r} (| 𝒜_{1} |)}{2})$ . By expanding the formulations of $S_{ℓ_{1}, ℓ_{2}}^{n} (1)$ and $D_{1} (G_{n}, G_{0})$ , we have that

\frac{\sum_{i \in 𝒜_{1}} \sum_{α \in 𝒥_{ℓ_{1}, ℓ_{2}}} \frac{π_{i}^{n}}{{(Γ_{1}^{0})}^{α_{1} + 2 ​ α_{2}} ​ 2^{α_{2} + α_{5}} ​ α!} ​ {(Δ ​ c_{i ​ 1}^{n})}^{α_{1}} ​ {(Δ ​ Γ_{i ​ 1}^{n})}^{α_{2}} ​ {(Δ ​ a_{i ​ 1}^{n})}^{α_{3}} ​ {(Δ ​ b_{i ​ 1}^{n})}^{α_{4}} ​ {(Δ ​ ν_{i ​ 1}^{n})}^{α_{5}}}{\sum_{i \in 𝒜_{1}} π_{i}^{n} ​ [{| Δ ​ c_{i ​ 1}^{n} |}^{\tilde{r} ​ (| 𝒜_{1} |)} + {| Δ ​ Γ_{i ​ 1}^{n} |}^{\frac{\tilde{r} ​ (| 𝒜_{1} |)}{2}} + {| Δ ​ a_{i ​ 1}^{n} |}^{\frac{\tilde{r} ​ (| 𝒜_{1} |)}{2}} + {| Δ ​ b_{i ​ 1}^{n} |}^{\tilde{r} ​ (| 𝒜_{1} |)} + {| Δ ​ ν_{i ​ 1}^{n} |}^{\frac{\tilde{r} ​ (| 𝒜_{1} |)}{2}}]} \to 0 .

(41)

Next, we define ${\bar{M}}_{n} := \max {| Δ c_{i 1}^{n} |, {| Δ Γ_{i 1}^{n} |}^{1 / 2}, {| Δ a_{i 1}^{n} |}^{1 / 2}, | Δ b_{i 1}^{n} |, {| Δ ν_{i 1}^{n} |}^{1 / 2} : i \in 𝒜_{1}}$ and ${\bar{π}}_{n} := \max_{i \in 𝒜_{1}} π_{i}^{n}$ . For any $i \in 𝒜_{1}$ , since the sequence ${(π_{i}^{n} / {\bar{π}}_{n})}_{i \in 𝒜_{1}}$ is bounded, we can substitute it with its subsequence that admits a non-negative limit $p_{i}^{2} = {lim}_{n \to \infty} π_{i}^{n} / {\bar{π}}_{n}$ .

Additionally, we define $(Δ c_{i 1}^{n}) / (Γ_{j}^{0} \cdot {\bar{M}}_{n}) \to q_{1 i}$ , $(Δ Γ_{i 1}^{n}) / [2 {(Γ_{j}^{0})}^{2} \cdot {\bar{M}}_{n}^{2}] \to q_{2 i}$ , $(Δ a_{i 1}^{n}) / {\bar{M}}_{n}^{2} \to q_{3 i}$ , $(Δ b_{i 1}^{n}) / {\bar{M}}_{n} \to q_{4 i}$ and $(Δ ν_{i 1}^{n}) / 2 {\bar{M}}_{n}^{2} \to q_{5 i}$ . It can be seen from the formulation of $𝒪_{k, β} (Θ)$ that $π_{i}^{n} \geq δ$ , therefore, $p_{i}$ ’s will not vanish and at least one of them is equal to 1. Similarly, at least one of the limits $q_{1 i}, q_{2 i}, \dots, q_{5 i}$ will be equal to either $1$ or $- 1$ .

Since

\frac{\sum_{i \in 𝒜_{1}} π_{i}^{n} ​ [{| Δ ​ c_{i ​ 1}^{n} |}^{\tilde{r} ​ (| 𝒜_{1} |)} + {| Δ ​ Γ_{i ​ 1}^{n} |}^{\frac{\tilde{r} ​ (| 𝒜_{1} |)}{2}} + {| Δ ​ a_{i ​ 1}^{n} |}^{\frac{\tilde{r} ​ (| 𝒜_{1} |)}{2}} + {| Δ ​ b_{i ​ 1}^{n} |}^{\tilde{r} ​ (| 𝒜_{1} |)} + {| Δ ​ ν_{i ​ 1}^{n} |}^{\frac{\tilde{r} ​ (| 𝒜_{1} |)}{2}}]}{{\bar{π}}_{n} ​ {\bar{M}}_{n}^{ℓ_{1} + ℓ_{2}}} \to̸ 0,

for all pairs $(ℓ_{1}, ℓ_{2})$ such that $ℓ_{1} + ℓ_{2} \in [\tilde{r} (| 𝒜_{1} |)]$ , we can divide both the numerator and the denominator in equation (41) by ${\bar{π}}_{n} {\bar{M}}_{n}^{ℓ_{1} + ℓ_{2}}$ , and then let $n \to \infty$ to achieve the following system of polynomial equations:

\sum_{i \in 𝒜_{1}} \sum_{α \in 𝒥_{ℓ_{1}, ℓ_{2}}} \frac{p_{i}^{2} ​ q_{1 ​ i}^{α_{1}} ​ q_{2 ​ i}^{α_{2}} ​ q_{3 ​ i}^{α_{3}} ​ q_{4 ​ i}^{α_{4}} ​ q_{5 ​ i}^{α_{5}}}{α_{1}! ​ α_{2}! ​ α_{3}! ​ α_{4}! ​ α_{5}!} = 0,

for all pairs $(ℓ_{1}, ℓ_{2})$ such that $0 \leq ℓ_{1} + ℓ_{2} \leq \tilde{r} (| 𝒜_{1} |)$ . Nevertheless, according to the definition of $\tilde{r} (| 𝒜_{1} |)$ , the above system cannot admit any non-trivial solutions, which is a contradiction. Thus, case 1 does not hold.

Case 2: $\frac{1}{\tilde{D} (G_{n}, G_{0})} \cdot \sum_{\begin{matrix} j : | 𝒜_{j} | > 1 \\ \tilde{k} + 1 \leq j \leq k_{0} \end{matrix}} \sum_{i \in 𝒜_{j}} π_{i}^{n} K_{- 3, i j}^{n} (\tilde{r} (| 𝒜_{j} |), \frac{\tilde{r} (| 𝒜_{j} |)}{2}, \tilde{r} (| 𝒜_{j} |), \frac{\tilde{r} (| 𝒜_{j} |)}{2}) \to̸ 0$ .

This assumption implies that there exists an index $\tilde{k} + 1 \leq j^{*} \leq k_{0} : | 𝒜_{j^{*}} | > 1$ such that

\frac{1}{\tilde{D} ​ (G_{n}, G_{0})} \cdot \sum_{i \in 𝒜_{j^{*}}} π_{i}^{n} ​ K_{- 3, i ​ j^{*}}^{n} ​ (\tilde{r} ​ (| 𝒜_{j^{*}} |), \frac{\tilde{r} ​ (| 𝒜_{j^{*}} |)}{2}, \tilde{r} ​ (| 𝒜_{j^{*}} |), \frac{\tilde{r} ​ (| 𝒜_{j^{*}} |)}{2}) \to̸ 0 .

(42)

By applying similar arguments for equation (22) in the proof of Theorem 1 to equation (42), we are able to point out that equation (42) cannot happen, which is a contradiction. As a result, case 2 cannot happen either.

Collect the results of the above two scenarios, we realize that the limit in equation (40) does not hold true, which is a contradiction. As a consequence, not all the coefficients of elements in the set $ℋ$ , defined in equation (B.2), in the representations of $A_{n} / \tilde{D} (G_{n}, G_{0})$ , $B_{n} / \tilde{D} (G_{n}, G_{0})$ and $E_{n} / \tilde{D} (G_{n}, G_{0})$ go to zero as $n \to \infty$ .

Step 3 - Application of Fatou’s lemma: Next, we denote by $m_{n}$ the maximum of the absolute values of those coefficients, which means that

m_{n} := \max {\max_{\begin{matrix} (j, ℓ_{1}, ℓ_{2}) \in 𝒬 \end{matrix}} \frac{| Q_{ℓ_{1}, ℓ_{2}}^{n} ​ (j) |}{\tilde{D} ​ (G_{n}, G_{0})}, \max_{\begin{matrix} (j, α_{3}, τ_{1}, τ_{2}) \in 𝒯 \end{matrix}} \frac{| T_{α_{3}, τ_{1}, τ_{2}}^{n} ​ (j) |}{\tilde{D} ​ (G_{n}, G_{0})}} .

In addition, let us define $Q_{ℓ_{1}, ℓ_{2}}^{n} (j) / m_{n} \to ζ_{ℓ_{1}, ℓ_{2}} (j)$ for $(j, ℓ_{1}, ℓ_{2}) \in 𝒬$ and $T_{α_{3}, τ_{1}, τ_{2}}^{n} (j) \to ξ_{α_{3}, τ_{1}, τ_{2}} (j)$ for $(j, α_{3}, τ_{1}, τ_{2}) \in 𝒯$ as $n \to \infty$ . As not all the coefficients of elements of $ℋ$ in the representations of $A_{n} / \tilde{D} (G_{n}, G_{0})$ , $B_{n} / \tilde{D} (G_{n}, G_{0})$ and $E_{n} / \tilde{D} (G_{n}, G_{0})$ vanish as $n \to \infty$ , at least one among $ζ_{ℓ_{1}, ℓ_{2}} (j)$ and $ξ_{α_{3}, τ_{1}, τ_{2}} (j^{'})$ is different from zero and $m_{n} \to̸ 0$ . By invoking the Fatou’s lemma, we get that

0 = lim_{n \to \infty} \frac{1}{m_{n}} \cdot \frac{2 ​ V ​ (p_{G_{n}}, p_{G})}{\tilde{D} ​ (G_{n}, G_{0})} \geq \int \underset{n \to \infty}{lim inf} \frac{1}{m_{n}} \cdot \frac{| p_{G_{n}} ​ (X, Y) - p_{G} ​ (X, Y) |}{\tilde{D} ​ (G_{n}, G_{0})} ​ d ​ (X, Y) \geq 0 .

Furthermore, we have that

	$\frac{1}{m_{n}} \cdot$	$\frac{p_{G_{n}} (X, Y) - p_{G} (X, Y)}{\tilde{D} (G_{n}, G_{0})}$
		$\to \sum_{(j, ℓ_{1}, ℓ_{2}) \in 𝒬} ζ_{ℓ_{1}, ℓ_{2}} (j) X^{ℓ_{1}} \cdot \frac{\partial^{ℓ_{2}} f_{𝒟}}{\partial h_{1}^{ℓ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) \cdot f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0})$
		$+ \sum_{(j, α_{3}, τ_{1}, τ_{2}) \in 𝒯} ξ_{α_{3}, τ_{1}, τ_{2}} (j) X^{α_{3}} \cdot \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \cdot \frac{\partial^{α_{3} + τ_{2}} f_{𝒟}}{\partial h_{1}^{α_{3} + τ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) .$

Consequently, we achieve that

	$\sum_{(j, ℓ_{1}, ℓ_{2}) \in 𝒬} ζ_{ℓ_{1}, ℓ_{2}} (j) X^{ℓ_{1}} \cdot \frac{\partial^{ℓ_{2}} f_{𝒟}}{\partial h_{1}^{ℓ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) \cdot f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0})$
	$+ \sum_{(j, α_{3}, τ_{1}, τ_{2}) \in 𝒯} ξ_{α_{3}, τ_{1}, τ_{2}} (j) X^{α_{3}} \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \frac{\partial^{α_{3} + τ_{2}} f_{𝒟}}{\partial h_{1}^{α_{3} + τ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) = 0,$

for almost surely $(X, Y)$ . Since elements of the set $ℋ$ defined in equation (B.2) are linearly independent (proof of this claim is deferred to the end of this proof), the above equation indicates that $ζ_{j, ℓ_{1}, ℓ_{2}} (j) = ξ_{α_{3}, τ_{1}, τ_{2}} (j^{'}) = 0$ for all $(j, ℓ_{1}, ℓ_{2}) \in 𝒬$ and $(j^{'}, α_{3}, τ_{1}, τ_{2}) \in 𝒯$ , which contradicts the fact that at least one among $ζ_{j, ℓ_{1}, ℓ_{2}} (j)$ , $ξ_{α_{3}, τ_{1}, τ_{2}} (j^{'})$ is different from zero. Hence, we reach the conclusion in equation (31).

Linear independence of elements in $ℋ$ : For completion, we will show that elements of the set $ℱ$ defined in equation (B.2) are linearly independent by definition. In particular, assume that there exist real numbers $ζ_{ℓ_{1}, ℓ_{2}} (j)$ and $ξ_{α_{3}, τ_{1}, τ_{2}} (j^{'})$ , where $(j, ℓ_{1}, ℓ_{2}) \in 𝒬$ and $(j^{'}, α_{3}, τ_{1}, τ_{2}) \in 𝒯$ , such that the following equation holds for almost surely $(X, Y)$ :

	$\sum_{(j, ℓ_{1}, ℓ_{2}) \in 𝒬} ζ_{ℓ_{1}, ℓ_{2}} (j) X^{ℓ_{1}} \cdot \frac{\partial^{ℓ_{2}} f_{𝒟}}{\partial h_{1}^{ℓ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) \cdot f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0})$
	$+ \sum_{(j^{'}, α_{3}, τ_{1}, τ_{2}) \in 𝒯} ξ_{α_{3}, τ_{1}, τ_{2}} (j^{'}) X^{α_{3}} \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j^{'}}^{0}, Γ_{j^{'}}^{0}) \frac{\partial^{α_{3} + τ_{2}} f_{𝒟}}{\partial h_{1}^{α_{3} + τ_{2}}} (Y \| a_{j^{'}}^{0} X + b_{j^{'}}^{0}, ν_{j^{'}}^{0}) = 0,$

Now, we rewrite the above equation as follows:

		$\sum_{j = 1}^{k_{0}} \sum_{ω = 0}^{2 \tilde{r} (\| 𝒜_{j} \|)} [\sum_{α_{3} + τ_{2} = ω} \sum_{τ_{1} = 0}^{2 (\tilde{r} (\| 𝒜_{j} \|) - α_{3}) - τ_{2}} ξ_{α_{3}, τ_{1}, τ_{2}} (j) X^{α_{3}} \cdot \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \cdot 𝟏_{{\tilde{k} + 1 \leq j \leq k_{0}}}$
		$+ \sum_{ℓ_{1} = 0}^{2 \tilde{r} (\| 𝒜_{j} \|) - ω} ζ_{ℓ_{1}, ω} (j) X^{ℓ_{1}} \cdot f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) \cdot 𝟏_{{j \in [\tilde{k}]}}] \frac{\partial^{ω} f_{𝒟}}{\partial h_{1}^{ω}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) = 0,$		(43)

for almost surely $(X, Y)$ . As $(a_{j}^{0}, b_{j}^{0}, ν_{j}^{0})$ for $j \in [k_{0}]$ are $k_{0}$ distinct tuples, we deduce that $({(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0})$ for $j \in [k_{0}]$ are also $k_{0}$ distinct tuples for almost surely $X$ . Thus, for almost surely $X$ , one has $\frac{\partial^{ω} f_{𝒟}}{\partial h_{1}^{ω}} (Y | {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0})$ for $j \in [k_{0}]$ and $0 \leq ω \leq 2 \tilde{r} (| 𝒜_{j} |)$ are linearly independent with respect to $Y$ . Given that result, the equation (B.2) indicates that for almost surely $X$ ,

	$\sum_{α_{3} + τ_{2} = ω} \sum_{τ_{1} = 0}^{2 (\tilde{r} (\| 𝒜_{j} \|) - α_{3}) - τ_{2}} ξ_{α_{3}, τ_{1}, τ_{2}} (j) X^{α_{3}} \cdot \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \cdot 𝟏_{{\tilde{k} + 1 \leq j \leq k_{0}}}$
	$+ \sum_{ℓ_{1} = 0}^{2 \tilde{r} (\| 𝒜_{j} \|) - ω} ζ_{ℓ_{1}, ω} (j) X^{ℓ_{1}} \cdot f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) \cdot 𝟏_{{j \in [\tilde{k}]}} = 0 .$

for all $j \in [k_{0}]$ and $0 \leq ω \leq 2 \tilde{r} (| 𝒜_{j} |)$ . This equation is equivalent to

	$\sum_{ℓ_{1} = 0}^{2 \tilde{r} (\| 𝒜_{j} \|) - ω} ζ_{ℓ_{1}, ω} (j) X^{ℓ_{1}} \cdot f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0})$	$= 0,$		(44)
	$\sum_{α_{3} + τ_{2} = ω^{'}} \sum_{τ_{1} = 0}^{2 (\tilde{r} (\| 𝒜_{j^{'}} \|) - α_{3}) - τ_{2}} ξ_{α_{3}, τ_{1}, τ_{2}} (j^{'}) X^{α_{3}} \cdot \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j^{'}}^{0}, Γ_{j^{'}}^{0})$	$= 0,$		(45)

for all $j \in [\tilde{k}]$ , $0 \leq ω \leq 2 \tilde{r} (| 𝒜_{j} |)$ and $\tilde{k} + 1 \leq j^{^{'}} \leq k_{0}$ , $0 \leq ω \leq 2 \tilde{r} (| 𝒜_{j^{'}} |)$ . We can treat the left hand side of equation (44) as a polynomial of the random vector $X \in 𝒳$ , which is a compact set in $ℝ$ . Meanwhile, the left hand side of equation (45) can be viewed as another polynomial of $X$ and $X - c_{j^{'}}^{0}$ , where $c_{j^{'}}^{0} \neq 0$ . As a result, the above equations hold when $ζ_{ℓ_{1}, ω} (j) = 0$ for all $j \in [\tilde{k}]$ , $0 \leq ω \leq 2 \tilde{r} (| 𝒜_{j} |)$ , $0 \leq ℓ_{1} \leq 2 \tilde{r} (| 𝒜_{j} |) - ω$ , and $ξ_{α_{3}, τ_{1}, τ_{2}} (j {}^{'}) = 0$ for all $\tilde{k} + 1 \leq j^{'} \leq k_{0}$ , $0 \leq ω^{'} \leq 2 \tilde{r} (| 𝒜_{j^{'}} |)$ , $α_{3} + τ_{2} = ω^{'}$ and $0 \leq τ_{1} \leq 2 (\tilde{r} (| 𝒜_{j} |) - α_{3}) - τ_{2}$ . This result is equivalent to $ζ_{ℓ_{1}, ℓ_{2}} (j) = 0$ , for all $(j, ℓ_{1}, ℓ_{2}) \in 𝒬$ and $ξ_{α_{3}, τ_{1}, τ_{2}} (j^{'}) = 0$ for all $(j^{'}, α_{3}, τ_{1}, τ_{2}) \in 𝒯$ .

Hence, the elements of $ℋ$ are linearly independent, which completes the proof.

Appendix C PROOF OF REMAINING RESULTS

In this appendix, we provide proofs for Proposition 1, Proposition 2 and Lemma 2 in that order.

C.1 Proof of Proposition 1

For any two mixing measures $G = \sum_{i = 1}^{k} π_{i} δ_{(c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i})}$ and $G^{'} = \sum_{i = 1}^{k^{'}} π_{i}^{'} δ_{(c_{i}^{'}, Γ_{i}^{'}, a_{i}^{'}, b_{i}^{'}, ν_{i}^{'})}$ , we assume that $p_{G} (X, Y) = p_{G^{'}} (X, Y)$ holds true for almost surely $(X, Y) \in 𝒳 \times 𝒴$ , or equivalently,

\sum_{i = 1}^{k} π_{i} ​ f_{ℒ} ​ (X | c_{i}, Γ_{i}) ​ f_{𝒟} ​ (Y | {(a_{i})}^{⊤} ​ X + b_{i}, ν_{i}) = \sum_{i = 1}^{k^{'}} π_{i}^{'} ​ f_{ℒ} ​ (X | c_{i}^{'}, Γ_{i}^{'}) ​ f_{𝒟} ​ (Y | {(a_{i}^{'})}^{⊤} ​ X + b_{i}^{'}, ν_{i}^{'}) .

(46)

Recall that if $Y | X \sim 𝒩_{1} (a^{⊤} X + b, ν)$ and $X \sim 𝒩_{d} (c, Γ)$ , then

(\begin{matrix} X \\ Y \end{matrix}) \sim 𝒩_{d + 1} ​ ((\begin{matrix} c \\ a^{⊤} ​ c + b \end{matrix}), (\begin{matrix} Γ & Γ ​ a \\ a^{⊤} ​ Γ & a^{⊤} ​ Γ ​ a + ν \end{matrix})) .

Let us denote

	$ψ_{i} := (\begin{matrix} c_{i} \\ {(a_{i})}^{⊤} c_{i} + b_{i} \end{matrix}), Σ_{i} := (\begin{matrix} Γ_{i} & Γ_{i} a_{i} \\ {(a_{i})}^{⊤} Γ_{i} & {(a_{i})}^{⊤} Γ_{i} a_{i} + ν_{i} \end{matrix}),$
	$ψ_{i}^{'} := (\begin{matrix} c_{i}^{'} \\ {(a_{i}^{'})}^{⊤} c_{i}^{'} + b_{i}^{'} \end{matrix}), Σ_{i}^{'} := (\begin{matrix} Γ_{i}^{'} & Γ_{i}^{'} a_{i}^{'} \\ {(a_{i}^{'})}^{⊤} Γ_{i}^{'} & {(a_{i})}^{⊤} Γ_{i}^{'} a_{i}^{'} + ν_{i}^{'} \end{matrix}) .$

Then, equation (46) can be rewritten as

\sum_{i = 1}^{k} π_{i} ​ f ​ (X, Y | ψ_{i}, Σ_{i}) = \sum_{i = 1}^{k^{'}} π_{i}^{'} ​ f ​ (X, Y | ψ_{i}^{'}, Σ_{i}^{'}),

(47)

for almost surely $(X, Y)$ , where $f$ belongs to the family of $(d + 1)$ -dimensional Gaussian density functions. Since the location-scale Gaussian mixtures are identifiable, it follows from the above equation that $k = k^{'}$ and ${π_{1}, π_{2}, \dots, π_{k}} \equiv {π_{1}^{'}, π_{2}^{'}, \dots, π_{k}^{'}}$ . WLOG, we may assume that $π_{i} = π_{i}^{'}$ for any $i \in [k]$ .

Subsequently, we construct a partition of the set $[k]$ , denoted by $P_{1}, P_{2}, \dots, P_{m}$ that satisfies the following properties:

(i)

$π_{i} = π_{i}^{'}$ for any $i \in P_{ℓ}$ and $ℓ \in [m]$ ;
(ii)

$π_{i} \neq π_{j}^{'}$ if $i$ and $j$ are not in the same set $P_{ℓ}$ for any $ℓ \in [m]$ .

Given this partition, we represent equation (47) as follows:

\sum_{ℓ = 1}^{m} \sum_{i \in P_{ℓ}} π_{i} ​ f ​ (X, Y | ψ_{i}, Σ_{i}) = \sum_{ℓ = 1}^{m} \sum_{i \in P_{ℓ}} π_{i}^{'} ​ f ​ (X, Y | ψ_{i}^{'}, Σ_{i}^{'}),

for almost surely $(X, Y)$ . Consequently, for each $ℓ \in [m]$ , we obtain that

{(ψ_{i}, Σ_{i}) : i \in P_{ℓ}} \equiv {(ψ_{i}^{'}, Σ_{i}^{'}) : i \in P_{ℓ}} .

WLOG, we may assume that $(ψ_{i}, Σ_{i}) = (ψ_{i}^{'}, Σ_{i}^{'})$ for any $i \in P_{ℓ}$ . Given this result, by some simple algebraic derivations, we achieve that $(c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i}) = (c_{i}^{'}, Γ_{i}^{'}, a_{i}^{'}, b_{i}^{'}, ν_{i}^{'})$ for any $i \in P_{ℓ}$ and $ℓ \in [m]$ . As a result, it follows that

G = \sum_{ℓ = 1}^{m} \sum_{i \in P_{ℓ}} π_{i} ​ δ_{(c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i})} = \sum_{ℓ = 1}^{m} \sum_{i \in P_{ℓ}} π_{i}^{'} ​ δ_{(c_{i}^{'}, Γ_{i}^{'}, a_{i}^{'}, b_{i}^{'}, ν_{i}^{'})} = G^{'} .

Hence, the proof is completed.

C.2 Proof of Proposition 2

Prior to presenting the proof of Proposition 2, let us review fundamental background on density estimation for M-estimators, which is covered in [53]. First of all, we define $𝒫_{k, β} (Θ) := {p_{G} (X, Y) : G \in 𝒪_{k, β} (Θ)}$ as the set of joint densities of all mixing measure in $𝒪_{k, β} (Θ)$ . In addition, we denote

	$𝒬_{k, β} (Θ)$	$:= {p_{(G + G_{0}) / 2} (X, Y) : G \in 𝒪_{k, β} (Θ)},$
	$𝒬_{k, β}^{1 / 2} (Θ)$	$:= {p_{(G + G_{0}) / 2}^{1 / 2} (X, Y) : G \in 𝒪_{k, β} (Θ)} .$

Subsequently, for any $δ > 0$ , the Hellinger ball centered around the density $p_{G_{0}} (X, Y)$ and intersected with the set $𝒬_{k, β}^{1 / 2} (Θ)$ is defined as

𝒬_{k, β}^{1 / 2} ​ (Θ, δ) := {g^{1 / 2} \in 𝒬_{k, β}^{1 / 2} ​ (Θ) : h ​ (g, p_{G_{0}}) \leq δ} .

Additionally, Geer et al. [53] introduce the following quantity to capture the size of the above Hellinger ball:

𝒥_{B} (δ, 𝒬_{k, β}^{1 / 2} (Θ)) := \int_{δ^{2} / 2^{13}}^{δ} H_{B}^{1 / 2} (u, 𝒬_{k, β}^{1 / 2} (Θ, u), ∥ \cdot ∥) d u \lor δ,

(48)

where $H_{B}^{1 / 2} (u, 𝒬_{k, β}^{1 / 2} (Θ, u), ∥ \cdot ∥)$ denotes the bracketing entropy of $𝒬_{k, β}^{1 / 2} (Θ, u)$ under the Euclidean distance, and $u \lor δ := \max {u, δ}$ . Given these notations, let us state the result regarding the joint density estimation rate presented in Theorem 7.4 in [53].

Lemma 3 (Theorem 7.4, [53]).

Take $Ψ (δ) \geq 𝒥_{B} (δ, 𝒬_{k, β}^{1 / 2} (Θ))$ such that $Ψ (δ) / δ^{2}$ is a non-increasing function of $δ$ . Then, for a universal constant $c$ and a sequence $(δ_{n})$ that satisfies $\sqrt{n} δ_{n}^{2} \geq c Ψ (δ_{n})$ , we obtain that

ℙ ​ (h ​ (p_{{\hat{G}}_{n}}, p_{G_{0}}) > δ) \leq c ​ \exp (- \frac{n ​ δ^{2}}{c^{2}}),

for any $δ \geq δ_{n}$ .

Proof of Lemma 3 is provided in [53]. Next, we introduce the upper bounds of the covering number (under the sup norm) $N (ε, 𝒫_{k, β} (Θ), ∥ \cdot ∥_{\infty})$ , and the bracketing entropy (under the Hellinger distance) $H_{B} (ε, 𝒫_{k, β} (Θ), h)$ of the metric space $𝒫_{k, β} (Θ)$ . For further detail about the definitions of these terms, readers are referred to [53].

Lemma 4.

Given a bounded set $Θ$ , we have for any $ε \in [0, 1 / 2]$ that

(i)

$\log N (ε, 𝒫_{k, β} (Θ), ∥ \cdot ∥_{\infty}) ≲ \log (1 / ε)$ ;
(ii)

$H_{B} (ε, 𝒫_{k, β} (Θ), h) ≲ \log (1 / ε)$ .

Proof of Lemma 4 is relegated to Appendix C.2.2. Now, we already have all necessary ingredients to provide the proof for Proposition 2 in Appendix C.2.1

C.2.1 Proof of Proposition 2

Note that for any $u > 0$ , we have

H_{B} (u, 𝒬_{k, β}^{1 / 2} (Θ), ∥ \cdot ∥) \leq H_{B} (u, 𝒫_{k, β} (Θ), h) \leq \log (1 / u),

where the second inequality is induced by part (ii) of Lemma 4. Then, it follows from equation (48) that

𝒥_{B} ​ (δ, 𝒬_{k, β}^{1 / 2} ​ (Θ)) \leq \int_{δ^{2} / 2^{13}}^{δ} \log (1 / u) ​ d u \lor δ .

(49)

By choosing $Ψ (δ) := δ \cdot {[\log (1 / δ)]}^{1 / 2}$ , we get that $Ψ (δ) / δ^{2}$ is a non-increasing function of $δ$ and $Ψ (δ) \geq 𝒥_{B} (δ, 𝒬_{k, β}^{1 / 2} (Θ))$ from equation (49). Let $δ_{n} := \sqrt{\log (n) / n}$ , we achieve that $\sqrt{n} δ_{n}^{2} \geq c Ψ (δ_{n})$ for some universal constant $c$ . As a result, Lemma 3 gives us that

ℙ ​ (h ​ (p_{{\hat{G}}_{n}}, p_{G_{0}}) > C_{1} ​ \sqrt{\log (n) / n}) ≲ \exp (- C_{2} ​ \log (n)) = n^{- C_{2}},

where $C_{1}$ and $C_{2}$ are some universal constants. Finally, since the Total Variation is upper bounded by the Hellinger distance, we obtain the desired conclusion.

C.2.2 Proof of Lemma 4

Part (i). Given some $ε > 0$ , since $Θ$ is a compact set, we can find an $ε$ -cover of $Θ$ , denoted by $Θ_{ε}$ . Additionally, let $Δ_{ε}$ be an $ε$ -cover of an $(k - 1)$ -dimensional simplex. Assume that $| Θ_{ε} | = T$ and $| Δ_{ε} | = S$ . Note that $Θ \subset ℝ^{d} \times 𝒮_{d}^{+} \times ℝ^{d} \times ℝ \times ℝ_{+}$ is a subspace of $ℝ^{d^{2} + 4 d}$ , then it can be checked that $T = 𝒪 (ε^{- (d^{2} + 4 d) k})$ and $S = 𝒪 (ε^{- (k - 1)})$ . Next, we define

𝒢 := {p_{G} \in 𝒫_{k, β} ​ (Θ) : (π_{1}, π_{2}, \dots, π_{k}) \in Δ_{ε}, (c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i}) \in Θ_{ε}} .

Given some mixing measure $G = \sum_{i = 1}^{k^{'}} π_{i} δ_{θ_{i}} \in 𝒪_{k, β} (Θ)$ with $k^{'} \leq k$ and $θ_{i} := (c_{i}, Γ_{i}, a_{i}, b_{i}, ν_{i}) \in Θ$ , let us consider $\bar{G} = \sum_{i = 1}^{k^{'}} π_{i} δ_{{\tilde{θ}}_{i}}$ where ${\tilde{θ}}_{i} := ({\tilde{c}}_{i}, {\tilde{Γ}}_{i}, {\tilde{a}}_{i}, {\tilde{b}}_{i}, {\tilde{ν}}_{i}) \in Θ_{ε}$ such that $‖ {\tilde{θ}}_{i} - θ_{i} ‖ \leq ε$ for any $i \in [k^{'}]$ . In addition, we also take into account another mixing measure $\tilde{G} = \sum_{i = 1}^{k^{'}} {\tilde{π}}_{i} δ_{{\tilde{θ}}_{i}}$ where $({\tilde{π}}_{1}, {\tilde{π}}_{2}, \dots, {\tilde{π}}_{k^{'}}, 0, \dots, 0) \in Δ_{ε}$ such that $‖ {({\tilde{π}}_{i})}_{i = 1}^{k^{'}} - {(π_{i})}_{i = 1}^{k^{'}} ‖ \leq ε$ . From the definition of $𝒢$ , we get that $p_{\tilde{G}} \in 𝒢$ . Since $‖ {({\tilde{π}}_{i})}_{i = 1}^{k^{'}} - {(π_{i})}_{i = 1}^{k^{'}} ‖ \leq ε$ , we can deduce that

∥ p_{\bar{G}} - p_{\tilde{G}} ∥_{\infty} \leq \sum_{i = 1}^{k^{'}} | {\tilde{π}}_{i} - π_{i} | \cdot ∥ f_{ℒ} (X | {\tilde{c}}_{i}, {\tilde{Γ}}_{i}) f_{𝒟} (Y | {({\tilde{a}}_{i})}^{⊤} X + {\tilde{b}}_{i}, {\tilde{ν}}_{i}) ∥_{\infty} ≲ ε .

Next, we consider

∥ p_{G} - p_{\bar{G}} ∥_{\infty} \leq \sum_{i = 1}^{k^{'}} π_{i} ∥ F (θ_{i} | X, Y) - F ({\tilde{θ}}_{i} | X, Y) ∥_{\infty},

where we denote $F (θ | X, Y) := f_{ℒ} (X | c, Γ) f_{𝒟} (Y | a^{⊤} X + b, ν)$ . As $F$ is twice differentiable with respect to $θ$ and $𝒳$ is a bounded set, we achieve the following inequality:

\sum_{i = 1}^{k^{'}} π_{i} ∥ F (θ_{i} | X, Y) - F ({\tilde{θ}}_{i} | X, Y) ∥_{\infty} \leq \sum_{i = 1}^{k^{'}} π ∥ {\tilde{θ}}_{i} - θ_{i} ∥ ≲ ε,

which leads to ${‖ p_{G} - p_{\bar{G}} ‖}_{\infty} \leq ε$ . As a consequence, by the triangle inequality, we have

{‖ p_{G} - p_{\tilde{G}} ‖}_{\infty} \leq {‖ p_{G} - p_{\bar{G}} ‖}_{\infty} + {‖ p_{\bar{G}} - p_{\tilde{G}} ‖}_{\infty} ≲ ε .

Given this result, it follows that $𝒢$ is an $ε$ -cover of $𝒫_{k, β} (Θ)$ , therefore,

N (ε, 𝒫_{k, β} (Θ), ∥ \cdot ∥_{\infty}) \leq | 𝒢 | = S \times T = 𝒪 (ε^{- (d^{2} + 4 ​ d) ​ k}) \times 𝒪 (ε^{- (k - 1)}) = 𝒪 (ε^{- (d^{2} + 4 ​ d + 1) ​ k + 1}),

which implies that $\log N (ε, 𝒫_{k, β} (Θ), ∥ \cdot ∥_{\infty}) ≲ \log (1 / ε)$ .

Part (ii). We begin with finding an upper bound for the density $f_{ℒ} (X | c, Γ) f_{𝒟} (Y | a^{⊤} X + b, ν)$ . Since $𝒳$ and $Θ$ are bounded sets, we can find positive constants $u, u_{1}, u_{2}, u_{3}, l_{1}, l_{3}$ such that $‖ c ‖ \leq u$ , $l_{1} \leq λ_{\min} (Γ) \leq λ_{\max} (Γ) \leq u_{1}$ , $- u_{2} \leq a^{⊤} X + b \leq u_{2}$ and $l_{3} \leq ν \leq u_{3}$ , where $λ_{\min} (Γ)$ and $λ_{\max} (Γ)$ are the smallest and the largest eigenvalues of $Γ$ , respectively. Firstly, it is clear that

f_{ℒ} ​ (X | c, Γ) = \frac{1}{\sqrt{{(2 ​ π)}^{d} ​ det (Γ)}} ​ \exp (- \frac{1}{2} ​ {(x - c)}^{⊤} ​ Γ^{- 1} ​ (x - c)) \leq \frac{1}{{(2 ​ π ​ l_{1})}^{d / 2}} .

Additionally, note that

{(X - c)}^{⊤} ​ Γ^{- 1} ​ (x - c) \geq λ_{\min} ​ (Γ^{- 1}) ​ {‖ X - c ‖}^{2} = \frac{1}{λ_{\max} ​ (Γ)} ​ {‖ X - c ‖}^{2} .

Moreover, for any $‖ X ‖ \geq 2 u$ , by the Cauchy-Schwartz inequality, we get

4 ​ {‖ X - c ‖}^{2} - {‖ X ‖}^{2} = 3 ​ {‖ X ‖}^{2} - 8 ​ X^{⊤} ​ c + 4 ​ {‖ c ‖}^{2} \geq 3 ​ {‖ X ‖}^{2} - 8 ​ ‖ X ‖ \cdot ‖ c ‖ + 4 ​ {‖ c ‖}^{2} \geq 0,

which implies that ${(X - c)}^{⊤} Γ^{- 1} (x - c) \geq \frac{1}{4 u_{1}} {‖ X ‖}^{2}$ . As a result,

f_{ℒ} ​ (X | c, Γ) = \frac{1}{\sqrt{{(2 ​ π)}^{d} ​ det (Γ)}} ​ \exp (- \frac{1}{2} ​ {(x - c)}^{⊤} ​ Γ^{- 1} ​ (x - c)) \leq \frac{1}{{(2 ​ π ​ l_{1})}^{d / 2}} ​ \exp (- \frac{{‖ X ‖}^{2}}{8 ​ u_{1}}),

for any $‖ X ‖ \geq 2 u$ . Combine this result with the previous bound, we obtain that $f_{ℒ} (X | c, Γ) \leq G_{1} (X)$ , where

G_{1} ​ (X) := {\begin{matrix} \frac{1}{{(2 ​ π ​ l_{1})}^{d / 2}} ​ \exp (- \frac{{‖ X ‖}^{2}}{8 ​ u_{1}}), ‖ X ‖ \geq 2 ​ u, \\ \frac{1}{{(2 ​ π ​ l_{1})}^{d / 2}}, ‖ X ‖ < 2 ​ u . \end{matrix}

By arguing in a similar fashion, we also have $f_{𝒟} (Y | a^{⊤} X + b, ν) \leq G_{2} (X, Y)$ where

G_{2} ​ (X, Y) := {\begin{matrix} \frac{1}{\sqrt{2 ​ π ​ l_{3}}} ​ \exp (- \frac{Y^{2}}{8 ​ u_{3}}), | Y | \geq 2 ​ u_{2} \\ \frac{1}{\sqrt{2 ​ π ​ l_{3}}}, | Y | < 2 ​ u_{2} . \end{matrix}

Consequently, we achieve that $f_{ℒ} (X | c, Γ) f_{𝒟} (Y | a^{⊤} X + b, ν) \leq G (X, Y) := G_{1} (X) G_{2} (X, Y)$ .

Next, given some $η > 0$ that we will choose later, we consider an $η$ -cover of $𝒫_{k, β} (Θ)$ which is assumed to have $N$ elements denoted by $f_{1}, f_{2}, \dots, f_{N}$ . For any $i \in [N]$ , we define

L_{i} ​ (X, Y) := \max {f_{i} ​ (X, Y) - η, 0}, U_{i} ​ (X, Y) := {f_{i} ​ (X, Y) + η, G ​ (X, Y)} .

Then, we can validate that $𝒫_{k, β} (Θ) \subset \cup_{i = 1}^{N} [L_{i} (X, Y), U_{i} (X, Y)]$ and $U_{i} (X, Y) - L_{i} (X, Y) \leq \min {2 η, G (X, Y)}$ . Furthermore, we also deduce that

	${‖ U_{i} - L_{i} ‖}_{1} = \int (U_{i} (X, Y) - L_{i} (X, Y)) d (X, Y)$
	$= \int_{\| Y \| < 2 u_{2}} (U_{i} (X, Y) - L_{i} (X, Y)) d (X, Y) + \int_{\| Y \| \geq 2 u_{2}} (U_{i} (X, Y) - L_{i} (X, Y)) d (X, Y)$
	$\leq c_{1} η + \exp (- c_{1}^{2} / (2 u_{3})) \leq c_{2} η,$

where $c_{1} = \max {2 u_{2}, \sqrt{8} u_{3}} \log (1 / η)$ and $c_{2} > 0$ is some universal constant. This means that each bracket $[L_{i} (X, Y), U_{i} (X, Y)]$ is of size $c_{2} η$ . Recall that the bracketing entropy is the logarithm of the smallest number of brackets to cover $𝒫_{k, β} (Θ)$ , it follows that

H_{B} (c_{2} η, 𝒫_{k, β} (Θ), ∥ \cdot ∥_{1})

\leq \log N (η, 𝒫_{k, β} (Θ), ∥ \cdot ∥_{1}) \leq \log N (η, 𝒫_{k, β} (Θ), ∥ \cdot ∥_{\infty}) ≲ \log (1 / η),

where the second inequality occurs since $∥ \cdot ∥_{\infty} \leq ∥ \cdot ∥_{1}$ , while the last inequality is due to the result in part (i). Moreover, as the Hellinger distance is upper bounded by the $L 1$ -norm $∥ \cdot ∥_{1}$ , we get that

H_{B} (c_{2} η, 𝒫_{k, β} (Θ), h) \leq H_{B} (c_{2} η, 𝒫_{k, β} (Θ), ∥ \cdot ∥_{1}) ≲ \log (1 / η) .

Here, if we choose $η = ε / c_{2}$ , we can conclude that $H_{B} (ε, 𝒫_{k, β} (Θ), h) \leq \log (1 / ε)$ .

C.3 Proof of Lemma 2

We begin with recalling the system of interest here:

\sum_{l = 1}^{m} \sum_{α \in 𝒥_{ℓ_{1}, ℓ_{2}}} \frac{p_{l}^{2} ​ q_{1 ​ l}^{α_{1}} ​ q_{2 ​ l}^{α_{2}} ​ q_{3 ​ l}^{α_{3}} ​ q_{4 ​ l}^{α_{4}} ​ q_{5 ​ l}^{α_{5}}}{α_{1}! ​ α_{2}! ​ α_{3}! ​ α_{4}! ​ α_{5}!} = 0,

(50)

with unknown variables ${(p_{l}, q_{1 l}, q_{2 l}, q_{3 l}, q_{4 l}, q_{5 l})}_{l = 1}^{m} \subset ℝ^{5}$ for all $ℓ_{1} \geq 0$ and $ℓ_{2} \geq 0$ that satisfy $1 \leq ℓ_{1} + ℓ_{2} \leq r$ , where

𝒥_{ℓ_{1}, ℓ_{2}} := {α = {(α_{i})}_{i = 1}^{5} \in ℕ^{5} : α_{1} + 2 ​ α_{2} + α_{3} = ℓ_{1}, α_{3} + α_{4} + 2 ​ α_{5} = ℓ_{2}} .

Let us consider only a part of the above system when $ℓ_{1} = 0$ as follows:

\sum_{l = 1}^{m} \sum_{\begin{matrix} α_{4}, α_{5} \in ℕ \\ α_{4} + 2 ​ α_{5} = ℓ_{2} \end{matrix}} \frac{p_{l}^{2} ​ q_{4 ​ l}^{α_{4}} ​ q_{5 ​ l}^{α_{5}}}{α_{4}! ​ α_{5}!} = 0,

(51)

for all $1 \leq ℓ_{2} \leq r$ , which takes the same form as the system in equation (9). Thus, it follows from Lemma 1 that the smallest positive integer $r$ such that the system (51) does not admit any non-trivial solutions is $\bar{r} (m)$ . Therefore, we obtain that $\tilde{r} (m) \leq \bar{r} (m)$ .

Next, we will respectively show that $\tilde{r} (2) = 4$ and $\tilde{r} (3) = 6$ .

When $m = 2$ : In this case, it follows from the above result that $\tilde{r} (m) \leq \bar{r} (m) = 4$ . Thus, it is sufficient to demonstrate $\tilde{r} (m) > 3$ , i.e. pointing out a non-trivial solution for the system (50) when $r = 3$ , which is given by

	$\sum_{l = 1}^{m} p_{l}^{2} q_{1 l} = 0, \sum_{l = 1}^{m} p_{l}^{2} q_{4 l} = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{2!} q_{1 l}^{2} + q_{2 l}) = 0, \sum_{l = 1}^{m} p_{l}^{2} (q_{1 l} q_{4 l} + q_{3 l}) = 0, \sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{2!} q_{4 l}^{2} + q_{5 l}) = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{3!} q_{1 l}^{3} + q_{1 l} q_{2 l}) = 0, \sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{2!} q_{1 l}^{2} q_{4 l} + q_{1 l} q_{3 l} + q_{2 l} q_{4 l}) = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{2!} q_{1 l} q_{4 l}^{2} + q_{1 l} q_{5 l} + q_{3 l} q_{4 l}) = 0, \sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{3!} q_{4 l}^{3} + q_{4 l} q_{5 l}) = 0 .$		(52)

We can check that the following is a non-trivial solution of the system (C.3):

	$p_{l} = 0, q_{1 l} = q_{2 l} = q_{3 l} = 0, \forall l \in [m],$
	$q_{41} = 1, q_{42} = - 1, q_{51} = q_{52} = - \frac{1}{2} .$

Hence, we conclude that $\tilde{r} (m) = 4$ .

When $m = 3$ : Again, according to Lemma 1, we have $\tilde{r} (m) \leq \bar{r} (m) = 6$ . Therefore, it suffices to show a non-trivial solution of the system (50) for $r = 5$ , which is a combination of the system (C.3) and the following system:

	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{4!} q_{1 l}^{4} + \frac{1}{2!} q_{1 l}^{2} q_{2 l} + \frac{1}{2!} q_{2 l}^{2}) = 0, \sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{4!} q_{4 l}^{2} + \frac{1}{2!} q_{4 l}^{2} q_{5 l} + \frac{1}{2!} q_{5 l}^{2}) = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{3!} q_{1 l}^{3} q_{4 l} + \frac{1}{2!} q_{1 l} q_{2 l}^{2} + \frac{1}{2!} q_{1 l}^{2} q_{2 l} + q_{2 l} q_{3 l}) = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{3!} q_{1 l} q_{4 l}^{3} + \frac{1}{2!} q_{1 l} q_{4 l} q_{5 l}^{2} + \frac{1}{2!} q_{3 l} q_{4 l}^{2} + q_{3 l} q_{5 l}) = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{2! 2!} q_{1 l}^{2} q_{4 l}^{2} + \frac{1}{2!} q_{2 l} q_{4 l}^{2} + \frac{1}{2!} q_{1 l}^{2} q_{5 l} + q_{2 l} q_{5 l} + q_{1 l} q_{3 l} q_{4 l} + \frac{1}{2!} q_{3 l}^{2}) = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{5!} q_{1 l}^{5} + \frac{1}{3!} q_{1 l}^{3} q_{2 l} + \frac{1}{2!} q_{1 l} q_{2 l}^{2}) = 0, \sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{5!} q_{4 l}^{5} + \frac{1}{3!} q_{4 l}^{3} q_{5 l} + \frac{1}{2!} q_{4 l} q_{5 l}^{2}) = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{4!} q_{1 l}^{4} q_{4 l} + \frac{1}{2!} q_{1 l}^{2} q_{2 l} + \frac{1}{2!} q_{2 l}^{2} q_{4 l} + \frac{1}{3!} q_{1 l}^{3} q_{3 l} + q_{1 l} q_{2 l} q_{3 l}) = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{4!} q_{1 l} q_{4 l}^{4} + \frac{1}{2!} q_{1 l} q_{4 l}^{2} q_{5 l} + \frac{1}{2!} q_{1 l} q_{5 l}^{2} + \frac{1}{3!} q_{3 l} q_{4 l}^{3} + q_{3 l} q_{4 l} q_{5 l}) = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{3! 2!} q_{1 l}^{3} q_{4 l}^{2} + \frac{1}{3!} q_{1 l}^{3} q_{5 l} + \frac{1}{2!} q_{1 l} q_{2 l} q_{4 l}^{2} + q_{1 l} q_{2 l} q_{4 l} + \frac{1}{2!} q_{1 l}^{2} q_{3 l} q_{4 l} + q_{2 l} q_{3 l} q_{4 l} + \frac{1}{2!} q_{1 l} q_{3 l}^{2}) = 0,$
	$\sum_{l = 1}^{m} p_{l}^{2} (\frac{1}{2! 3!} q_{1 l}^{2} q_{4 l}^{3} + q_{1 l}^{2} q_{4 l} q_{5 l} + \frac{1}{3!} q_{2 l} q_{3 l}^{3} + q_{2 l} q_{4 l} q_{5 l} + \frac{1}{2!} q_{1 l} q_{3 l} q_{4 l}^{2} + q_{1 l} q_{3 l} q_{5 l} + \frac{1}{2!} q_{3 l}^{2} q_{4 l}) = 0 .$

It can be verified that the following is a non-trivial of this system:

	$p_{l} = 0, q_{1 l} = q_{2 l} = q_{3 l} = 0, \forall l \in [m],$
	$q_{41} = \frac{\sqrt{3}}{3}, q_{42} = - \frac{\sqrt{3}}{3}, q_{43} = 0, q_{51} = q_{52} = - \frac{1}{6}, q_{53} = 0 .$

As a consequence, we obtain that $\tilde{r} (m) > 5$ , which implies the desired conclusion that $\tilde{r} (m) = 6$ .

[bib.bib1] [1] J.-P. Baudry, C. Maugis, and B. Michel. Slope heuristics: overview and implementation. Statistics and Computing, 22(2):455–470, 2012. ↩ a

[bib.bib2] [2] L. Birgé and P. Massart. Minimal penalties for Gaussian model selection. Probability Theory and Related Fields, 138(1):33–73, 2007. Publisher: Springer. ↩ a

[bib.bib3] [3] F. Chamroukhi and B.-T. Huynh. Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models. Journal de la Société Française de Statistique, 160(1):57–85, 2019. ↩ a

[bib.bib4] [4] A. Deleforge, F. Forbes, and R. Horaud. High-dimensional regression with gaussian mixtures and partially-latent response variables. Statistics and Computing, 25(5):893–911, 2015. ↩ a b

[bib.bib5] [5] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data Via the EM Algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–22, Sept. 1977. Publisher: John Wiley & Sons, Ltd. ↩ a

[bib.bib6] [6] C. Diani, G. Galimberti, and G. Soffritti. Multivariate cluster-weighted models based on seemingly unrelated linear regression. Computational Statistics & Data Analysis, page 107451, Feb. 2022. ↩ a

[bib.bib7] [7] T. G. Do, H. K. Le, T. Nguyen, Q. Pham, B. T. Nguyen, T.-N. Doan, C. Liu, S. Ramasamy, X. Li, and S. HOI. HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, Dec. 2023. Association for Computational Linguistics. ↩ a

[bib.bib8] [8] W. Fedus, J. Dean, and B. Zoph. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667, 2022. ↩ a

[bib.bib9] [9] W. Fedus, B. Zoph, and N. Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. ↩ a

[bib.bib10] [10] F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel. Mixture of expert posterior surrogates for approximate Bayesian computation. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique (SFdS), Lyon, France, June 2022. ↩ a

[bib.bib11] [11] F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel. Summary statistics and discrepancy measures for approximate Bayesian computation via surrogate posteriors. Statistics and Computing, 32(5):85, Oct. 2022. ↩ a b

[bib.bib12] [12] J. Fritsch, M. Finke, and A. Waibel. Adaptively growing hierarchical mixtures of experts. In Advances in Neural Information Processing Systems, volume 9, 1996. ↩ a

[bib.bib13] [13] A. Guha, N. Ho, and X. Nguyen. On posterior contraction of parameters and interpretability in Bayesian mixture modeling. Bernoulli, 27(4):2159 – 2188, 2021. Publisher: Bernoulli Society for Mathematical Statistics and Probability. ↩ a b c

[bib.bib14] [14] X. Han, H. Nguyen, C. Harris, N. Ho, and S. Saria. Fusemoe: Mixture-of-experts transformers for fleximodal fusion. arXiv preprint arXiv:2402.03226, 2024. ↩ a

[bib.bib15] [15] N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. The Annals of Statistics, 44(6):2726 – 2755, 2016. Publisher: Institute of Mathematical Statistics and Bernoulli Society. ↩ a b c d

[bib.bib16] [16] N. Ho and X. Nguyen. On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics, 10(1):271–307, 2016. Publisher: The Institute of Mathematical Statistics and the Bernoulli Society. ↩ a

[bib.bib17] [17] N. Ho and X. Nguyen. Singularity Structures and Impacts on Parameter Estimation in Finite Mixtures of Distributions. SIAM Journal on Mathematics of Data Science, 1(4):730–758, Jan. 2019. Publisher: Society for Industrial and Applied Mathematics. ↩ a

[bib.bib18] [18] N. Ho, C.-Y. Yang, and M. I. Jordan. Convergence Rates for Gaussian Mixtures of Experts. Journal of Machine Learning Research, 23(323):1–81, 2022. ↩ a b

[bib.bib19] [19] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991. Publisher: MIT Press. ↩ a b

[bib.bib20] [20] W. Jiang and M. A. Tanner. Hierarchical mixtures-of-experts for exponential family regression models: approximation and maximum likelihood estimation. Annals of Statistics, pages 987–1011, 1999. ↩ a b

[bib.bib21] [21] W. Jiang and M. A. Tanner. Hierarchical Mixtures-of-Experts for Generalized Linear Models: Some Results on Denseness and Consistency. In D. Heckerman and J. Whittaker, editors, Proceedings of the Seventh International Workshop on Artificial Intelligence and Statistics, volume R2 of Proceedings of Machine Learning Research. PMLR, Jan. 1999. ↩ a

[bib.bib22] [22] W. Jiang and M. A. Tanner. On the identifiability of mixtures-of-experts. Neural Networks, 12(9):1253–1258, 1999. ↩ a

[bib.bib23] [23] M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural computation, 6(2):181–214, 1994. Publisher: MIT Press. ↩ a b

[bib.bib24] [24] A. Khalili. New estimation and feature selection methods in mixture-of-experts models. Canadian Journal of Statistics, 38(4):519–539, 2010. Publisher: Wiley Online Library. ↩ a

[bib.bib25] [25] B. Kugler, F. Forbes, and S. Douté. Fast Bayesian inversion for high dimensional inverse problems. Statistics and Computing, 32(2):31, Mar. 2022. ↩ a

[bib.bib26] [26] J. Kwon and C. Caramanis. EM Converges for a Mixture of Many Linear Regressions. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1727–1736. PMLR, Aug. 2020. ↩ a

[bib.bib27] [27] J. Kwon, N. Ho, and C. Caramanis. On the Minimax Optimality of the EM Algorithm for Learning Two-Component Mixed Linear Regression. In A. Banerjee and K. Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 1405–1413. PMLR, Apr. 2021. ↩ a

[bib.bib28] [28] J. Kwon, W. Qian, C. Caramanis, Y. Chen, and D. Davis. Global Convergence of the EM Algorithm for Mixtures of Two Component Linear Regression. In A. Beygelzimer and D. Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 2055–2110. PMLR, June 2019. ↩ a

[bib.bib29] [29] S. Lathuilière, R. Juge, P. Mesejo, R. Muñoz-Salinas, and R. Horaud. Deep mixture of linear inverse regressions applied to head-pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4817–4825, 2017. ↩ a

[bib.bib30] [30] T. Manole and N. Ho. Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Models. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14979–15006. PMLR, July 2022. ↩ a

[bib.bib31] [31] T. Manole and A. Khalili. Estimating the number of components in finite mixture models via the Group-Sort-Fuse procedure. The Annals of Statistics, 49(6):3043 – 3069, 2021. Publisher: Institute of Mathematical Statistics. ↩ a

[bib.bib32] [32] S. Masoudnia and R. Ebrahimpour. Mixture of experts: a literature survey. Artificial Intelligence Review, 42(2):275–293, 2014. ↩ a

[bib.bib33] [33] E. F. Mendes and W. Jiang. On convergence rates of mixtures of polynomial experts. Neural computation, 24(11):3025–3051, 2012. Publisher: MIT Press. ↩ a

[bib.bib34] [34] J. Moody and C. J. Darken. Fast Learning in Networks of Locally-Tuned Processing Units. Neural Computation, 1(2):281–294, 1989. ↩ a

[bib.bib35] [35] B. Mustafa, C. R. Ruiz, J. Puigcerver, R. Jenatton, and N. Houlsby. Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. ↩ a

[bib.bib36] [36] H. Nguyen, P. Akbarian, and N. Ho. Is temperature sample efficient for softmax Gaussian mixture of experts? arXiv preprint arXiv:2401.13875, 2024. ↩ a

[bib.bib37] [37] H. Nguyen, P. Akbarian, T. Nguyen, and N. Ho. A general theory for softmax gating multinomial logistic mixture of experts. arXiv preprint arXiv:2310.14188, 2023. ↩ a

[bib.bib38] [38] H. Nguyen, P. Akbarian, F. Yan, and N. Ho. Statistical perspective of top-k sparse softmax gating mixture of experts. In International Conference on Learning Representations, 2024. ↩ a

[bib.bib39] [39] H. Nguyen, N. Ho, and A. Rinaldo. On least squares estimation in softmax gating mixture of experts. arXiv preprint arXiv:2402.02952, 2024. ↩ a

[bib.bib40] [40] H. Nguyen, T. Nguyen, and N. Ho. Demystifying softmax gating function in Gaussian mixture of experts. In Advances in Neural Information Processing Systems, 2023. ↩ a

[bib.bib41] [41] H. D. Nguyen, T. Nguyen, F. Chamroukhi, and G. J. McLachlan. Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models. Journal of Statistical Distributions and Applications, 8(1):13, Aug. 2021. ↩ a

[bib.bib42] [42] T. Nguyen, F. Chamroukhi, H. D. Nguyen, and F. Forbes. Model selection by penalization in mixture of experts models with a non-asymptotic approach. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique (SFdS), Lyon, France, June 2022. ↩ a

[bib.bib43] [43] T. Nguyen, F. Forbes, J. Arbel, and H. D. Nguyen. Bayesian nonparametric mixture of experts for high-dimensional inverse problems. hal-04015203, Mar. 2023. ↩ a

[bib.bib44] [44] T. Nguyen, D. N. Nguyen, H. D. Nguyen, and F. Chamroukhi. A non-asymptotic theory for model selection in high-dimensional mixture of experts via joint rank and variable selection. Preprint hal-03984011, Feb. 2023. ↩ a

[bib.bib45] [45] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and F. Forbes. A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models. Electronic Journal of Statistics, 16(2):4742 – 4822, 2022. Publisher: Institute of Mathematical Statistics and Bernoulli Society. ↩ a

[bib.bib46] [46] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and G. J. McLachlan. Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts. arXiv:2009.10622, Feb. 2023. ↩ a

[bib.bib47] [47] A. Norets and D. Pati. Adaptive Bayesian estimation of conditional densities. Econometric Theory, 33(4):980–1012, 2017. Publisher: Cambridge University Press. ↩ a

[bib.bib48] [48] A. Norets and J. Pelenis. Adaptive Bayesian estimation of conditional discrete-continuous distributions with an application to stock market trading activity. Journal of Econometrics, 2021. ↩ a

[bib.bib49] [49] Q. Pham, G. Do, H. Nguyen, T. Nguyen, C. Liu, M. Sartipi, B. T. Nguyen, S. Ramasamy, X. Li, S. Hoi, and N. Ho. Competesmoe – effective training of sparse mixture of experts via competition, 2024. ↩ a

[bib.bib50] [50] J. Puigcerver, C. R. Ruiz, B. Mustafa, C. Renggli, A. S. Pinto, S. Gelly, D. Keysers, and N. Houlsby. Scalable Transfer Learning with Expert Models. In International Conference on Learning Representations, 2021. ↩ a

[bib.bib51] [51] M. Sato and S. Ishii. On-line EM algorithm for the normalized gaussian network. Neural computation, 12(2):407–432, feb 2000. ↩ a

[bib.bib52] [52] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations, 2017. ↩ a

[bib.bib53] [53] S. van de Geer. Empirical Processes in M-estimation. Cambridge University Press, 2000. ↩ a b c d e f

[bib.bib54] [54] L. Xu, M. Jordan, and G. E. Hinton. An Alternative Model for Mixtures of Experts. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7. MIT Press, 1995. ↩ a

[bib.bib55] [55] Z. You, S. Feng, D. Su, and D. Yu. Speechmoe2: Mixture-of-Experts Model with Improved Routing. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7217–7221, 2022. ↩ a

[bib.bib56] [56] S. E. Yuksel, J. N. Wilson, and P. D. Gader. Twenty Years of Mixture of Experts. IEEE Transactions on Neural Networks and Learning Systems, 23(8):1177–1193, 2012. ↩ a

		$\tilde{D} (G, G_{0}) := \sum_{\begin{matrix} j \in [\tilde{k}] : \| 𝒜_{j} \| > 1, \\ i \in 𝒜_{j} \end{matrix}} π_{i} K_{i j} ({\tilde{r}}_{j}, \frac{{\tilde{r}}_{j}}{2}, \frac{{\tilde{r}}_{j}}{2}, {\tilde{r}}_{j}, \frac{{\tilde{r}}_{j}}{2})$
		$+ \sum_{\begin{matrix} j \in [k_{0}] ∖ [\tilde{k}] : \| 𝒜_{j} \| > 1, \\ i \in 𝒜_{j} \end{matrix}} π_{i} K_{i j} ({\bar{r}}_{j}, \frac{{\bar{r}}_{j}}{2}, 2, {\bar{r}}_{j}, \frac{{\bar{r}}_{j}}{2})$
		$+ \sum_{\begin{matrix} j \in [k_{0}] : \| 𝒜_{j} \| = 1, \\ i \in 𝒜_{j} \end{matrix}} π_{i} K_{i j} (1, 1, 1, 1, 1) + \sum_{j = 1}^{k_{0}} \| \sum_{i \in 𝒜_{j}} π_{i} - π_{j}^{0} \| .$		(13)

	$F (X, Y \| θ_{i}^{n}) - F (X, Y \| θ_{j}^{0})$
	$= \sum_{ℓ_{1} + ℓ_{2} = 1}^{2 r_{1 j}} Q_{ℓ_{1}, ℓ_{2}}^{n} (j) \cdot X^{ℓ_{1}} f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0})$
	$\times \frac{\partial^{ℓ_{2}} f_{𝒟}}{\partial h_{1}^{ℓ_{2}}} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0}) + R_{1 i j} (X, Y),$
	$F (X, Y \| θ_{i^{'}}^{n}) - F (X, Y \| θ_{j^{'}}^{0})$
	$= R_{2 i^{'} j^{'}} (X, Y) + \sum_{α_{3} = 0}^{r_{2 j^{'}}} \sum_{τ_{1} + τ_{2} = 0}^{2 (r_{2 j^{'}} - α_{3})} T_{α_{3}, τ_{1}, τ_{2}}^{n} (j^{'}) \cdot X^{α_{3}}$
	$\times \frac{\partial^{τ_{1}} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j^{'}}^{0}, Γ_{j^{'}}^{0}) \frac{\partial^{α_{3} + τ_{2}} f_{𝒟}}{\partial h_{1}^{α_{3} + τ_{2}}} (Y \| a_{j^{'}}^{0} X + b_{j^{'}}^{0}, ν_{j^{'}}^{0}) .$

	$ℱ := {X^{α_{3}} \frac{\partial^{\| τ_{1} \|} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}) \frac{\partial^{\| α_{3} \| + τ_{2}} f_{𝒟}}{\partial h_{1}^{\| α_{3} \| + τ_{2}}} (Y \|$	${(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0}) : j \in [k_{0}], 0 \leq \| α_{3} \| \leq \bar{r} (\| 𝒜_{j} \|),$
		$0 \leq \| τ_{1} \| + τ_{2} \leq 2 (\bar{r} (\| 𝒜_{j} \|) - \| α_{3} \|)} .$		(18)

	$\sum_{j = 1}^{k_{0}} \sum_{ω = 0}^{2 \bar{r} (\| 𝒜_{j} \|)} (\sum_{\| α_{3} \| + τ_{2} = ω} \sum_{\| τ_{1} \| = 0}^{2 (\bar{r} (\| 𝒜_{j} \|) - α_{3}) - τ_{2}}$	$ξ_{α_{3}, τ_{1}, τ_{2}} (j) X^{α_{3}} \cdot \frac{\partial^{\| τ_{1} \|} f_{ℒ}}{\partial c^{τ_{1}}} (X \| c_{j}^{0}, Γ_{j}^{0}))$
		$\times \frac{\partial^{ω} f_{𝒟}}{\partial h_{1}^{ω}} (Y \| {(a_{j}^{0})}^{⊤} X + b_{j}^{0}, ν_{j}^{0}) = 0,$		(29)

	$p_{G_{n}} (X, Y) - p_{G_{0}} (X, Y)$
	$= \sum_{j : \| 𝒜_{j} \| > 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} [f_{ℒ} (X \| c_{i}^{n}, Γ_{i}^{n}) f_{𝒟} (Y \| a_{i}^{n} X + b_{i}^{n}, ν_{i}^{n}) - f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) f_{𝒟} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0})]$
	$+ \sum_{j : \| 𝒜_{j} \| = 1} \sum_{i \in 𝒜_{j}} π_{i}^{n} [f_{ℒ} (X \| c_{i}^{n}, Γ_{i}^{n}) f_{𝒟} (Y \| a_{i}^{n} X + b_{i}^{n}, ν_{i}^{n}) - f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) f_{𝒟} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0})]$
	$+ \sum_{j = 1}^{k_{0}} (\sum_{i \in 𝒜_{j}} π_{i}^{n} - π_{j}^{0}) f_{ℒ} (X \| c_{j}^{0}, Γ_{j}^{0}) f_{𝒟} (Y \| a_{j}^{0} X + b_{j}^{0}, ν_{j}^{0})$
	$:= A_{n} + B_{n} + E_{n} .$