Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models

Hien D. Nguyen; TrungTin Nguyen; Faicel Chamroukhi; Geoffrey J. McLachlan

Abstract

Mixture of experts (MoE) models are widely applied for conditional probability density estimation problems. We demonstrate the richness of the class of MoE models by proving denseness results in Lebesgue spaces, when inputs and outputs variables are both compactly supported. We further prove an almost uniform convergence result when the input is univariate. Auxiliary lemmas are proved regarding the richness of the soft-max gating function class, and their relationships to the class of Gaussian gating functions.

¹Department of Mathematics and Statistics, La Trobe University, Bundoora Victoria, Australia.
²Normandie Univ, UNICAEN, CNRS, LMNO, 14000 Caen, France.
³School of Mathematics and Physics, University of Queensland, St. Lucia Brisbane, Australia.
^∗Corresponding author—email: h.nguyen5@latrobe.edu.au.

Key words: mixture of experts; conditional probability density functions; approximation theory; mixture models; Lebesgue spaces

1 Introduction

Mixture of experts (MoE) models are a widely applicable class of conditional probability density approximations that have been considered as solution methods across the spectrum of statistical and machine learning problems; see, for example, the reviews of Yuksel et al., (2012), Masoudnia & Ebrahimpour, (2014), and Nguyen & Chamroukhi, (2018).

Let $ℤ = 𝕏 \times 𝕐$ , where $𝕏 \subseteq ℝ^{d}$ and $𝕐 \subseteq ℝ^{q}$ , for $d, q \in ℕ$ . Suppose that the input and output random variables, $𝑿 \in 𝕏$ and $𝒀 \in 𝕐$ , are related via the conditional probability density function (PDF) $f (𝒚 | 𝒙)$ in the functional class:

ℱ = {f : ℤ \to [0, \infty) | \int_{𝕐} f (𝒚 | 𝒙) d λ (𝒚) = 1, \forall 𝒙 \in 𝕏},

where $λ$ denotes the Lebesgue measure. The MoE approach seeks to approximate the unknown target conditional PDF $f$ by a function of the MoE form:

m ​ (𝒚 | 𝒙) = \sum_{k = 1}^{K} {Gate}_{k} ​ (𝒙) ​ {Expert}_{k} ​ (𝒚) ​,

where $𝐆𝐚𝐭𝐞 = {({Gate}_{k})}_{k \in [K]} \in 𝒢^{K}$ ( $[K] = {1, \dots, K}$ ), ${Expert}_{1}, \dots, {Expert}_{K} \in ℰ$ , and $K \in ℕ$ . Here, we say that $m$ is a $K -component$ MoE model with gates arising from the class $𝒢^{K}$ and experts arising from $ℰ$ , where $ℰ$ is a class of PDFs with support $𝕐$ .

The most popular choices for $𝒢^{K}$ are the parametric soft-max and Gaussian gating classes:

𝒢_{S}^{K} = {𝐆𝐚𝐭𝐞 = {({Gate}_{k} ​ (\cdot; 𝜸))}_{k \in [K]} | \forall k \in [K], {Gate}_{k} ​ (\cdot; 𝜸) = \frac{\exp (a_{k} + 𝒃_{k}^{⊤} \cdot)}{\sum_{l = 1}^{K} \exp (a_{l} + 𝒃_{l}^{⊤} \cdot)}, 𝜸 \in 𝔾_{S}^{K}}

and

𝒢_{G}^{K} = {𝐆𝐚𝐭𝐞 = {({Gate}_{k} ​ (\cdot; 𝜸))}_{k \in [K]} | \forall k \in [K], {Gate}_{k} ​ (\cdot; 𝜸) = \frac{π_{k} ​ ϕ ​ (\cdot; 𝝂_{k}, 𝚺_{k})}{\sum_{l = 1}^{K} π_{l} ​ ϕ ​ (\cdot; 𝝂_{l}, 𝚺_{l})}, 𝜸 \in 𝔾_{G}^{K}} ​,

respectively, where

𝔾_{S}^{K} = {𝜸 = (a_{1}, \dots, a_{K}, 𝒃_{1}, \dots, 𝒃_{K}) \in ℝ^{K} \times {(ℝ^{d})}^{K}}

and

𝔾_{G}^{K} = {𝜸 = (𝝅, 𝝂_{1}, \dots, 𝝂_{K}, 𝚺_{1}, \dots, 𝚺_{K}) \in Π_{K - 1} \times {(ℝ^{d})}^{K} \times 𝕊_{d}^{K}} ​ .

Here,

ϕ (\cdot; 𝝂, 𝚺) = | 2 π 𝚺 |^{- 1 / 2} \exp [- \frac{1}{2} {(\cdot - 𝝂)}^{⊤} 𝚺^{- 1} (\cdot - 𝝂)]

is the multivariate normal density function with mean vector $𝝂$ and covariance matrix $𝚺$ , $𝝅^{⊤} = (π_{1}, \dots, π_{K})$ is a vector of weights in the simplex:

Π_{K - 1} = {𝝅 = {(π_{k})}_{k \in [K]} | \forall k \in [K], π_{k} > 0, \sum_{k = 1}^{K} π_{k} = 1} ​,

and $𝕊_{d}$ is the class of $d \times d$ symmetric positive definite matrices. The soft-max and Gaussian gating classes were first introduced by Jacobs et al., (1991) and Jordan & Xu, (1995), respectively. Typically, one chooses experts that arise from some location-scale class:

ℰ_{ψ} = {g_{ψ} ​ (\cdot; 𝝁, σ) : 𝕐 \to [0, \infty) | g_{ψ} ​ (\cdot; 𝝁, σ) = \frac{1}{σ^{q}} ​ ψ ​ (\frac{\cdot - 𝝁}{σ}), 𝝁 \in ℝ^{q}, σ \in (0, \infty)} ​,

where $ψ$ is a PDF, with respect to $ℝ^{q}$ in the sense that $ψ : ℝ^{q} \to [0, \infty)$ and $\int_{ℝ^{q}} ψ (𝒚) d λ (𝒚) = 1$ .

We shall say that $f \in ℒ_{p} (ℤ)$ for any $p \in [1, \infty)$ if

{‖ f ‖}_{p, ℤ} = {(\int_{ℤ} {| 𝟏_{ℤ} ​ f |}^{p} ​ d ​ λ ​ (𝒛))}^{1 / p} < \infty ​,

where $𝟏_{ℤ}$ is the indicator function that takes value $1$ when $𝒛 \in ℤ$ , and 0 otherwise. Further, we say that $f \in ℒ_{\infty} (ℤ)$ if

{‖ f ‖}_{\infty, ℤ} = inf {a \geq 0 | λ ​ ({𝒛 \in ℤ | | f ​ (𝒛) | > a}) = 0} < \infty ​ .

We shall refer to $∥ \cdot ∥_{p, ℤ}$ as the $ℒ_{p}$ norm on $ℤ$ , for $p \in [0, \infty]$ , and where the context is obvious, we shall drop the reference to $ℤ$ .

Suppose that the target conditional PDF $f$ is in the class $ℱ_{p} = ℱ \cap ℒ_{p}$ . We address the problem of approximating $f$ , with respect to the $ℒ_{p}$ norm, using MoE models in the soft-max and Gaussian gated classes,

	$ℳ_{S}^{ψ}$	$=$	${m_{K}^{ψ} : ℤ \to [0, \infty) \| m_{K}^{ψ} (𝒚 \| 𝒙) = \sum_{k = 1}^{K} {Gate}_{k} (𝒙) g_{ψ} (𝒚; 𝝁_{k}, σ_{k}),$
			$g_{ψ} \in ℰ_{ψ} \cap ℒ_{\infty}, 𝐆𝐚𝐭𝐞 \in 𝒢_{S}^{K}, 𝝁_{k} \in 𝕐, σ_{k} \in (0, \infty), k \in [K], K \in ℕ}$

and

	$ℳ_{G}^{ψ}$	$=$	${m_{K}^{ψ} : ℤ \to [0, \infty) \| m_{K}^{ψ} (𝒚 \| 𝒙) = \sum_{k = 1}^{K} {Gate}_{k} (𝒙) g_{ψ} (𝒚; 𝝁_{k}, σ_{k}),$
			$g_{ψ} \in ℰ_{ψ} \cap ℒ_{\infty}, 𝐆𝐚𝐭𝐞 \in 𝒢_{G}^{K}, 𝝁_{k} \in 𝕐, σ_{k} \in (0, \infty), k \in [K], K \in ℕ},$

by showing that both $ℳ_{S}^{ψ}$ and $ℳ_{G}^{ψ}$ are dense in the class $ℱ_{p}$ , when $𝕏 = {[0, 1]}^{d}$ and $𝕐$ is a compact subset of $ℝ^{q}$ . Our denseness results are enabled by the indicator function approximation result of Jiang & Tanner, 1999b , and the finite mixture model denseness theorems of Nguyen et al., 2020a and Nguyen et al., 2020c .

Our theorems contribute to an enduring continuity of sustained interest in the approximation capabilities of MoE models. Related to our results are contributions regarding the approximation capabilities of the conditional expectation function of the classes $ℳ_{S}^{ψ}$ and $ℳ_{G}^{ψ}$ (Wang & Mendel, 1992, Zeevi et al., 1998, Jiang & Tanner, 1999b, Krzyzak & Schafer, 2005, Mendes & Jiang, 2012, Nguyen et al., 2016, 2019) and the approximation capabilities of subclasses of $ℳ_{S}^{ψ}$ and $ℳ_{G}^{ψ}$ , with respect to the Kullback–Leibler divergence (Jiang & Tanner, 1999a, Norets, 2010, Norets & Pelenis, 2014). Our results can be seen as complements to the Kullback–Leibler approximation theorems of Norets, (2010) and Norets & Pelenis, (2014), by the relationship between the Kullback–Leibler divergence and the $ℒ_{2}$ norm (Zeevi & Meir, 1997). That is, when $f > 1 / κ$ , for all $(𝒙, 𝒚) \in ℤ$ and some constant $κ > 0$ , we have that the integrated conditional Kullback–Leibler divergence considered by Norets & Pelenis, (2014):

\int_{𝕏} D (f (\cdot | 𝒙) ∥ m_{K}^{ψ} (\cdot | 𝒙)) d λ (𝒙) = \int_{𝕏} \int_{𝕐} f (𝒚 | 𝒙) \log \frac{f ​ (𝒚 | 𝒙)}{m_{K}^{ψ} ​ (𝒚 | 𝒙)} d λ (𝒚) d λ (𝒙)

satisfies

\int_{𝕏} D (f (\cdot | 𝒙) ∥ m_{K}^{ψ} (\cdot | 𝒙)) d λ (𝒙) \leq κ^{2} ∥ f - m_{K}^{ψ} ∥_{2, ℤ}^{2},

and thus a good approximation in the integrated Kullback–Leibler divergence is guaranteed if one can find a good approximation in the $ℒ_{2}$ norm, which is guaranteed by our main result.

The remainder of the manuscript proceeds as follows. The main result is presented in Section 2. Technical lemmas are provided in Section 3. The proofs of our results are then presented in Section 4. Proofs of required lemmas that do not appear elsewhere are provided in Section 5. A summary of our work and some conclusions are drawn in Section 6.

2 Main results

Denote the class of bounded functions on $ℤ$ by

ℬ ​ (ℤ) = {f \in ℒ_{\infty} ​ (ℤ) | \exists a \in [0, \infty) ​, such that ​ | f ​ (𝒛) | \leq a ​, ​ \forall 𝒛 \in ℤ} ​,

and write its norm as ${‖ f ‖}_{ℬ (ℤ)} = {sup}_{𝒛 \in ℤ} | f (𝒛) |$ . Further, let $𝒞$ denote the class of continuous functions. Note that if $ℤ$ is compact and $f \in 𝒞$ , then $f \in ℬ$ .

Theorem 1.

Assume that $𝕏 = {[0, 1]}^{d}$ for $d \in ℕ$ . There exists a sequence ${m_{K}^{ψ}}_{K \in ℕ} \subset ℳ_{S}^{ψ}$ , such that if $𝕐 \subset ℝ^{q}$ is compact, $f \in ℱ \cap 𝒞$ , and $ψ \in 𝒞 (ℝ^{q})$ is a PDF on support $ℝ^{q}$ , then ${lim}_{K \to \infty} {‖ f - m_{K}^{ψ} ‖}_{p} = 0$ , for $p \in [1, \infty)$ .

Since convergence in Lebesgue spaces does not imply point-wise modes of convergence, the following result is also useful and interesting in some restricted scenarios. Here, we note that the mode of convergence is almost uniform, which implies almost everywhere convergence and convergence in measure (cf. Bartle, 1995, Lem 7.10 and Thm. 7.11). The almost uniform convergence of ${m_{K}^{ψ}}_{K \in ℕ}$ to $f$ in the following result is to be understood in the sense of Bartle, (1995, Def. 7.9). That is, for every $δ > 0$ , there exists a set $𝔼_{δ} \subset ℤ$ with $λ (ℤ) < δ$ , such that ${m_{K}^{ψ}}_{K \in ℕ}$ converges to $f$ , uniformly on $ℤ \ 𝔼_{δ}$ .

Theorem 2.

Assume that $𝕏 = [0, 1]$ . There exists a sequence ${m_{K}^{ψ}}_{K \in ℕ} \subset ℳ_{S}^{ψ}$ , such that if $𝕐 \subset ℝ^{q}$ is compact, $f \in ℱ \cap 𝒞$ , and $ψ \in 𝒞 (ℝ^{q})$ is a PDF on support $ℝ^{q}$ , then ${lim}_{K \to \infty} m_{K}^{ψ} = f$ , almost uniformly.

The following result establishes the connection between the gating classes $𝒢_{S}^{K}$ and $𝒢_{G}^{K}$ .

Lemma 1.

For each $K \in ℕ$ , $𝒢_{S}^{K} \subset 𝒢_{G}^{K}$ . Further, if we define the class of Gaussian gating vectors with equal covariance matrices:

𝒢_{E}^{K} = {𝐆𝐚𝐭𝐞 = {({Gate}_{k} ​ (\cdot; 𝜸))}_{k \in [K]} | \forall k \in [K], {Gate}_{k} ​ (\cdot; 𝜸) = \frac{π_{k} ​ ϕ ​ (\cdot; 𝝂_{k}, 𝚺)}{\sum_{l = 1}^{K} π_{l} ​ ϕ ​ (\cdot; 𝝂_{l}, 𝚺)}, 𝜸 \in 𝔾_{E}^{K}} ​,

where

𝔾_{E}^{K} = {𝜸 = (𝝅, 𝝂_{1}, \dots, 𝝂_{K}, 𝚺) \in Π_{K - 1} \times {(ℝ^{d})}^{K} \times 𝕊_{d}} ​,

then $𝒢_{E}^{K} \subset 𝒢_{S}^{K}$ .

We can directly apply Lemma 1 to establish the following corollary to Theorems 1 and 2, regarding the approximation capability of the class $ℳ_{G}^{ψ}$ .

Corollary 1.

Theorems 1 and 2 hold when $ℳ_{S}^{ψ}$ is replaced by $ℳ_{G}^{ψ}$ in their statements.

3 Technical lemmas

Let $𝕂^{n} = {(k_{1}, \dots, k_{d}) \in {[n]}^{d}}$ and $κ : 𝕂^{n} \to [n^{d}]$ be a bijection for each $n \in ℕ$ . For each $(k_{1}, \dots, k_{d}) \in 𝕂^{n}$ and $k \in [n^{d}]$ , we define $𝕏_{k}^{n} = 𝕏_{κ (k_{1}, \dots, k_{d})}^{n} = \prod_{i = 1}^{d} 𝕀_{k_{i}}^{n}$ , where $𝕀_{k_{i}}^{n} = [(k_{i} - 1) / n, k_{i} / n)$ for $k_{i} \in [n - 1]$ , and $𝕀_{n}^{n} = [(n - 1) / n, 1]$ .

We call ${𝕏_{k}^{n}}_{k \in [n^{d}]}$ a fine partition of $𝕏$ , in the sense that $𝕏 = {[0, 1]}^{d} = ⋃_{k = 1}^{n^{d}} 𝕏_{k}^{n}$ , for each $n$ , and that $λ (𝕏_{k}^{n}) = n^{- d}$ gets smaller, as $n$ increases. The following result from Jiang & Tanner, 1999b establishes the approximation capability of soft-max gates.

Lemma 2 (Jiang and Tanner, 1999, p. 1189).

For each $n \in ℕ$ , $p \in [1, \infty)$ and $ϵ > 0$ , there exists a gating functions

𝐆𝐚𝐭𝐞 = {({Gate}_{k} ​ (\cdot; 𝜸))}_{k \in [n^{d}]} \in 𝒢_{S}^{n^{d}}

for some $𝛄 \in 𝔾_{S}^{n^{d}}$ , such that

sup_{k \in [n^{d}]} {‖ 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} ​ (\cdot; 𝜸) ‖}_{p, 𝕏} \leq ϵ ​ .

When, $d = 1$ , we have also the following almost uniform convergence alternative to Lemma 2.

Lemma 3.

Let $𝕏 = [0, 1]$ . Then, for each $n \in ℕ$ , there exists a sequence of gating functions:

{{𝐆𝐚𝐭𝐞}_{l} = {({Gate}_{k} ​ (\cdot; 𝜸_{l}))}_{k \in [n^{d}]}}_{l \in ℕ} \subset 𝒢_{S}^{n} ​,

defined by ${𝛄_{l}}_{l \in ℕ} \subset 𝔾_{S}^{n}$ , such that

{Gate}_{k} ​ (\cdot; 𝜸_{l}) \to 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} ​,

almost uniformly, simultaneously for all $k \in [n^{d}]$ .

For PDF $ψ$ on support $ℝ^{q}$ , define the class of finite mixture models by

	$ℋ^{ψ}$	$=$	${h_{K}^{ψ} : ℝ^{q} \to [0, \infty) \| h_{K}^{ψ} (𝒚) = \sum_{k = 1}^{K} c_{k} g_{ψ} (𝒚; 𝝁_{k}, σ_{k}),$
			$g_{ψ} \in ℰ_{ψ} \cap ℒ_{\infty}, {(c_{k})}_{k \in [K]} \in Π_{K - 1}, 𝝁_{k} \in 𝕐, σ_{k} \in (0, \infty), k \in [K], K \in ℕ} .$

We require the following result, from Nguyen et al., 2020a , regarding the approximation capabilities of $ℋ^{ψ}$ .

Lemma 4 (Nguyen et al., 2020a, Thm. 2(b)).

If $f \in 𝒞 (𝕐)$ is a PDF on $𝕐$ , $ψ \in 𝒞 (ℝ^{q})$ is a PDF on $ℝ^{q}$ , and $𝕐 \subset ℝ^{q}$ is compact, then there exists a sequence ${h_{K}^{ψ}}_{K \in ℕ} \subset ℋ^{ψ}$ , such that ${lim}_{K \to \infty} {‖ f - h_{K}^{ψ} ‖}_{ℬ (𝕐)} = 0$ .

4 Proofs of main results

4.1 Proof of Theorem 1

To prove the result, it suffices to show that for each $ϵ > 0$ , there exists a $m_{K}^{ψ} \in ℳ_{S}^{ψ}$ , such that

{‖ f - m_{K}^{ψ} ‖}_{p} < ϵ ​ .

The main steps of the proof are as follows. We firstly approximate $f (𝒚 | 𝒙)$ by

υ_{n} ​ (𝒚 | 𝒙) = \sum_{k = 1}^{n^{d}} 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} ​ f ​ (𝒚 | 𝒙_{k}^{n}) ​,

(1)

where $𝒙_{k}^{n} \in 𝕏_{k}^{n}$ , for each $k \in [n^{d}]$ , such that

{‖ f - υ_{n} ‖}_{p} < \frac{ϵ}{3} ​,

(2)

for all $n \geq N_{1} (ϵ)$ , for some sufficiently large $N_{1} (ϵ) \in ℕ$ . Then we approximate $υ_{n} (𝒚 | 𝒙)$ by

η_{n} ​ (𝒚 | 𝒙) = \sum_{k = 1}^{n^{d}} {Gate}_{k} ​ (𝒙; 𝜸_{n}) ​ f ​ (𝒚 | 𝒙_{k}^{n}) ​,

(3)

where $𝜸_{n} \in 𝔾_{S}^{n^{d}}$ and $𝐆𝐚𝐭𝐞 = {({Gate}_{k} (\cdot; 𝜸_{n}))}_{k \in [n^{d}]} \in 𝒢_{S}^{n^{d}}$ , so that

{‖ υ_{n} - η_{n} ‖}_{p}

\leq sup_{k \in [n^{d}]} ∥ {Gate}_{k} (\cdot; 𝜸) - 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} ∥_{p, 𝕏} \sum_{k = 1}^{n^{d}} ∥ f (\cdot | 𝒙_{k}^{n}) ∥_{p, 𝕐} < \frac{ϵ}{3},

(4)

using Lemma 2.

Finally, we approximate $η_{n} (𝒚 | 𝒙)$ by $m_{K_{n}}^{ψ} (𝒚 | 𝒙)$ , where

m_{K_{n}}^{ψ} ​ (𝒚 | 𝒙) = \sum_{k = 1}^{n^{d}} {Gate}_{k} ​ (𝒙; 𝜸) ​ h_{n_{k}}^{k} ​ (𝒚 | 𝒙_{k}^{n})

(5)

and

h_{n_{k}}^{k} ​ (𝒚 | 𝒙_{k}^{n}) = \sum_{i = 1}^{n_{k}} c_{i}^{k} ​ g_{ψ} ​ (𝒚; 𝝁_{i}^{k}, σ_{i}^{k}) \in ℋ^{ψ}

(6)

for $n_{k} \in ℕ$ ( $k \in [n^{d}]$ ), such that $K_{n} = \sum_{k = 1}^{n^{d}} n_{k}$ . Here, we establish that there exists $N_{2} (ϵ, n, 𝜸_{n}) \in ℕ$ , so that when $n_{k} \geq N_{2} (ϵ, n, 𝜸_{n})$ ,

∥ η_{n} - m_{K_{n}}^{ψ} ∥_{p} \leq sup_{k \in [n^{d}]} ∥ {Gate}_{k} (\cdot; 𝜸) ∥_{p, 𝕏} \sum_{k = 1}^{n^{d}} ∥ f (\cdot | 𝒙_{k}^{n}) - h_{n_{k}}^{k} (\cdot | 𝒙_{k}^{n}) ∥_{p, 𝕐} < \frac{ϵ}{3} .

(7)

Results (2)–(7) then imply that for each $ϵ > 0$ , there exists $N_{1} (ϵ)$ , $𝜸_{n}$ , and $N_{2} (ϵ, n, 𝜸_{n})$ , such that for all $K_{n} = \sum_{k = 1}^{n^{d}} n_{k}$ , where $n_{k} \geq N_{2} (ϵ, n, 𝜸_{n})$ (for each $k \in [n^{d}]$ ) and $n \geq N_{1} (ϵ)$ . The following inequality results from an application of the triangle inequality:

{‖ f - m_{K_{n}}^{ψ} ‖}_{p}

\leq {‖ f - υ_{n} ‖}_{p} + {‖ υ_{n} - η_{n} ‖}_{p} + {‖ η_{n} - m_{K_{n}}^{ψ} ‖}_{p} < 3 \times \frac{ϵ}{3} = ϵ ​ .

We now focus our attention to proving each of the results: (2)–(7). To prove (2), we note that since $f$ is uniformly continuous (because $ℤ = 𝕏 \times 𝕐$ is compact, and $f \in 𝒞$ ), there exists a function (1) such that for all $ε > 0$ ,

sup_{(𝒙, 𝒚) \in ℤ} | f (𝒚 | 𝒙) - υ (𝒚 | 𝒙) | < ε .

(8)

We can construct such an approximation by considering the fact that as $n$ increases, the diameter $δ_{n} = {sup}_{k \in n^{d}} diam (𝕏_{k}^{n})$ of the fine partition goes to zero. By the uniform continuity of $f$ , for every $ε > 0$ , there exists a $δ (ϵ) > 0$ , such that if $‖ (𝒙_{1}, 𝒚_{1}) - (𝒙_{2}, 𝒚_{2}) ‖ < δ (ϵ)$ , then $| f (𝒚_{1} | 𝒙_{1}) - f (𝒚_{2} | 𝒙_{2}) | < ε$ , for all pairs $(𝒙_{1}, 𝒚_{1}), (𝒙_{2}, 𝒚_{2}) \in ℤ$ . Here, $∥ \cdot ∥$ denotes the Euclidean norm. Furthermore, for any $(𝒙, 𝒚) \in ℤ$ , we have

	$\| f (𝒚 \| 𝒙) - υ_{n} (𝒚 \| 𝒙) \|$	$= \| \sum_{k = 1}^{n^{d}} 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} [f (𝒚 \| 𝒙) - f (𝒚 \| 𝒙_{k}^{n})] \|$
		$\leq \sum_{k = 1}^{n^{d}} 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} \| f (𝒚 \| 𝒙) - f (𝒚 \| 𝒙_{k}^{n}) \|,$		(9)

by the triangle inequality.

Since $𝒙_{k}^{n} \in 𝕏_{k}^{n}$ , for each $k$ and $n$ , we have the fact that $‖ (𝒙, 𝒚) - (𝒙_{k}^{n}, 𝒚) ‖ < δ_{n}$ for $(𝒙, 𝒚) \in 𝕏_{k}^{n} \times 𝕐$ . By uniform continuity, for each $ε$ , we can find a sufficiently small $δ (ϵ)$ , such that $| f (𝒚 | 𝒙) - f (𝒚 | 𝒙_{k}^{n}) | < ε$ , if $‖ (𝒙, 𝒚) - (𝒙_{k}^{n}, 𝒚) ‖ < δ (ϵ)$ , for all $k$ . The desired result (8) can be obtained by noting that the right hand side of (9) consists of only one non-zero summand for any $(𝒙, 𝒚) \in ℤ$ , and by choosing $n \in ℕ$ sufficiently large, so that $δ_{n} < δ (ϵ)$ .

By (8), we have the fact that $υ_{n} \to f$ , point-wise. We can bound $υ_{n}$ as follows:

υ_{n} ​ (𝒚 | 𝒙) \leq \sum_{i = 1}^{n^{p}} 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} ​ sup_{𝜻 \in 𝕐, 𝝃 \in 𝕏} f ​ (𝜻 | 𝝃) = sup_{𝜻 \in 𝕐, 𝝃 \in 𝕏} f ​ (𝜻 | 𝝃) ​,

(10)

where the right-hand side is a constant and is therefore in $ℒ_{p}$ , since $ℤ$ is compact. An application of the Lebesgue dominated convergence theorem in $ℒ_{p}$ then yields (2).

Next we write

	${‖ υ_{n} - η_{n} ‖}_{p}$	$= ∥ \sum_{k = 1}^{n^{d}} 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} f (𝒚 \| 𝒙_{k}^{n}) - \sum_{k = 1}^{n^{d}} {Gate}_{k} (𝒙; 𝜸_{n}) f (𝒚 \| 𝒙_{k}^{n}) ∥_{p}$
		$\leq \sum_{k = 1}^{n^{d}} ∥ [𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{n})] f (𝒚 \| 𝒙_{k}^{n}) ∥_{p} .$

Since the norm arguments are separable in $𝒙$ and $𝒚$ , we apply Fubini’s theorem to get

	${‖ υ_{n} - η_{n} ‖}_{p}$	$= \sum_{k = 1}^{n^{d}} ∥ [𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{n})] ∥_{p, 𝕏} ∥ f (𝒚 \| 𝒙_{k}^{n}) ∥_{p, 𝕐}$
		$\leq sup_{k \in [n^{d}]} ∥ [𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{n})] ∥_{p, 𝕏} \sum_{k = 1}^{n^{d}} ∥ f (𝒚 \| 𝒙_{k}^{n}) ∥_{p, 𝕐}$

Because $f \in ℬ$ and $n^{d}$ is finite, for any fixed $n \in ℕ$ , we have $C_{1} (n) = \sum_{k = 1}^{n^{d}} ∥ f (𝒚 | 𝒙_{k}^{n}) ∥_{p, 𝕐} < \infty$ . For each $ϵ > 0$ , we need to choose a $𝜸_{n} \in 𝔾_{S}^{n^{d}}$ , such that

sup_{k \in [n^{d}]} {‖ [𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} ​ (𝒙; 𝜸_{n})] ‖}_{p, 𝕏} < \frac{ϵ}{3 ​ C_{1} ​ (n)} ​,

which can be achieved via a direct application of Lemma 2. We have thus shown (4).

Lastly, we are required to approximate $f (𝒚 | 𝒙_{k}^{n})$ for each $k \in [n^{d}]$ , by a function of form (6). Since $𝕐$ is compact and $f$ and $ψ$ are continuous, we can apply of Lemma 4, directly. Note that over a set of finite measure, convergence in $∥ \cdot ∥_{ℬ}$ implies convergence in $ℒ_{p}$ norm, for all $p \in [1, \infty]$ (cf. Oden & Demkowicz, 2010, Prop. 3.9.3).

We can then write (5) as

$m_{K_{n}}^{ψ} (𝒚 \| 𝒙)$	$= \sum_{k = 1}^{n^{d}} \frac{\exp (a_{n, k} + 𝒃_{n, k}^{⊤} 𝒙)}{\sum_{l = 1}^{n^{d}} \exp (a_{n, l} + 𝒃_{n, l}^{⊤} 𝒙)} h_{n_{k}}^{k} (𝒚 \| 𝒙_{k}^{n})$
	$= \sum_{k = 1}^{n^{d}} \sum_{i = 1}^{n_{k}} \frac{\exp (a_{n, k} + 𝒃_{n, k}^{⊤} 𝒙)}{\sum_{l = 1}^{n^{d}} \exp (a_{n, l} + 𝒃_{n, l}^{⊤} 𝒙)} \frac{c_{i}^{k}}{\sum_{l = 1}^{n_{k}} c_{l}^{k}} g_{ψ} (𝒚; 𝝁_{i}^{k}, σ_{i}^{k})$
	$= \sum_{k = 1}^{n^{d}} \sum_{i = 1}^{n_{k}} \frac{\exp (\log c_{i}^{k} + a_{n, k} + 𝒃_{n, k}^{⊤} 𝒙)}{\sum_{l = 1}^{n^{d}} \sum_{j = 1}^{n_{k}} \exp (\log c_{j}^{k} + a_{n, l} + 𝒃_{n, l}^{⊤} 𝒙)} g_{ψ} (𝒚; 𝝁_{i}^{k}, σ_{i}^{k}) ,$	(11)

where $𝜸_{n} = (a_{n, 1}, \dots, a_{n, n^{d}}, 𝒃_{n, 1}, \dots, 𝒃_{n, n^{d}})$ . From (11), we observe that $m_{K_{n}}^{ψ} \in ℳ_{S}^{ψ}$ , with $K_{n} = \sum_{k = 1}^{n^{d}} n_{k}$ .

To obtain (7), we write

	${‖ η_{n} - m_{K_{n}}^{ψ} ‖}_{p}$	$= ∥ \sum_{k = 1}^{n^{d}} {Gate}_{k} (𝒙; 𝜸_{n}) f (𝒚 \| 𝒙_{k}^{n}) - \sum_{k = 1}^{n^{d}} {Gate}_{k} (𝒙; 𝜸) h_{n_{k}}^{k} (𝒚 \| 𝒙_{k}^{n}) ∥_{p}$
		$\leq \sum_{k = 1}^{n^{d}} ∥ {Gate}_{k} (𝒙; 𝜸_{n}) [f (𝒚 \| 𝒙_{k}^{n}) - h_{n_{k}}^{k} (𝒚 \| 𝒙_{k}^{n})] ∥_{p} .$

By separability and Fubini’s theorem, we then have

	$‖ η_{n} - m_{K_{n}}^{ψ} ‖$	$\leq \sum_{k = 1}^{n^{d}} ∥ {Gate}_{k} (𝒙; 𝜸_{n}) ∥_{p, 𝕏} ∥ f (𝒚 \| 𝒙_{k}^{n}) - h_{n_{k}}^{k} (𝒚 \| 𝒙_{k}^{n}) ∥_{p, 𝕐}$
		$\leq sup_{k \in [n^{d}]} ∥ {Gate}_{k} (𝒙; 𝜸_{n}) ∥_{p, 𝕏} \sum_{k = 1}^{n^{d}} ∥ f (𝒚 \| 𝒙_{k}^{n}) - h_{n_{k}}^{k} (𝒚 \| 𝒙_{k}^{n}) ∥_{p, 𝕐} .$

Let $C_{2} (n, 𝜸_{n}) = {sup}_{k \in [n^{d}]} {‖ {Gate}_{k} (𝒙; 𝜸_{n}) ‖}_{p, 𝕏}$ . Then, we apply Lemma 4 $n^{d}$ times to establish the existence of a constant $N_{2} (ϵ, n, 𝜸_{n}) \in ℕ$ , such that for all $k \in [n^{d}]$ and $n_{k} \geq N_{2} (ϵ, n, 𝜸_{n})$ ,

∥ f (𝒚 | 𝒙_{k}^{n}) - h_{n_{k}}^{k} (𝒚 | 𝒙_{k}^{n}) ∥_{p, 𝕐} \leq \frac{ϵ}{3 ​ C_{2} ​ (n, 𝜸_{n}) ​ n^{d}} .

Thus, we have

‖ η_{n} - m_{K_{n}}^{ψ} ‖ \leq C_{2} ​ (n, 𝜸_{n}) \times n^{d} \times \frac{ϵ}{3 ​ C_{2} ​ (n, 𝜸_{n}) ​ n^{d}} = \frac{ϵ}{3} ​,

which completes our proof.

4.2 Proof of Theorem 2

The proof is procedurally similar to that of Theorem 1 and thus we only seek to highlight the important differences. Firstly, for any $ϵ > 0$ , we approximate $f (𝒚 | 𝒙)$ by $υ_{n} (𝒙 | 𝒚)$ of form (1), with $d = 1$ . Result (2) implies uniform convergence, in the sense that there exists an $N_{1} (ϵ) \in ℕ$ , such that for all $n \geq N_{1} (ϵ)$ ,

{‖ f - υ_{n} ‖}_{ℬ} < \frac{ϵ}{3} ​ .

(12)

We now seek to approximate $υ_{n}$ by $η_{n}$ of form (3), with $𝜸_{n} = 𝜸_{l}$ for some $l \in ℕ$ . Upon application of Lemma 3, it follows that for each $k \in [n^{d}]$ and $ε > 0$ , there exists a measurable set $𝔹_{k} (ε) \subseteq 𝕏$ , such that

λ ​ (𝔹_{k} ​ (ε)) < \frac{ε}{n^{d} ​ λ ​ (𝕐)}

and

{‖ {Gate}_{k} ​ (\cdot; 𝜸_{l}) - 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} ‖}_{ℬ ​ (𝔹_{k}^{∁} ​ (ε))} < \frac{ϵ}{3}

for all $l \geq M_{k} (ϵ, n)$ , for some $M_{k} (ϵ, n) \in ℕ$ . Here, ${(\cdot)}^{∁}$ is the set complement operator.

Since $f \in ℬ$ , we have the bound $C (n) = \sum_{k = 1}^{n^{d}} ∥ f (𝒚 | 𝒙_{k}^{n}) ∥_{ℬ (𝕐)} < \infty$ . Write $𝔹 (ε) = ⋃_{k = 1}^{n^{d}} 𝔹_{k} (ε)$ . Then, $𝔹^{∁} (ε) = ⋂_{k = 1}^{n^{d}} 𝔹_{k}^{∁} (ε)$ ,

λ ​ (𝔹 ​ (ε)) \leq \sum_{k = 1}^{n^{d}} λ ​ (𝔹_{k}^{∁} ​ (ε)) < \frac{ε}{λ ​ (𝕐)} ​,

and

{‖ {Gate}_{k} ​ (\cdot; 𝜸_{l}) - 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} ‖}_{ℬ ​ (𝔹^{∁} ​ (ε))}

\leq \min_{k \in [n^{d}]} {‖ {Gate}_{k} ​ (\cdot; 𝜸_{l}) - 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} ‖}_{ℬ ​ (𝔹_{k}^{∁} ​ (ε))} < \frac{ϵ}{3 ​ C ​ (n)} ​,

for all $l \geq M (ϵ, n) = \max_{k \in [n^{d}]} M_{k} (ϵ, n)$ . Here we use the fact that the supremum over some intersect of sets is less than or equal to the minimum of the supremum over each individual set.

Upon defining $ℂ (ε) = 𝔹 (ε) \times 𝕐 \subset ℤ$ , we observe that

λ ​ (ℂ ​ (ε)) = λ ​ (𝔹 ​ (ε)) ​ λ ​ (𝕐) \leq \frac{ε}{λ ​ (𝕐)} \times λ ​ (𝕐) = ε ​,

and $ℂ (ε) \subset 𝔹 (ε) \times 𝕐$ . Note also that

{(𝔹 ​ (ε) \times 𝕐)}^{∁} = ℤ \ (𝔹 ​ (ε) \times 𝕐) = 𝔹^{∁} ​ (ε) \times 𝕐

and

ℂ^{∁} ​ (ε) = (𝔹^{∁} ​ (ε) \times 𝕐) \cup (𝔹 ​ (ε) \times 𝕐^{∁}) \cup (𝔹^{∁} ​ (ε) \times 𝕐^{∁}) ​ .

It follows that

{‖ υ_{n} - η_{n} ‖}_{ℬ ​ (ℂ^{∁} ​ (ε))} \leq \max {{‖ υ_{n} - η_{n} ‖}_{ℬ ​ (𝔹^{∁} ​ (ε) \times 𝕐)}, {‖ υ_{n} - η_{n} ‖}_{ℬ ​ (𝔹 ​ (ε) \times 𝕐^{∁})}, {‖ υ_{n} - η_{n} ‖}_{ℬ ​ (𝔹^{∁} ​ (ε) \times 𝕐^{∁})}} ​ .

Since $𝔹 (ε) \times 𝕐^{∁}$ and $𝔹^{∁} (ε) \times 𝕐^{∁}$ are empty, via separability, we have

	${‖ υ_{n} - η_{n} ‖}_{ℬ (ℂ^{∁} (ε))}$	$= {‖ υ_{n} - η_{n} ‖}_{ℬ (𝔹^{∁} (ε) \times 𝕐)}$
		$= sup_{𝒛 \in 𝔹^{∁} (ε) \times 𝕐} \| \sum_{k = 1}^{n^{d}} [𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{l})] f (𝒚 \| 𝒙_{k}^{n}) \|$
		$\leq sup_{𝒛 \in 𝔹^{∁} (ε) \times 𝕐} \sum_{k = 1}^{n^{d}} \| 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{l}) \| f (𝒚 \| 𝒙_{k}^{n})$
		$\leq \sum_{k = 1}^{n^{d}} sup_{𝒛 \in 𝔹^{∁} (ε) \times 𝕐} \| 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{l}) \| f (𝒚 \| 𝒙_{k}^{n})$
		$= \sum_{k = 1}^{n^{d}} ∥ 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{l}) ∥_{ℬ (𝔹^{∁} (ε))} ∥ f (𝒚 \| 𝒙_{k}^{n}) ∥_{ℬ (𝕐)}$
		$\leq sup_{k \in [n]} ∥ 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{l}) ∥_{ℬ (𝔹^{∁} (ε))} \sum_{k = 1}^{n^{d}} ∥ f (𝒚 \| 𝒙_{k}^{n}) ∥_{ℬ (𝕐)} .$

Recall that the $\sum_{k = 1}^{n^{d}} ∥ f (𝒚 | 𝒙_{k}^{n}) ∥_{ℬ (𝕐)} = C (n) < \infty$ and that we can choose $l \geq M (ϵ, n)$ so that

sup_{k \in [n]} {‖ 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} ​ (𝒙; 𝜸_{l}) ‖}_{ℬ ​ (𝔹^{∁} ​ (ε))} < \frac{ϵ}{3 ​ C ​ (n)} ​,

and thus

{‖ υ_{n} - η_{n} ‖}_{ℬ ​ (ℂ^{∁} ​ (ε))} < \frac{ϵ}{3 ​ C ​ (n)} \times C ​ (n) = \frac{ϵ}{3} ​,

(13)

as required.

Finally, by noting that for each $k \in [n^{d}]$ , both (6) and $f (\cdot | 𝒙_{k}^{n})$ are continuous over $𝕐$ , we apply Lemma 4 to obtain an $N_{2} (ϵ, n, l) \in ℕ$ , such that for any $ϵ > 0$ and $n_{k} \geq N_{2} (ϵ, n, l)$ , we have

∥ f (\cdot | 𝒙_{k}^{n}) - h_{n_{k}}^{k} (\cdot | 𝒙_{k}^{n}) ∥_{ℬ ​ (𝕐)} < \frac{ϵ}{3 ​ M_{1} ​ n} .

Here $M_{1} = {sup}_{k \in [n^{d}]} {‖ {Gate}_{k} (\cdot; 𝜸_{l}) ‖}_{ℬ (𝕏)} < \infty$ , since ${Gate}_{k} (𝒙; 𝜸_{l})$ is continuous in $𝒙$ , and $𝕏$ is compact. Therefore, for all $K_{n} = \sum_{k = 1}^{n^{d}} n_{k}$ , $n_{k} \geq N_{2} (ϵ, n, l)$ ,

	${‖ η_{n} - m_{K_{n}}^{ψ} ‖}_{ℬ}$	$\leq sup_{k \in [n^{d}]} ∥ {Gate}_{k} (𝒙; 𝜸_{l}) ∥_{ℬ (𝕏)} \sum_{k = 1}^{n^{d}} ∥ f (\cdot \| 𝒙_{k}^{n}) - h_{n_{k}}^{k} (\cdot \| 𝒙_{k}^{n}) ∥_{ℬ (𝕐)}$
		$= M_{1} \times n^{d} \times \frac{ϵ}{3 M_{1} n^{d}} = \frac{ϵ}{3} .$		(14)

In summary, via (12), (13), and (14), for each $ϵ > 0$ , for any $ε > 0$ , there exists a $ℂ (ε) \subset ℤ$ and constants $N_{1} (ϵ), M (ϵ, n), N_{2} (ϵ, n, l) \in ℕ$ , such that for all $K_{n} = \sum_{k = 1}^{n^{d}} n_{k}$ , with $n_{k} \geq N_{2} (ϵ, n, l)$ , $l \geq M (ϵ, n)$ , and $n \geq N_{1} (ϵ)$ , it follows that $λ (ℂ (ε)) < ε$ , and

	${‖ f - m_{K_{n}}^{ψ} ‖}_{ℬ (ℂ^{∁} (ε))}$	$\leq {‖ f - υ_{n} ‖}_{ℬ (ℂ^{∁} (ε))} + {‖ υ_{n} - η_{n} ‖}_{ℬ (ℂ^{∁} (ε))} + {‖ η_{n} - m_{K_{n}}^{ψ} ‖}_{ℬ (ℂ^{∁} (ε))}$
		$\leq {‖ f - υ_{n} ‖}_{ℬ} + {‖ υ_{n} - η_{n} ‖}_{ℬ (ℂ^{∁} (ε))} + {‖ η_{n} - m_{K_{n}}^{ψ} ‖}_{ℬ}$
		$< 3 \times \frac{ϵ}{3} = ϵ .$

This completes the proof.

5 Proofs of lemmas

5.1 Proof of Lemma 1

We firstly prove that any gating vector from $𝒢_{S}^{K}$ can be equivalently represented as an element of $𝒢_{G}^{K}$ . For any $𝒙 \in ℝ^{d}$ , $d \in ℕ$ , $k \in [K]$ , $a_{k} \in ℝ$ , $𝒃_{k} \in ℝ^{d}$ , and $K \in ℕ$ , choose $𝝂_{k} = 𝒃_{k}$ , $τ_{k} = a_{k} + 𝒃_{k}^{⊤} 𝒃_{k} / 2$ and

π_{k} = \exp (τ_{k}) / \sum_{l = 1}^{K} \exp (τ_{l}) ​ .

This implies that $\sum_{l = 1}^{K} π_{l} = 1$ , $π_{l} > 0$ , for all $l \in [K]$ , and

	$\frac{\exp (a_{k} + 𝒃_{k}^{⊤} 𝒙)}{\sum_{l = 1}^{K} \exp (a_{k} + 𝒃_{k}^{⊤} 𝒙)}$	$= \frac{\exp (τ_{k} - 𝒗_{k}^{⊤} 𝒗_{k} / 2 + 𝒗_{k}^{⊤} 𝒙)}{\sum_{l = 1}^{K} \exp (τ_{l} - 𝒗_{l}^{⊤} 𝒗_{l} / 2 + 𝒗_{l}^{⊤} 𝒙)}$
		$= \frac{\exp (τ_{k}) \exp (- {(𝒙 - 𝝂_{k})}^{⊤} (𝒙 - 𝝂_{k}) / 2)}{\sum_{l = 1}^{K} \exp (τ_{l}) \exp (- {(𝒙 - 𝝂_{l})}^{⊤} (𝒙 - 𝝂_{l}) / 2)}$
		$= \frac{π_{k} {(2 π)}^{- d / 2} \exp (- {(𝒙 - 𝝂_{k})}^{⊤} (𝒙 - 𝝂_{k}) / 2)}{\sum_{l = 1}^{K} π_{l} {(2 π)}^{- d / 2} \exp (- {(𝒙 - 𝝂_{l})}^{⊤} (𝒙 - 𝝂_{l}) / 2)}$
		$= \frac{π_{k} ϕ (𝒙; 𝝂_{k}, 𝐈)}{\sum_{l = 1}^{K} π_{l} ϕ (𝒙; 𝝂_{l}, 𝐈)} ,$

where $𝐈$ is the identity matrix of appropriate size. This proves that $𝒢_{S}^{K} \subset 𝒢_{G}^{K}$ .

Next, to show that $𝒢_{E}^{K} \subset 𝒢_{S}^{K}$ , we write

		$\frac{π_{k} ϕ (𝒙; 𝝂_{k}, 𝚺)}{\sum_{l = 1}^{K} π_{l} ϕ (𝒙; 𝝂_{l}, 𝚺)}$
	$=$	$\frac{π_{k} {\| 2 π 𝚺 \|}^{- 1 / 2} \exp (- {(𝒙 - 𝝂_{k})}^{⊤} 𝚺^{- 1} (𝒙 - 𝝂_{k}) / 2)}{\sum_{l = 1}^{K} π_{l} {\| 2 π 𝚺 \|}^{- 1 / 2} \exp (- {(𝒙 - 𝝂_{l})}^{⊤} 𝚺^{- 1} (𝒙 - 𝝂_{l}) / 2)}$
	$=$	$\frac{1}{\sum_{l = 1}^{K} \exp (- \log (π_{l}^{- 2} / π_{k}^{- 2}) / 2 - {(𝒙 - 𝝂_{l})}^{⊤} 𝚺^{- 1} (𝒙 - 𝝂_{l}) / 2 - {(𝒙 - 𝝂_{k})}^{⊤} 𝚺^{- 1} (𝒙 - 𝝂_{k}) / 2)} ,$

and note that

		${(𝒙 - 𝝂_{l})}^{⊤} 𝚺^{- 1} (𝒙 - 𝝂_{l}) - {(𝒙 - 𝝂_{k})}^{⊤} 𝚺^{- 1} (𝒙 - 𝝂_{k})$
	$=$	$- 2 {(𝝂_{l} - 𝝂_{k})}^{⊤} 𝚺^{- 1} 𝒙 + {(𝝂_{l} + 𝝂_{k})}^{⊤} 𝚺^{- 1} (𝝂_{l} - 𝝂_{k}) .$

Thus, we have

		$\frac{π_{k} ϕ (𝒙; 𝝂_{k}, 𝚺)}{\sum_{l = 1}^{K} π_{l} ϕ (𝒙; 𝝂_{l}, 𝚺)}$
	$=$	$\frac{1}{\sum_{l = 1}^{K} \exp (- \log (π_{l}^{- 2} / π_{k}^{- 2}) / 2 - {(𝝂_{l} + 𝝂_{k})}^{⊤} 𝚺^{- 1} (𝝂_{l} - 𝝂_{k}) / 2 - {(𝝂_{l} - 𝝂_{k})}^{⊤} 𝚺^{- 1} 𝒙)} .$

Next, notice that we can write

\frac{\exp (a_{k} + 𝒃_{k}^{⊤} ​ 𝒙)}{\sum_{l = 1}^{K} \exp (a_{l} + 𝒃_{l}^{⊤} ​ 𝒙)} = \frac{1}{\sum_{l = 1}^{K} \exp (α_{l} + 𝜷_{l}^{⊤} ​ 𝒙)} ​,

where $α_{l} = a_{l} - a_{k}$ and $𝜷_{l} = 𝜷_{l} - 𝜷_{k}$ . We now choose $a_{k}$ and $𝒃_{k}$ , such that for every $l \in [K]$ ,

α_{l}

= a_{l} - a_{k} = - \frac{1}{2} ​ \log (\frac{π_{l}^{- 2}}{π_{k}^{- 2}}) - \frac{1}{2} ​ (𝝂_{l}^{⊤} ​ 𝚺^{- 1} ​ 𝝂_{l} - 𝝂_{k}^{⊤} ​ 𝚺^{- 1} ​ 𝝂_{k}) ​,

and

𝜷_{l} = 𝜷_{l} - 𝜷_{k} = 𝝂_{l}^{⊤} ​ 𝚺^{- 1} - 𝝂_{k}^{⊤} ​ 𝚺^{- 1} ​ .

To complete the proof, we choose

a_{k} = \log (π_{k}) - \frac{1}{2} ​ 𝝂_{k}^{⊤} ​ 𝚺^{- 1} ​ 𝝂_{k}

and $b_{k} = 𝝂_{k}^{⊤} 𝚺^{- 1}$ , for each $k \in [K]$ .

5.2 Proof of Lemma 3

For $l \in [0, \infty)$ , write

{Gate}_{k} ​ (x, l) = \frac{\exp ([x - c_{k}] ​ l ​ k)}{\sum_{i = 1}^{n} \exp ([x - c_{i}] ​ l ​ i)} ​,

where $x \in 𝕏 = [0, 1]$ , and $c_{k} = (k - 1) / (2 k)$ . We identify that $𝐆𝐚𝐭𝐞 = {({Gate}_{k} (x, l))}_{k \in [n]}$ belongs to the class $𝒢_{S}^{n}$ . The proof of the Section 4 Proposition from Jiang & Tanner, 1999b reveals that for all $k \in [n]$ ,

{Gate}_{k} ​ (x, l) \to 𝟏_{{x \in 𝕀_{k}^{n}}}

almost everywhere in $λ$ , as $l \to \infty$ . The result then follows via an application of Egorov’s theorem (cf. Folland, 1999, Thm. 2.33).

6 Summary and conclusions

Using recent results mixture model approximation results Nguyen et al., 2020a and Nguyen et al., 2020c , and the indicator approximation theorem of Jiang & Tanner, 1999b (cf. Section 3), we have proved two approximation theorems (Theorems 1 and 2) regarding the class of soft-max gated MoE models with experts arising from arbitrary location-scale families of conditional density functions. Via an equivalence result (Lemma 1), the results of Theorems 1 and 2 also extend to the setting of Gaussian gated MoE models (Corollary 1), which can be seen as a generalization of the soft-max gated MoE models.

Although we explicitly make the assumption that $𝕏 = {[0, 1]}^{d}$ , for the sake of mathematical argument (so that we can make direct use of Lemma 2), a simple shift-and-scale argument can be used to generalize our result to cases where $𝕏$ is any generic compact domain. The compactness assumption regarding the input domain is common in the MoE and mixture of regression models literature, as per the works of Jiang & Tanner, 1999a , Jiang & Tanner, 1999b , Norets, (2010), Montuelle & Le Pennec, (2014), Pelenis, (2014), Devijver, 2015a , and Devijver, 2015b .

The assumption permits the application of the result to the settings where the inputs $𝑿$ is assumed to be non-random design vectors that take value on some compact set $𝕏$ . This is often the case when there is only a finite number of possible design vector elements for which $𝑿$ can take. Otherwise, the assumption also permits the scenario where $𝑿$ is some random element with compactly supported distribution, such as uniformly distributed, or beta distributed inputs. Unfortunately, the case of random $𝑿$ over an unbounded domain (e.g., if $𝑿$ has multivariate Gaussian distribution) is not covered under our framework. An extension to such cases would require a more general version of Lemma 2, which we believe is a nontrivial direction for future work.

Like the input, we also assume that the output domain is restricted to a compact set $𝕐$ . However, the output domain of the approximating class of MoE models is unrestricted to $𝕐$ and thus the functions (i.e., we allow $ψ$ to be a PDF over $ℝ^{q}$ ). The restrictions placed on $𝕐$ is also common in the mixture approximation literature, as per the works of Zeevi & Meir, (1997), Li & Barron, (1999), and Rakhlin et al., (2005), and is also often made in the context of nonparametric regression (see, e.g., Gyorfi et al., 2002 and Cucker & Zhou, 2007). Here, our use of the compactness of $𝕐$ is to bound the integral of $v_{n}$ , in (10). A more nuanced approach, such as via the use of a generalized Lebesgue spaces (see e.g., Castillo & Rafeiro, 2010 and Cruze-Uribe & Fiorenza, 2013), may lead to result for unbounded $𝕐$ . This is another exciting future direction of our research program.

A trivial modification to the proof of Lemma 4 allows us to replace the assumption that $f$ is a PDF with a sub-PDF assumption (i.e., $\int_{𝕐} f d λ \leq 1$ ), instead. This in turn permits us to replace the assumption that $f (\cdot | 𝒙)$ is a conditional PDF in Theorems 1 and 2 with sub-PDF assumptions as well (i.e., for each $𝒙 \in 𝕏$ , $\int_{𝕐} f (𝒚 | 𝒙) d λ (𝒚) \leq 1$ ). Thus, in this modified form, we have a useful interpretation for situations when the input $𝒀$ is unbounded. That is, when $𝒀$ is unbounded, we can say that the conditional PDF $f$ can be arbitrarily well approximated in $ℒ_{p}$ norm by a sequence ${m_{K}^{ψ}}_{K \in ℕ}$ of either soft-max or Gaussian gated MoEs over any compact subdomain $𝕐$ of the unbounded domain of $𝒀$ . Thus, although we cannot provide guarantees of the entire domain of $𝒀$ , we are able to guarantee arbitrary approximate fidelity over any arbitrarily large compact subdomain. This is a useful result in practice since one is often not interested in the entire domain of $𝒀$ , but only on some subdomain where the probability of $𝒀$ is concentrated. This version of the result resembles traditional denseness results in approximation theory, such as those of Cheney & Light, (2000, Ch. 20).

Finally, our results can be directly applied to provide approximation guarantees for a large number of currently used models in applied statistics and machine learning research. Particularly, our approximation guarantees are applicable to the recent MoE models of Ingrassia et al., (2012), Chamroukhi et al., (2013), Ingrassia et al., (2014), Chamroukhi, (2016), Nguyen & McLachlan, (2016), Deleforge et al., 2015b , Deleforge et al., 2015a , Kalliovirta et al., (2016), and Perthame et al., (2018), among many others. Here, we may guarantee that the underlying data generating processes, if satisfying our assumptions, can be adequately well approximated by sufficiently complex forms of the models considered in each of the aforementioned work.

The rate and manner of which good approximation can be achieved as a function of the number of experts $K$ and the sample size is a currently active research area, with pioneering work conducted in Cohen & Le Pennec, (2012) and Montuelle & Le Pennec, (2014). More recent results in this direction appear in Nguyen et al., 2020b , Nguyen et al., 2021b , and Nguyen et al., 2021a .

List of abbreviations

MoE: Mixture of experts
PDF: Probability density function

Acknowledgements

Hien Duy Nguyen and Geoffrey John McLachlan are funded by Australian Research Council grants: DP180101192 and IC170100035. TrungTin Nguyen is supported by a “Contrat doctoral” from the French Ministry of Higher Educationand Research. Faicel Chamroukhi is granted by the French National Research Agency (ANR) grant SMILES ANR-18-CE40-0014. The authors also thank the Editor and Reviewer, whose careful and considerate comments lead to improvements in of text.

Declarations

Funding: HDN and GJM are funded by Australian Research Council grants: DP180101192 and IC170100035. FC is funded by ANR grant: SMILES ANR-18-CE40-0014.
Conflicts of interest: None.
Availability of data and material: Not applicable.
Authors’ contributions: All authors contributed equally to the exposition and to the mathematical derivations.
Code availability: Not applicable.

References

Bartle, (1995) Bartle, R. (1995). The Elements of Integration and Lebesgue Measure. New York: Wiley. ↩ 2
Castillo & Rafeiro, (2010) Castillo, R. E. & Rafeiro, H. (2010). An Introductory Course in Lebesgue Spaces. Switzerland: Springer. ↩ 6
Chamroukhi, (2016) Chamroukhi, F. (2016). Robust mixture of experts modeling using the t distribution. Neural Networks, 79, 20–36. ↩ 6
Chamroukhi et al., (2013) Chamroukhi, F., Mohammed, S., Trabelsi, D., Oukhellou, L., & Amirat, Y. (2013). Joint segmentation of multivariate time series with hidden process regression for human activity recognition. Neurocomputing, 120, 633–644. ↩ 6
Cheney & Light, (2000) Cheney, W. & Light, W. (2000). A Course in Approximation Theory. Pacific Grove: Brooks/Cole. ↩ 6
Cohen & Le Pennec, (2012) Cohen, S. & Le Pennec, E. (2012). Conditional density estimation by penalized likelihood model selection and application. ArXiv, (arXiv:1103.2021). ↩ 6
Cruze-Uribe & Fiorenza, (2013) Cruze-Uribe, D. V. & Fiorenza, A. (2013). Variable Lebesgue Spaces: Foundations and Harmonic Analysis. Basel: Birkhauser. ↩ 6
Cucker & Zhou, (2007) Cucker, F. & Zhou, D.-X. (2007). Learning Theory: An Approximation Theory Viewpoint. Cambridge: Cambridge University Press. ↩ 6
(9) Deleforge, A., Forbes, F., & Horaud, R. (2015a). Acoustic space learning for sound-source separation and localization on binaural manifolds. International Journal of Neural Systems, 25, 1440003. ↩ 6
(10) Deleforge, A., Forbes, F., & Horaud, R. (2015b). High-dimensional regression with Gaussian mixtures and partially-latent response variables. Statistics and Computing, 25, 893–911. ↩ 6
(11) Devijver, E. (2015a). An $ℓ_{1}$ -oracle inequality for the Lasso in multivariate finite mixture of multivariate Gaussian regression models. ESAIM: Probability and Statistics, 19, 649–670. ↩ 6
(12) Devijver, E. (2015b). Finite mixture regression: a sparse variable selection by model selection for clustering. Electronic Journal of Statistics, 9, 2642–2674. ↩ 6
Folland, (1999) Folland, G. B. (1999). Real Analysis: Modern Techniques and Their Applications. New York: Wiley. ↩ 5.2
Gyorfi et al., (2002) Gyorfi, L., Kohler, M., Krzyzak, A., & Walk, H. (2002). A Distribution-free Theory Of Nonparametric Regression. New York: Springer. ↩ 6
Ingrassia et al., (2014) Ingrassia, S., Minotti, S. C., & Punzo, A. (2014). Model-based clustering via linear cluster-weighted models. Computational Statistics and Data Analysis, 71, 159–182. ↩ 6
Ingrassia et al., (2012) Ingrassia, S., Minotti, S. C., & Vittadini, G. (2012). Local statistical modeling via a cluster-weighted approach with elliptical distributions. Journal of Classification, 29, 363–401. ↩ 6
Jacobs et al., (1991) Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation, 3, 79–87. ↩ 1
(18) Jiang, W. & Tanner, M. A. (1999a). Hierachical mixtures-of-experts for exponential family regression models: approximation and maximum likelihood estimation. Annals of Statistics, 27, 987–1011. ↩ 1 6
(19) Jiang, W. & Tanner, M. A. (1999b). On the approximation rate of hierachical mixtures-of-experts for generalized linear models. Neural Computation, 11, 1183–1198. ↩ 1 3 5.2 6
Jordan & Xu, (1995) Jordan, M. I. & Xu, L. (1995). Convergence results for the EM approach to mixtures of experts architectures. Neural Networks, 8, 1409–1431. ↩ 1
Kalliovirta et al., (2016) Kalliovirta, L., Meitz, M., & Saikkonen, P. (2016). Gaussian mixture vector autoregression. Journal of Econometrics, 192, 485–498. ↩ 6
Krzyzak & Schafer, (2005) Krzyzak, A. & Schafer, D. (2005). Nonparametric regression estimation by normalized radial basis function networks. IEEE Transactions on Information Theory, 51, 1003–1010. ↩ 1
Li & Barron, (1999) Li, J. Q. & Barron, A. R. (1999). Mixture density estimation. In S. A. Solla, T. K. Leen, & K. R. Mueller (Eds.), Advances in Neural Information Processing Systems, volume 12 Cambridge: MIT Press. ↩ 6
Masoudnia & Ebrahimpour, (2014) Masoudnia, S. & Ebrahimpour, R. (2014). Mixture of experts: a literature survey. Artificial Intelligence Review, 42, 275–293. ↩ 1
Mendes & Jiang, (2012) Mendes, E. F. & Jiang, W. (2012). On convergence rates of mixture of polynomial experts. Neural Computation, 24. 3025-3051. ↩ 1
Montuelle & Le Pennec, (2014) Montuelle, L. & Le Pennec, E. (2014). Mixture of Gaussian regressions model with logistic weights, a penalized maximum likelihood approach. Electronic Journal of Statistics, 8, 1661–1695. ↩ 6
Nguyen & Chamroukhi, (2018) Nguyen, H. D. & Chamroukhi, F. (2018). Practical and theoretical aspects of mixture-of-experts modeling: an overview. WIREs Data Mining and Knowledge Discovery, (pp. e1246). ↩ 1
Nguyen et al., (2019) Nguyen, H. D., Chamroukhi, F., & Forbes, F. (2019). Approximation results regarding the multiple-output Gaussian gated mixture of linear experts model. Neurocomputing, 366, 208–214. ↩ 1
Nguyen et al., (2016) Nguyen, H. D., Lloyd-Jones, L. R., & McLachlan, G. J. (2016). A universal approximation theorem for mixture-of-experts models. Neural Computation, 28, 2585–2593. ↩ 1
Nguyen & McLachlan, (2016) Nguyen, H. D. & McLachlan, G. J. (2016). Laplace mixture of linear experts. Computational Statistics and Data Analysis, 93, 177–191. ↩ 6
(31) Nguyen, T. T., Chamroukhi, F., Nguyen, H. D., & Forbes, F. (2021a). Non-asymptotic model selection in block-diagonal mixture of polynomial experts models. arXiv preprint arXiv:2104.08959. ↩ 6
(32) Nguyen, T. T., Chamroukhi, F., Nguyen, H. D., & McLachlan, G. J. (2020a). Approximation of probability density functions via location-scale finite mixtures in Lebesgue spaces. arXiv:2008.09787. ↩ 1 3 6
(33) Nguyen, T. T., Nguyen, H. D., Chamroukhi, F., & Forbes, F. (2021b). A non-asymptotic penalization criterion for model selection in mixture of experts models. ArXiv, (arXiv:2104.02640). ↩ 6
(34) Nguyen, T. T., Nguyen, H. D., Chamroukhi, F., & McLachlan, G. J. (2020b). An $l_{1}$ -oracle inequality for the Lasso in mixture-of-experts regression models. ArXiv, (arXiv:2009.10622). ↩ 6
(35) Nguyen, T. T., Nguyen, H. D., Chamroukhi, F., & McLachlan, G. J. (2020c). Approximation by finite mixtures of continuous density functions that vanish at infinity. Cogent Mathematics and Statistics, 7, 1750861. ↩ 1 6
Norets, (2010) Norets, A. (2010). Approximation of conditional densities by smooth mixtures of regressions. Annals of Statistics, 38, 1733–1766. ↩ 1 6
Norets & Pelenis, (2014) Norets, A. & Pelenis, J. (2014). Posterior consistency in conditional density estimation by covariate dependent mixtures. Econometric Theory, 30, 606–646. ↩ 1
Oden & Demkowicz, (2010) Oden, J. T. & Demkowicz, L. F. (2010). Applied Functional Analysis. Boca Raton: CRC Press. ↩ 4.1
Pelenis, (2014) Pelenis, J. (2014). Bayesian regression with heteroscedastic error density and parametric mean function. Journal of Econometrics, 178, 624–638. ↩ 6
Perthame et al., (2018) Perthame, E., Forbes, F., & Deleforge, A. (2018). Inverse regression approach to robust nonlinear high-to-low dimensional mapping. Journal of Multivariate Analysis, 163, 1–14. ↩ 6
Rakhlin et al., (2005) Rakhlin, A., Panchenko, D., & Mukherjee, S. (2005). Risk bounds for mixture density estimation. ESAIM: Probability and Statistics, 9, 220–229. ↩ 6
Wang & Mendel, (1992) Wang, L.-X. & Mendel, J. M. (1992). Fuzzy basis functions, universal approximation, and orthogonal least-squares learning. IEEE Transactions on Neural Networks, 3, 807–814. ↩ 1
Yuksel et al., (2012) Yuksel, S. E., Wilson, J. N., & Gader, P. D. (2012). Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems, 23, 1177–1193. ↩ 1
Zeevi & Meir, (1997) Zeevi, A. J. & Meir, R. (1997). Density estimation through convex combinations of densities: approximation and estimation bounds. Neural Computation, 10, 99–109. ↩ 1 6
Zeevi et al., (1998) Zeevi, A. J., Meir, R., & Maiorov, V. (1998). Error bounds for functional approximation and estimation using mixtures of experts. IEEE Transactions on Information Theory, 44, 1010–1025. ↩ 1

[bib.bib1] Bartle, (1995) Bartle, R. (1995). The Elements of Integration and Lebesgue Measure. New York: Wiley. ↩ 2

[bib.bib2] Castillo & Rafeiro, (2010) Castillo, R. E. & Rafeiro, H. (2010). An Introductory Course in Lebesgue Spaces. Switzerland: Springer. ↩ 6

[bib.bib3] Chamroukhi, (2016) Chamroukhi, F. (2016). Robust mixture of experts modeling using the t distribution. Neural Networks, 79, 20–36. ↩ 6

[bib.bib4] Chamroukhi et al., (2013) Chamroukhi, F., Mohammed, S., Trabelsi, D., Oukhellou, L., & Amirat, Y. (2013). Joint segmentation of multivariate time series with hidden process regression for human activity recognition. Neurocomputing, 120, 633–644. ↩ 6

[bib.bib5] Cheney & Light, (2000) Cheney, W. & Light, W. (2000). A Course in Approximation Theory. Pacific Grove: Brooks/Cole. ↩ 6

[bib.bib6] Cohen & Le Pennec, (2012) Cohen, S. & Le Pennec, E. (2012). Conditional density estimation by penalized likelihood model selection and application. ArXiv, (arXiv:1103.2021). ↩ 6

[bib.bib7] Cruze-Uribe & Fiorenza, (2013) Cruze-Uribe, D. V. & Fiorenza, A. (2013). Variable Lebesgue Spaces: Foundations and Harmonic Analysis. Basel: Birkhauser. ↩ 6

[bib.bib8] Cucker & Zhou, (2007) Cucker, F. & Zhou, D.-X. (2007). Learning Theory: An Approximation Theory Viewpoint. Cambridge: Cambridge University Press. ↩ 6

[bib.bib9] (9) Deleforge, A., Forbes, F., & Horaud, R. (2015a). Acoustic space learning for sound-source separation and localization on binaural manifolds. International Journal of Neural Systems, 25, 1440003. ↩ 6

[bib.bib10] (10) Deleforge, A., Forbes, F., & Horaud, R. (2015b). High-dimensional regression with Gaussian mixtures and partially-latent response variables. Statistics and Computing, 25, 893–911. ↩ 6

[bib.bib11] (11) Devijver, E. (2015a). An $ℓ_{1}$ -oracle inequality for the Lasso in multivariate finite mixture of multivariate Gaussian regression models. ESAIM: Probability and Statistics, 19, 649–670. ↩ 6

[bib.bib12] (12) Devijver, E. (2015b). Finite mixture regression: a sparse variable selection by model selection for clustering. Electronic Journal of Statistics, 9, 2642–2674. ↩ 6

[bib.bib13] Folland, (1999) Folland, G. B. (1999). Real Analysis: Modern Techniques and Their Applications. New York: Wiley. ↩ 5.2

[bib.bib14] Gyorfi et al., (2002) Gyorfi, L., Kohler, M., Krzyzak, A., & Walk, H. (2002). A Distribution-free Theory Of Nonparametric Regression. New York: Springer. ↩ 6

[bib.bib15] Ingrassia et al., (2014) Ingrassia, S., Minotti, S. C., & Punzo, A. (2014). Model-based clustering via linear cluster-weighted models. Computational Statistics and Data Analysis, 71, 159–182. ↩ 6

[bib.bib16] Ingrassia et al., (2012) Ingrassia, S., Minotti, S. C., & Vittadini, G. (2012). Local statistical modeling via a cluster-weighted approach with elliptical distributions. Journal of Classification, 29, 363–401. ↩ 6

[bib.bib17] Jacobs et al., (1991) Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation, 3, 79–87. ↩ 1

[bib.bib18] (18) Jiang, W. & Tanner, M. A. (1999a). Hierachical mixtures-of-experts for exponential family regression models: approximation and maximum likelihood estimation. Annals of Statistics, 27, 987–1011. ↩ 1 6

[bib.bib19] (19) Jiang, W. & Tanner, M. A. (1999b). On the approximation rate of hierachical mixtures-of-experts for generalized linear models. Neural Computation, 11, 1183–1198. ↩ 1 3 5.2 6

[bib.bib20] Jordan & Xu, (1995) Jordan, M. I. & Xu, L. (1995). Convergence results for the EM approach to mixtures of experts architectures. Neural Networks, 8, 1409–1431. ↩ 1

[bib.bib21] Kalliovirta et al., (2016) Kalliovirta, L., Meitz, M., & Saikkonen, P. (2016). Gaussian mixture vector autoregression. Journal of Econometrics, 192, 485–498. ↩ 6

[bib.bib22] Krzyzak & Schafer, (2005) Krzyzak, A. & Schafer, D. (2005). Nonparametric regression estimation by normalized radial basis function networks. IEEE Transactions on Information Theory, 51, 1003–1010. ↩ 1

[bib.bib23] Li & Barron, (1999) Li, J. Q. & Barron, A. R. (1999). Mixture density estimation. In S. A. Solla, T. K. Leen, & K. R. Mueller (Eds.), Advances in Neural Information Processing Systems, volume 12 Cambridge: MIT Press. ↩ 6

[bib.bib24] Masoudnia & Ebrahimpour, (2014) Masoudnia, S. & Ebrahimpour, R. (2014). Mixture of experts: a literature survey. Artificial Intelligence Review, 42, 275–293. ↩ 1

[bib.bib25] Mendes & Jiang, (2012) Mendes, E. F. & Jiang, W. (2012). On convergence rates of mixture of polynomial experts. Neural Computation, 24. 3025-3051. ↩ 1

[bib.bib26] Montuelle & Le Pennec, (2014) Montuelle, L. & Le Pennec, E. (2014). Mixture of Gaussian regressions model with logistic weights, a penalized maximum likelihood approach. Electronic Journal of Statistics, 8, 1661–1695. ↩ 6

[bib.bib27] Nguyen & Chamroukhi, (2018) Nguyen, H. D. & Chamroukhi, F. (2018). Practical and theoretical aspects of mixture-of-experts modeling: an overview. WIREs Data Mining and Knowledge Discovery, (pp. e1246). ↩ 1

[bib.bib28] Nguyen et al., (2019) Nguyen, H. D., Chamroukhi, F., & Forbes, F. (2019). Approximation results regarding the multiple-output Gaussian gated mixture of linear experts model. Neurocomputing, 366, 208–214. ↩ 1

[bib.bib29] Nguyen et al., (2016) Nguyen, H. D., Lloyd-Jones, L. R., & McLachlan, G. J. (2016). A universal approximation theorem for mixture-of-experts models. Neural Computation, 28, 2585–2593. ↩ 1

[bib.bib30] Nguyen & McLachlan, (2016) Nguyen, H. D. & McLachlan, G. J. (2016). Laplace mixture of linear experts. Computational Statistics and Data Analysis, 93, 177–191. ↩ 6

[bib.bib31] (31) Nguyen, T. T., Chamroukhi, F., Nguyen, H. D., & Forbes, F. (2021a). Non-asymptotic model selection in block-diagonal mixture of polynomial experts models. arXiv preprint arXiv:2104.08959. ↩ 6

[bib.bib32] (32) Nguyen, T. T., Chamroukhi, F., Nguyen, H. D., & McLachlan, G. J. (2020a). Approximation of probability density functions via location-scale finite mixtures in Lebesgue spaces. arXiv:2008.09787. ↩ 1 3 6

[bib.bib33] (33) Nguyen, T. T., Nguyen, H. D., Chamroukhi, F., & Forbes, F. (2021b). A non-asymptotic penalization criterion for model selection in mixture of experts models. ArXiv, (arXiv:2104.02640). ↩ 6

[bib.bib34] (34) Nguyen, T. T., Nguyen, H. D., Chamroukhi, F., & McLachlan, G. J. (2020b). An $l_{1}$ -oracle inequality for the Lasso in mixture-of-experts regression models. ArXiv, (arXiv:2009.10622). ↩ 6

[bib.bib35] (35) Nguyen, T. T., Nguyen, H. D., Chamroukhi, F., & McLachlan, G. J. (2020c). Approximation by finite mixtures of continuous density functions that vanish at infinity. Cogent Mathematics and Statistics, 7, 1750861. ↩ 1 6

[bib.bib36] Norets, (2010) Norets, A. (2010). Approximation of conditional densities by smooth mixtures of regressions. Annals of Statistics, 38, 1733–1766. ↩ 1 6

[bib.bib37] Norets & Pelenis, (2014) Norets, A. & Pelenis, J. (2014). Posterior consistency in conditional density estimation by covariate dependent mixtures. Econometric Theory, 30, 606–646. ↩ 1

[bib.bib38] Oden & Demkowicz, (2010) Oden, J. T. & Demkowicz, L. F. (2010). Applied Functional Analysis. Boca Raton: CRC Press. ↩ 4.1

[bib.bib39] Pelenis, (2014) Pelenis, J. (2014). Bayesian regression with heteroscedastic error density and parametric mean function. Journal of Econometrics, 178, 624–638. ↩ 6

[bib.bib40] Perthame et al., (2018) Perthame, E., Forbes, F., & Deleforge, A. (2018). Inverse regression approach to robust nonlinear high-to-low dimensional mapping. Journal of Multivariate Analysis, 163, 1–14. ↩ 6

[bib.bib41] Rakhlin et al., (2005) Rakhlin, A., Panchenko, D., & Mukherjee, S. (2005). Risk bounds for mixture density estimation. ESAIM: Probability and Statistics, 9, 220–229. ↩ 6

[bib.bib42] Wang & Mendel, (1992) Wang, L.-X. & Mendel, J. M. (1992). Fuzzy basis functions, universal approximation, and orthogonal least-squares learning. IEEE Transactions on Neural Networks, 3, 807–814. ↩ 1

[bib.bib43] Yuksel et al., (2012) Yuksel, S. E., Wilson, J. N., & Gader, P. D. (2012). Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems, 23, 1177–1193. ↩ 1

[bib.bib44] Zeevi & Meir, (1997) Zeevi, A. J. & Meir, R. (1997). Density estimation through convex combinations of densities: approximation and estimation bounds. Neural Computation, 10, 99–109. ↩ 1 6

[bib.bib45] Zeevi et al., (1998) Zeevi, A. J., Meir, R., & Maiorov, V. (1998). Error bounds for functional approximation and estimation using mixtures of experts. IEEE Transactions on Information Theory, 44, 1010–1025. ↩ 1

	$\| f (𝒚 \| 𝒙) - υ_{n} (𝒚 \| 𝒙) \|$	$= \| \sum_{k = 1}^{n^{d}} 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} [f (𝒚 \| 𝒙) - f (𝒚 \| 𝒙_{k}^{n})] \|$
		$\leq \sum_{k = 1}^{n^{d}} 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} \| f (𝒚 \| 𝒙) - f (𝒚 \| 𝒙_{k}^{n}) \|,$		(9)

	${‖ υ_{n} - η_{n} ‖}_{ℬ (ℂ^{∁} (ε))}$	$= {‖ υ_{n} - η_{n} ‖}_{ℬ (𝔹^{∁} (ε) \times 𝕐)}$
		$= sup_{𝒛 \in 𝔹^{∁} (ε) \times 𝕐} \| \sum_{k = 1}^{n^{d}} [𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{l})] f (𝒚 \| 𝒙_{k}^{n}) \|$
		$\leq sup_{𝒛 \in 𝔹^{∁} (ε) \times 𝕐} \sum_{k = 1}^{n^{d}} \| 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{l}) \| f (𝒚 \| 𝒙_{k}^{n})$
		$\leq \sum_{k = 1}^{n^{d}} sup_{𝒛 \in 𝔹^{∁} (ε) \times 𝕐} \| 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{l}) \| f (𝒚 \| 𝒙_{k}^{n})$
		$= \sum_{k = 1}^{n^{d}} ∥ 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{l}) ∥_{ℬ (𝔹^{∁} (ε))} ∥ f (𝒚 \| 𝒙_{k}^{n}) ∥_{ℬ (𝕐)}$
		$\leq sup_{k \in [n]} ∥ 𝟏_{{𝒙 \in 𝕏_{k}^{n}}} - {Gate}_{k} (𝒙; 𝜸_{l}) ∥_{ℬ (𝔹^{∁} (ε))} \sum_{k = 1}^{n^{d}} ∥ f (𝒚 \| 𝒙_{k}^{n}) ∥_{ℬ (𝕐)} .$