Approximation capabilities of the mixture of experts models
Mar 29, 2020
A central theme of my research focuses on Data Sciences, at the interface of Statistical Learning, Machine Learning, and Optimization.
Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models
Mixture of experts (MoE) models are widely applied for conditional probability density estimation problems. We demonstrate the richness of the class of MoE models by proving denseness results in Lebesgue spaces, when inputs and outputs variables are both compactly supported. We further prove an almost uniform convergence result when the input is univariate. Auxiliary lemmas are proved regarding the richness of the soft-max gating function class, and their relationships to the class of Gaussian gating functions.
Hien Duy Nguyen, TrungTin Nguyen, Faicel Chamroukhi, Geoffrey J McLachlan
Approximation of probability density functions via location-scale finite mixtures in Lebesgue spaces
The class of location-scale finite mixtures is of enduring interest both from applied and theoretical perspectives of probability and statistics. We prove the following results; to an arbitrary degree of accuracy, (a) location-scale mixtures of a continuous probability density function (PDF) can approximate any continuous PDF, uniformly, on a compact set; and (b) for any finite $p \in [1,\infty)$, location-scale mixtures of an essentially bounded PDF can approximate any PDF in $L_p$, in the $L_p$ norm.
TrungTin Nguyen, Faicel Chamroukhi, Hien Duy Nguyen, Geoffrey J McLachlan
Given sufficiently many components, it is often cited that finite mixture models can approximate any other probability density function (PDF) to an arbitrary degree of accuracy. Unfortunately, the nature of this approximation result is often left unclear. We prove that finite mixture models constructed from pdfs in $C_0$ can be used to conduct approximation of various classes of approximands in a number of different modes. That is, we prove approximands in C0 can be uniformly approximated, approximands in $C_b$ can be uniformly approximated on compact sets, and approximands in Lp can be approximated with respect to the $L_p$, for $p\in [1,\infty)$. Furthermore, we also prove that measurable functions can be approximated, almost everywhere.
TrungTin Nguyen, Hien Duy Nguyen, Faicel Chamroukhi, Geoffrey J McLachlan