A Unified Framework for Variable Selection in Model-Based Clustering with Missing Not at Random

Binh H. Ho; Long Nguyen Chi; TrungTin Nguyen; Binh T. Nguyen; Van Ha Hoang; Christopher Drovandi

Abstract

Model-based clustering integrated with variable selection is a powerful tool for uncovering latent structures within complex data. However, its effectiveness is often hindered by challenges such as identifying relevant variables that define heterogeneous subgroups and handling data that are missing not at random, a prevalent issue in fields like transcriptomics. While several notable methods have been proposed to address these problems, they typically tackle each issue in isolation, thereby limiting their flexibility and adaptability. This paper introduces a unified framework designed to address these challenges simultaneously. Our approach incorporates a data-driven penalty matrix into penalized clustering to enable more flexible variable selection, along with a mechanism that explicitly models the relationship between missingness and latent class membership. We demonstrate that, under certain regularity conditions, the proposed framework achieves both asymptotic consistency and selection consistency, even in the presence of missing data. This unified strategy significantly enhances the capability and efficiency of model-based clustering, advancing methodologies for identifying informative variables that define homogeneous subgroups in the presence of complex missing data patterns. The performance of the framework, including its computational efficiency, is evaluated through simulations and demonstrated using both synthetic and real-world transcriptomic datasets. ^†^†^⋆Co-first author, ^†Corresponding author.

1 Introduction

Model-based Clustering. Model-based clustering formulates clustering as a probabilistic inference task, assuming data are generated from a finite mixture model, with each component corresponding to a cluster. This enables likelihood-based estimation and principled model selection. Gaussian mixture models (GMMs) [42, 43] are a classical instance, which can be estimated via the expectation-maximization (EM) algorithm [16], providing soft assignments and flexibility in modeling non-spherical and overlapping clusters. Bayesian mixture models extend this framework by treating parameters, and even the number of components, as random variables, thereby enabling the quantification of uncertainty. Techniques such as Markov chain Monte Carlo (MCMC) and variational inference are employed to sample the posterior distribution, although challenges such as label switching persist [64]. Bayesian nonparametric approaches, including reversible-jump MCMC and birth-death processes, impose prior constraints on model complexity and infer the number of clusters directly. Despite offering interpretability and robustness, especially in small-sample settings, Bayesian methods suffer from high computational costs, label switching, and convergence issues [29]. Therefore, this paper focuses on the maximum likelihood estimation (MLE) approach for dealing with variable selection and missing data in model-based clustering.

Related Works on Variable Selection for Model-based Clustering. Traditional variable selection methods like best-subset and stepwise regression rely on information criteria such as Akaike information criterion (AIC) [1] and Bayesian information criterion (BIC) [58], but become computationally infeasible as dimensionality increases. Penalized likelihood approaches, notably least absolute shrinkage and selection operator (LASSO) [70], address scalability by inducing sparsity; other earlier contributions include ridge regression [28] and the nonnegative garrote [9]. Efficient algorithms like least-angle regression [18] and coordinate descent enable application in high dimensions. To reduce dimensionality before modeling, screening methods such as sure independence screening [19] and its conditional extension [5] filter variables based on marginal or conditional correlations. In clustering, Law et al. [32] propose simultaneous variable selection and clustering using feature saliency. Andrews and McNicholas [3] introduce variable selection for clustering and classification, combining filter and wrapper strategies to discard noisy variables. Bayesian methods such as those by Tadesse et al. [67] and Kim et al. [30] extend variable relevance indicators to mixture models, though MCMC scalability remains a challenge. Raftery and Dean [54] develop a BIC-based stepwise method for identifying clustering-relevant variables, later optimized by Scrucca [59]. Finally, Maugis et al. [38] enrich this framework by incorporating redundancy modeling, enhancing selection accuracy in correlated settings at the cost of increased computational complexity. To overcome this computational complexity, Celeux et al. [13] propose a two-step strategy for selecting variables in mixture models. First, variables are ranked by optimizing a penalized likelihood function that shrinks both component means and precisions, following the approach of Zhou, Pan, and Shen [75]. This method is scalable to moderate dimensions and yields an ordered list where informative variables are expected to appear early. Second, roles are assigned in a single linear pass through the ranked list, replacing combinatorial search with a faster and interpretable procedure that maintains competitive clustering accuracy. However, there is no theoretical guarantee that this ranking recovers the true signal-redundant-uninformative partition.

Handling Missing Data. Missing data are commonly classified into three categories: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). The last category, MNAR, is the most challenging, as the probability of missingness depends on unobserved values. Under the MAR assumption, the method of multiple imputation provides a principled framework by replacing each missing value with several plausible alternatives, then combining analyses across imputed datasets to account for uncertainty [57]. The multivariate imputation by chained equations algorithm [71] is a flexible implementation that fits a sequence of conditional models, accommodating mixed data types. In high-dimensional settings, estimating full joint or conditional models becomes unstable. To address this, regularized regression techniques, such as the LASSO and its Bayesian counterpart, have been integrated into imputation models to improve predictive performance by selecting and shrinking predictors [74]. Nonparametric and machine learning-based imputation methods have also been developed. The MissForest algorithm [63] uses random forests to iteratively impute missing values, capturing nonlinear relations. Deep generative models, such as the missing data importance-weighted autoencoder [36] and its federated variant [4], rely on latent variable representations to generate imputations, assuming data lie near a low-dimensional manifold. For MNAR data, model-based approaches face difficulties due to unidentifiability without external constraints. Two common strategies include selection models, which jointly specify the data and missingness mechanism, and pattern mixture models, which condition on observed missingness patterns. Both frameworks require strong assumptions or instruments to yield valid inferences. To address this challenge, Sportisse et al. [62] studied the identifiability of selection models under the MNAR assumption, particularly when missingness depends on unobserved values. They showed that identifiability can be achieved under structural assumptions, such as requiring each variable’s missingness to be explained either by its value or by the latent class, but not both simultaneously. Their proposed joint modeling framework, termed MNARz, ensures that cluster-specific missingness reflects meaningful distributional differences. A key insight is that under MNARz, the missing data problem can be reformulated as MAR on an augmented data matrix, enabling tractable inference.

Main Contribution. Inspired by the work of Maugis et al. [39], who addressed variable selection under MAR without imputation, we generalize their framework to latent class MNARz. Under the MNARz mechanism, we extend the LASSO-based variable selection framework for model-based clustering proposed by [13], establishing identifiability and consistency guarantees in Section 4 for recovering the true signal-redundant-uninformative partition. Our method achieves substantial empirical improvements in Section 5 over existing approaches. This yields a unified, high-dimensional model-based clustering method that jointly addresses variable selection and MNAR inference in a principled manner.

Paper Organization. The remainder of the paper is structured as follows. Section 2 reviews variable selection in model-based clustering with MNAR data. Section 3 and Section 4 present our proposed framework and its theoretical guarantees, respectively. Simulation and real data results are reported in Section 5. We conclude with a summary, limitations, and future directions in Section 6. Proofs and additional details are provided in the supplementary material.

Notation. Throughout the paper, we use the shorthand $[N]$ to denote the index set ${1, 2, \dots, N}$ for any positive integer $N \in ℕ$ . We denote $ℝ$ as the set of real numbers. The notation $| 𝕊 |$ represents the cardinality of any set $𝕊$ . For any vector $𝒗 \in ℝ^{D}$ , ${‖ 𝒗 ‖}_{p}$ denotes its $p$ -norm. The operator $⊙$ indicates the element-wise (Hadamard) product between two matrices. We use $n \in [N]$ to index observations and $d \in [D]$ to index variables. The vector $𝒚_{n} \in ℝ^{D}$ denotes the full data vector for observation $n$ , which may be further specified in more granular forms. The binary vector $𝒄_{n} = (c_{n 1}, \dots, c_{n D}) \in {0, 1}^{D}$ represents the missingness mask, where $c_{n d} = 1$ if $y_{n d}$ is missing. The latent variable $z_{n} \in [K]$ denotes the mixture component assignment for observation $n$ , with indicator variable $z_{n k} = 𝟏_{{z_{n} = k}}$ . We use $ℓ (\cdot)$ to denote the log-likelihood.

2 Background

We begin by introducing some preliminaries on model-based clustering with variable selection, penalized clustering, and MNARz formulation.

Model-based Clustering and Parsimonious Mixture Form. Model-based clustering adopts a probabilistic framework by assuming that the data matrix $𝐘 = {(𝒚_{1}, \dots, 𝒚_{N})}^{⊤} \in ℝ^{N \times D}$ , with each $𝒚_{n} \in ℝ^{D}$ , is independently drawn from a location-scale mixture distribution, a class known for its universal approximation capabilities and favorable convergence properties [25, 55, 14, 53, 52, 46, 47, 49, 60, 26, 27]. The density of an observation $𝒚_{n}$ under a $K$ -component GMM with parameters $𝜶 = {𝝅, 𝝁, 𝚺}$ is given by: $f_{GMM} (𝒚_{n} ∣ K, m, 𝜶) = \sum_{k = 1}^{K} π_{k} 𝒩 (𝒚_{n} ∣ 𝝁_{k}, 𝚺_{k})$ where $π_{k} > 0$ and $\sum_{k = 1}^{K} π_{k} = 1$ . The term $𝒩 (𝒚_{n} ∣ 𝝁_{k}, 𝚺_{k})$ denotes the multivariate normal density with mean vector $𝝁_{k}$ and covariance matrix $𝚺_{k}$ , which is compactly denoted by $𝚯_{k} = (𝝁_{k}, 𝚺_{k})$ . The covariance matrix $𝚺_{k}$ encodes the structure determined by model form $m$ , allowing for parsimonious modeling via spectral decompositions that control cluster volume, shape, and orientation [12]. These constraints are particularly valuable in high-dimensional settings, where $D ≫ N$ and standard GMMs require estimating $𝒪 (K^{2} D)$ parameters, leading to overparameterization. Model selection criteria such as the BIC [58, 39, 22, 21, 48], slope heuristics [6, 51, 50], integrated classification likelihood (ICL) [7, 24], extended BIC (eBIC) [23, 45], dendrogram selection criterion [17, 69], and Sin-White information criterion (SWIC) [61, 72] can be employed to select both the number of components $K$ and the covariance structure $m$ .

Variable Selection as a Model Selection Problem in Model-based Clustering. In high-dimensional settings, many variables may be irrelevant or redundant with respect to the underlying cluster structure, thereby degrading clustering performance and interpretability. To address this, [38] proposed the $𝕊 ℝ 𝕌 𝕎$ model, extending their earlier work [37], which assigns variables to one of four roles. Let $𝕊$ denote the set of relevant clustering variables. Its complement, $𝕊^{c}$ , comprises the irrelevant variables and is partitioned into subsets $𝕌$ and $𝕎$ . Variables in $𝕌$ are linearly explained by a subset $ℝ \subseteq 𝕊$ , while variables in $𝕎$ are assumed independent of all relevant variables. This framework facilitates variable-specific interpretation and avoids over-penalization from complex covariance structures, as typically encountered in constrained GMMs [12]. The joint density under the $𝕊 ℝ 𝕌 𝕎$ model, combining mixture components for clustering, regression, and independence, is defined as:

f_{𝕊 ​ ℝ ​ 𝕌 ​ 𝕎} ​ (𝒚_{n} ∣ K, m, r, l, 𝕍, 𝚯) = f_{clust} ​ (𝒚_{n}^{𝕊} ∣ K, m, 𝜶) ​ f_{reg} ​ (𝒚_{n}^{𝕌} ∣ r, 𝒂 + 𝒚_{n}^{ℝ} ​ 𝜷, 𝛀) ​ f_{indep} ​ (𝒚_{n}^{𝕎} ∣ l, 𝜸, 𝚪) .

(1)

Here, $𝕍 = (𝕊, 𝕌, ℝ, 𝕎)$ denotes the variable partition, and $𝚯$ is the full parameter set. The components are defined as: $f_{clust} := f_{GMM}$ , $f_{reg} (𝒚_{n}^{𝕌} ∣ r, 𝒂 + 𝒚_{n}^{ℝ} 𝜷, 𝛀) := 𝒩 (𝒚_{n}^{𝕌} ∣ 𝒂 + 𝒚_{n}^{ℝ} 𝜷, 𝛀)$ , with $𝒂 \in ℝ^{1 \times | 𝕌 |}$ , $𝜷 \in ℝ^{| ℝ | \times | 𝕌 |}$ , and $𝛀 \in ℝ^{| 𝕌 | \times | 𝕌 |}$ structured by $r$ . The independent part is $f_{indep} := 𝒩$ , with variance structure $l$ and covariance $𝚪$ . Model selection proceeds by maximizing a BIC-type criterion:

{crit}_{BIC} ​ (K, m, r, l, 𝕍) = {BIC}_{clust} ​ (𝒀^{𝕊} ∣ K, m) + {BIC}_{reg} ​ (𝒀^{𝕌} ∣ r, 𝒀^{ℝ}) + {BIC}_{indep} ​ (𝒀^{𝕎} ∣ l),

(2)

over $(K, m, r, l, 𝕍)$ , where each term scores the corresponding model component. Although this approach offers fine-grained variable treatment, it can be computationally demanding in high dimensions due to stepwise selection for clustering and regression.

Penalized Log-Likelihood Methods for Simultaneous Clustering and Variable Selection. We follow the framework of [11], which extends [75, 13], introducing cluster-specific penalties via group-wise weighting matrices $𝑷_{k}$ for adaptive regularization in Gaussian graphical mixture models. This method improves upon stepwise procedures [38, 37] by handling high-dimensional data more efficiently. The penalized log-likelihood is given by:

\sum_{n = 1}^{N} \ln [\sum_{k = 1}^{K} π_{k} ​ 𝒩 ​ ({\bar{𝒚}}_{n} ∣ 𝝁_{k}, 𝚺_{k})] - λ ​ \sum_{k = 1}^{K} {‖ 𝝁_{k} ‖}_{1} - ρ ​ \sum_{k = 1}^{K} \sum_{d \neq d^{'}} | {(𝑷_{k} ⊙ 𝚺_{k}^{- 1})}_{d ​ d^{'}} | .

(3)

Here, ${\bar{𝒚}}_{n}$ denotes centered and scaled observations, and $𝑷_{k}$ is typically chosen to be inversely proportional to initial partial correlations, enabling adaptive shrinkage akin to the adaptive lasso [77]. Given grids of regularization parameters $𝒢_{λ}$ and $𝒢_{ρ}$ , the EM algorithm from [75] is used to estimate the parameters $\hat{𝜶} (λ, ρ) = (\hat{𝝅} (λ, ρ), {\hat{𝝁}}_{1} (λ, ρ), \dots, {\hat{𝝁}}_{K} (λ, ρ), {\hat{𝚺}}_{1} (λ, ρ), \dots, {\hat{𝚺}}_{K} (λ, ρ))$ . For each variable $d \in [D]$ , clustering scores $𝒪_{K} (d)$ are computed, and the relevant set $𝕊$ is formed from those that improve BIC, while $𝕌$ captures non-informative variables. The final model is selected by maximizing ${crit}_{BIC} (K, m, r, l, 𝕍_{(K, m, r)})$ in Equation 2, where $𝕍_{(K, m, r)} = (𝕊_{(K, m)}, ℝ_{(K, m, r)}, 𝕌_{(K, m)}, 𝕎_{(K, m)})$ .

3 Our Proposal: Variable Selection in Model-Based Clustering with MNAR

We propose a compact framework integrating adaptive variable selection with robust missing data handling in model-based clustering.

Global GMM Representation of $𝕊 ℝ 𝕌 𝕎$ under MAR. A key aspect is expressing the observed-data likelihood in a tractable form. Under the MAR assumption, the $𝕊 ℝ 𝕌 𝕎$ model admits a global GMM representation, extending the result of [39] for the $𝕊 ℝ$ model. Given the density in Equation 1 and Gaussian properties, the observed likelihood becomes

f ​ (𝒀^{o} ∣ K, m, r, l, 𝕍, 𝚯) = \prod_{n = 1}^{N} (\sum_{k = 1}^{K} π_{k} ​ 𝒩 ​ (𝒚_{n}^{o} ∣ {\tilde{𝝂}}_{k, o}, {\tilde{𝚫}}_{k, o ​ o})),

(4)

where ${\tilde{𝝂}}_{k, o} = (\begin{matrix} 𝝂_{k, o} \\ 𝜸_{o} \end{matrix}), {\tilde{𝚫}}_{k, o o} = (\begin{matrix} 𝚫_{k, o o} & 𝟎 \\ 𝟎 & 𝚪_{o o} \end{matrix}) .$ Here, $𝝂_{k, o}, 𝜸_{o}$ and $𝚫_{k, o o}, 𝚪_{o o}$ correspond to the block means and covariances from the $𝕊 ℝ$ model. The derivation extends the technical proof from [39] and is detailed in the supplement. This representation offers: (i) a unified GMM encoding the variable roles in $𝕊 ℝ 𝕌 𝕎$ via observed-data parameters, (ii) compatibility with standard EM algorithms, enabling MAR-aware estimation and internal imputation that respects cluster structure, and (iii) theoretical guarantees of identifiability and consistency in selecting the true variable partition via Equation 2 (see Section 4).

Incorporating MNARz Mechanism into $𝕊 ℝ 𝕌 𝕎$ . To explicitly address MNAR data, we integrate the MNARz mechanism from [62], where missingness depends on the latent class membership. Estimation proceeds via EM, while identifiability and the transformation of MNARz into MAR on augmented data (i.e., original data with the missingness indicator matrix $𝑪$ ) are retained (see Section 4). Let $𝒚_{n} = (𝒚_{n}^{𝕊}, 𝒚_{n}^{𝕌}, 𝒚_{n}^{𝕎})$ denote the full data for observation $n$ , partitioned by role (Section 2), and let $𝒄_{n}$ be its binary missingness pattern. With $z_{n} \in [K]$ denoting the latent class, the complete-data density under $z_{n k} = 1$ is

\begin{matrix} f (𝒚_{n}, 𝒄_{n} ∣ z_{n ​ k} = 1; 𝜶_{k}, 𝝍_{k}) = f_{clust} (𝒚_{n}^{𝕊} ∣ 𝜶_{k}) f_{reg} (𝒚_{n}^{𝕌} ∣ 𝒚_{n}^{ℝ}; 𝜽_{reg}) \\ \times f_{indep} ​ (𝒚_{n}^{𝕎} ∣ 𝜽_{indep}) ​ f_{MNARz} ​ (𝒄_{n} ∣ z_{n ​ k} = 1; 𝝍_{k}), \end{matrix}

where $f_{MNARz} (𝒄_{n} ∣ z_{n k} = 1; 𝝍_{k}) = \prod_{d = 1}^{D} ρ {(𝝍_{k})}^{c_{n d}} {(1 - ρ (𝝍_{k}))}^{1 - c_{n d}}$ and $ρ (𝝍_{k})$ is the class-specific missingness probability. The model parameters are $𝚯 = (π_{1}, \dots, π_{K}, {𝜶_{k}}_{k = 1}^{K}, 𝝃 = {𝜽_{reg}, 𝜽_{indep}}, {𝝍_{k}}_{k = 1}^{K})$ . The observed-data log-likelihood is:

ℓ ​ (𝚯; 𝒀, 𝑪)

= \sum_{n = 1}^{N} \log ​ \sum_{k = 1}^{K} π_{k} ​ f_{k}^{o} ​ (𝒚_{n}^{o}; 𝜶_{k}, 𝝃) ​ f_{c} ​ (𝒄_{n}; 𝝍_{k}),

(5)

with the MNARz mechanism factored out due to independence. The complete-data log-likelihood is:

ℓ_{comp} ​ (𝚯; 𝒀, 𝑪) = \sum_{n = 1}^{N} \sum_{k = 1}^{K} z_{n ​ k} ​ \log (π_{k} ​ f_{k} ​ (𝒚_{n}; 𝜶_{k}, 𝝃) ​ f_{c} ​ (𝒄_{n}; 𝝍_{k})) .

(6)

To accommodate both MAR and MNAR patterns, we extend the model with a binary partition of variable indices: $𝒟_{MAR} \cup 𝒟_{MNAR} = [D], | 𝒟_{MAR} | = D_{M}, | 𝒟_{MNAR} | = D_{M^{'}} .$ Variables in $𝒟_{MAR}$ follow a MAR mechanism, and those in $𝒟_{MNAR}$ follow MNARz:

	$f_{k} (𝒚_{n}; 𝜶_{k}) = f_{clust} (𝒚_{n}^{𝕊} ∣ 𝜶_{k}) f_{reg} (𝒚_{n}^{𝕌} ∣ 𝒚_{n}^{ℝ}; 𝜽_{reg}) f_{indep} (𝒚_{n}^{𝕎} ∣ 𝜽_{indep}), 𝝃 = (𝜽_{reg}, 𝜽_{indep}),$
	$f_{c}^{MAR} (𝒄_{n, M} ∣ 𝒚_{n}^{o}; 𝝍_{M}) = \prod_{d \in 𝒟_{MAR}} ρ_{d} {(𝒚_{n}^{o}; ψ_{M d})}^{c_{n d}} {(1 - ρ_{d} (𝒚_{n}^{o}; ψ_{M d}))}^{1 - c_{n d}},$
	$f_{c}^{MNARz} (𝒄_{n, N}; 𝝍_{k}) = \prod_{d \in 𝒟_{MNAR}} ρ {(𝝍_{k})}^{c_{n d}} {(1 - ρ (𝝍_{k}))}^{1 - c_{n d}} .$

The parameter blocks expand as: $𝚯 = (π_{1}, \dots, π_{K}, {𝜶_{k}}_{k = 1}^{K}, 𝝃, 𝝍_{M}, {𝝍_{k}}_{k = 1}^{K}) .$ Let $𝒚_{n} = (𝒚_{n}^{o}, 𝒚_{n}^{m})$ , and define $𝒄_{n, M} = {(c_{n d})}_{d \in 𝒟_{MAR}}$ , $𝒄_{n, M^{'}} = {(c_{n d})}_{d \in 𝒟_{MNAR}}$ . Then, since the MNARz mask is independent of $𝒚_{n}$ given $k$ , the observed-data likelihood becomes:

ℓ ​ (𝚯; 𝒀, 𝑪) = \sum_{n = 1}^{N} \log [\sum_{k = 1}^{K} π_{k} ​ f_{c}^{MNARz} ​ (𝒄_{n, M^{'}}; 𝝍_{k}) ​ f_{k, M}^{o} ​ (𝒚_{n}^{o}; 𝜶_{k}, 𝝍_{M})] .

(7)

where $f_{k, M}^{o} (𝒚_{n}^{o}; 𝜶_{k}, 𝝍_{M}) := \int f_{k} (𝒚_{n}; 𝜶_{k}) f_{c}^{MAR} (𝒄_{n, M} ∣ 𝒚_{n}^{o}; 𝝍_{M}) 𝑑 𝒚_{n}^{m}$ . The MNARz term factors out of the integral, preserving the ignorability property for that block [62]. Only the MAR component requires integration or imputation.

Adaptive Weighting for Penalty of Precision Matrix. To enhance performance, we replace the inverse partial correlation scheme with a spectral-based computation of $π_{𝐤}$ in Equation 3. Starting from the initial precision matrix ${\hat{𝚿}}_{k}^{(0)}$ , we construct an unweighted, undirected graph $𝒢^{(k)} = (𝒱, ℰ^{(k)})$ , where $𝒱 = [D]$ , and include edge $(i, j)$ in $ℰ^{(k)}$ if $∥ {\hat{𝝍}}_{k, i j}^{(0)} ∥ > 𝚪_{a d j}$ . Let $𝐀^{(k)}$ and $𝐃^{(k)}$ denote the adjacency and degree matrices, respectively. The symmetrically normalized Laplacian is then defined as $𝐋_{s y m}^{(k)} = 𝐈 - {(𝐃^{(k)})}^{- 1 / 2} 𝐀^{(k)} {(𝐃^{(k)})}^{- 1 / 2},$ which offers a scale-invariant measure of connectivity. We aim to shrink $𝚿_{k}$ toward a diagonal target (i.e., an empty graph), with $𝐋_{target} = 𝟎$ . The spectral distance between ${\hat{𝚿}}_{k}^{(0)}$ and this target is $D_{LS} (𝚿_{k}^{(0)}) = {∥ spec (𝐋_{sym}^{k}) ∥}_{2},$ and the corresponding adaptive weights are given by $𝐏_{k, i j} = {(D_{LS} (𝚿_{k}^{(0)}) + ϵ)}^{- 1} .$

Parameter Estimation. Following [75], we adopt similar parameter estimation for variable ranking and focus here on estimating $𝕊 ℝ 𝕌 𝕎$ under the MNARz mechanism using the EM algorithm. Other estimation details are provided in the Supplement. Let each observation be partitioned as $𝒚_{n} = (𝒚_{n}^{o}, 𝒚_{n}^{m})$ , and define $𝝃 = (𝜽_{reg}, 𝜽_{indep})$ . Since $f_{c}$ is independent of $𝒚_{n}$ , the observed-data log-likelihood becomes:

ℓ ​ (𝚯; 𝒀^{o}, 𝑪) = \sum_{n = 1}^{N} \log ​ \sum_{k = 1}^{K} π_{k} ​ f_{c} ​ (𝒄_{n}; 𝝍_{k}) ​ f_{k}^{o} ​ (𝒚_{n}^{o}; 𝜶_{k}, 𝝃),

where $f_{k}^{o} (𝒚_{n}^{o}; 𝜶_{k}, 𝝃) := \int f_{clust} (𝒚_{n}^{S} ∣ 𝜶_{k}) f_{reg} (𝒚_{n}^{U} ∣ 𝒚_{n}^{R}; 𝜽_{reg}) f_{indep} (𝒚_{n}^{W} ∣ 𝜽_{indep}) 𝑑 𝒚_{n}^{m}$ .

The E-step computes responsibilities:

t_{n ​ k}^{(t)} = \frac{π_{k}^{(t - 1)} ​ f_{k}^{o} ​ (𝒚_{n}^{o}; 𝜶_{k}^{(t - 1)}, 𝝃^{(t - 1)}) ​ f_{c} ​ (𝒄_{n}; 𝝍_{k}^{(t - 1)})}{\sum_{l = 1}^{K} π_{l}^{(t - 1)} ​ f_{l}^{o} ​ (𝒚_{n}^{o}; 𝜶_{l}^{(t - 1)}, 𝝃^{(t - 1)}) ​ f_{c} ​ (𝒄_{n}; 𝝍_{l}^{(t - 1)})} .

The $Q$ -function to be maximized in the M-step is:

	$Q (𝚯; 𝚯^{(t - 1)})$	$= \sum_{n = 1}^{N} \sum_{k = 1}^{K} t_{n k}^{(t)} {\log π_{k} + g_{y} (𝜶_{k}) + g_{c} (𝝍_{k})} .$
	$g_{y} (𝜶_{k})$	$= 𝔼_{𝚯^{(t - 1)}} [\log f_{k} (𝒚_{n}; 𝜶_{k}, 𝝃_{k}) ∣ 𝒚_{n}^{o}, 𝒄_{n}, z_{n k} = 1],$
	$g_{c} (𝝍_{k})$	$= 𝔼_{𝚯^{(t - 1)}} [\log f_{c} (𝒄_{n} ∣ 𝒚_{n}; 𝝍_{k}) ∣ 𝒚_{n}^{o}, 𝒄_{n}, z_{n k} = 1] .$

In the M-step, we update:

	$π_{k}^{(t)}$	$= \frac{1}{N} \sum_{n} t_{n k}^{(t)}, 𝜶_{k}^{(t)} = \arg \max_{𝜶_{k}} \sum_{n} t_{n k}^{(t)} g_{y} (𝜶_{k}),$
	$𝝍_{k}^{(t)}$	$= \arg \max_{𝝍_{k}} \sum_{n} t_{n k}^{(t)} g_{c} (𝝍_{k}), 𝝃^{(t)} = \arg \max_{𝝃} \sum_{n, k} t_{n k}^{(t)} g_{y} (𝝃) .$

We now summarize the workflow; the penalty acts only in Stage A (ranking), while Stage B performs unpenalized SRUW model selection by a single pass over the ranked list.

Algorithm 1 High level two-stage SRUW-MNARz procedure

Input: incomplete data $𝒀$ , mask $𝑪$ , model grid $(K, m)$ ; regularization grids $𝒢_{λ}$ (and $𝒢_{ρ}$ ).
Stage A: Ranking (penalized GMM on a fast-imputed data)

1.

Produce a fast single imputation $\tilde{𝒀}$ only to enable ranking.
2.

For each $(λ, ρ) \in 𝒢_{λ} \times 𝒢_{ρ}$ , fit the penalized GMM to $\bar{𝒀}$ where $\bar{𝒀}$ denotes the centered/scaled $\tilde{𝒀}$ ; record ${\hat{μ}}_{k d} (λ, ρ)$ .
3.

Compute ranking score $𝒪_{K} (d)$ for each variable $d$ by counting along the path how often ${{\hat{μ}}_{k d} (λ, ρ)}_{k}$ remain nonzero; sort variables by $𝒪_{K} (d)$ (see [13] for details).

Stage B: Role assignment (SRUW on original incomplete data)

1.

Traverse ranked variables once (similar to [13]); at each step, fit unpenalized SRUW-MNARz on incomplete $𝒀$ and decide the role (S/U/W) of the new variable using BIC criterion using Equation 2 as in [38]
2.

Return $(\hat{𝕊}, \hat{ℝ}, \hat{𝕌}, \hat{𝕎})$ , $\hat{K}, \hat{m}$ , and parameter estimates.

Output: final role sets and estimates.

Practical Initialization. Like all mixture EM algorithms, ours is sensitive to initialization. We use a robust warm start: (i) fast single imputation $\tilde{𝒀}$ (only for ranking); (ii) $K$ -means++/hierarchical clustering/random start on $\tilde{𝒀}$ for $(π_{k}, 𝝁_{k}, 𝚺_{k})$ ; (iii) given initial partitions from step (ii) $𝚿_{k}^{(0)}$ set to diagonal estimates using samples assigned to each cluster (cluster-aware estimates); (iv) $𝝍_{k}$ initialized from class-wise missing rates after a soft E-step. We then run a $λ$ -path (as illustrated below) for Stage A, followed by unpenalized SRUW estimation for Stage B. Per- $K$ penalty grids are constructed by data-dependent upper bounds and a geometric path: For $λ$ , with the hard partition $𝒁^{(0)}$ from initialization,

λ_{\max} = \max_{k \in [K], d \in [D]} | {(𝒁^{(0) ⊤} ​ 𝑿)}_{k ​ d} |, λ_{ℓ} = λ_{\max} ​ {(\frac{λ_{\min}}{λ_{\max}})}^{\frac{ℓ - 1}{L - 1}}, λ_{\min} = ξ ​ λ_{\max}, ℓ = [L] .

This matches the KKT threshold at which the $ℓ_{1}$ penalty zeros all $𝝁_{k}$ entries. For $ρ$ , let $𝑺_{k}$ be the (hard-label) empirical covariance and note our Stage-A glasso objective uses the coefficient $(2 ρ / n_{k})$ in front of the weighted $ℓ_{1}$ term. A diagonal-forcing threshold is

ρ_{\max} = \max_{k} \max_{i \neq j} \frac{n_{k} ​ | {(𝑺_{k})}_{i ​ j} |}{{(𝑷_{k})}_{i ​ j}},

followed by the same $L$ -point geometric path $ρ_{ℓ} = ρ_{\max} {(ρ_{\min} / ρ_{\max})}^{(ℓ - 1) / (L - 1)}$ with $ρ_{\min} = ξ ρ_{\max}$ . We set $diag (𝑷_{k}) = 𝟎$ when we do not impose a penalty on diagonal entries.

4 Theoretical Properties

In this section, we present theoretical guarantees for our extended framework, including identifiability and selection consistency under the MNARz mechanism, and extend the results to the general $𝕊 ℝ 𝕌 𝕎$ model. We also show that these guarantees apply to the regularized approach of [13]. Given the known issues in finite mixture models, such as degeneracy and non-identifiability, we adopt standard assumptions, largely consistent with those in [38]. For any configuration $(𝕊, ℝ, 𝕌, 𝕎)$ , let $𝜽_{(𝕊, ℝ, 𝕌, 𝕎)}^{*}$ denote the true parameter and ${\hat{𝜽}}_{(𝕊, ℝ, 𝕌, 𝕎)}$ its maximum likelihood estimator.

Theorem 1 (Informal: Identifiability of the $𝕊 ℝ 𝕌 𝕎$ Model).

Let $(K, m, r, l, 𝕍)$ and $(K^{*}, m^{*}, r^{*}, l^{*}, 𝕍^{*})$ denote two models under the MNARz mechanism. Let $𝚯_{(K, m, r, l, 𝕍)} \subseteq 𝚼_{(K, m, r, l, 𝕍)}$ denote the parameter space such that each element $𝛉 = (𝛂, 𝐚, 𝛃, 𝛀, 𝛄, 𝚪)$ satisfies the following: (i) the component-specific parameters $(𝛍_{k}, 𝚺_{k})$ are distinct and satisfy, for all $s \subseteq 𝕊$ , there exist $1 \leq k < k^{'} \leq K$ such that $𝛍_{k, \bar{s} | s} \neq 𝛍_{k^{'}, \bar{s} | s} or 𝚺_{k, \bar{s} | s} \neq 𝚺_{k^{'}, \bar{s} | s} or 𝚺_{k, \bar{s} \bar{s} | s} \neq 𝚺_{k^{'}, \bar{s} \bar{s} | s},$ where $\bar{s}$ denotes the complement of $s$ in $𝕊$ ; (ii) if $𝕌 \neq \emptyset$ , then for every $j \in ℝ$ , there exists $u \in 𝕌$ such that $𝛃_{u j} \neq 0$ , and each $u \in 𝕌$ affects at least one $j \in ℝ$ with $𝛃_{u j} \neq 0$ ; and (iii) the parameters $𝚪$ and $𝛄$ exactly respect the structural forms $r$ and $l$ , respectively. If there exist $𝛉 \in 𝚯_{(K, m, r, l, 𝕍)}$ and $𝛉^{*} \in 𝚯_{(K^{*}, m^{*}, r^{*}, l^{*}, 𝕍^{*})}$ such that $f (\cdot ∣ K, m, r, l, 𝕍, 𝛉) = f (\cdot ∣ K^{*}, m^{*}, r^{*}, l^{*}, 𝕍^{*}, 𝛉^{*}),$ then $(K, m, r, l, 𝕍) = (K^{*}, m^{*}, r^{*}, l^{*}, 𝕍^{*})$ and $𝛉 = 𝛉^{*}$ .

Theorem 2 (Informal: BIC consistency for the $𝕊 ℝ 𝕌 𝕎$ model).

Assume the MNARz mechanism, the regularity conditions specified in the Supplementary Material, and the existence of a unique tuple $(K_{0}, m_{0}, r_{0}, l_{0}, 𝕍_{0})$ such that the data-generating distribution satisfies $h = f (\cdot ∣ 𝛉_{(K_{0}, m_{0}, r_{0}, l_{0}, 𝕍_{0})}^{*})$ for some true parameter $𝛉^{*}$ where $h$ is the density function of the sample $𝐘$ and the model specification $(K_{0}, m_{0}, r_{0}, l_{0}, 𝕍_{0})$ is assumed to be known. Let $(K_{0}, m_{0}, r_{0}, l_{0})$ be fixed. Then the variable selection procedure that selects the subsets $(\hat{𝕊}, \hat{ℝ}, \hat{𝕌}, \hat{𝕎})$ by maximizing the BIC criterion is consistent, in the sense that $ℙ ((\hat{𝕊}, \hat{ℝ}, \hat{𝕌}, \hat{𝕎}) = (𝕊_{0}, ℝ_{0}, 𝕌_{0}, 𝕎_{0})) \to_{N \to \infty}^{} 1 .$

Theorem 3 (Informal: Selection consistency of the two-stage procedure).

Under certain regularity conditions in Supplementary Material, the backward stepwise selection procedure in the $𝕊 ℝ 𝕌 𝕎$ model, guided by the variable ranking stage, consistently recovers the true relevant variable set $𝕊_{0}^{*}$ with high probability. That is, $ℙ ({\hat{𝕊}}^{*} = 𝕊_{0}^{*}) \to_{N \to \infty}^{} 1 .$

Proof sketch. The proof of Theorem 3 proceeds in three main steps: 1) We first establish the consistency of the penalized M-estimator used in the variable ranking stage, specifically the penalized GMMs. 2) This consistency implies that the variables are ranked in a manner that, with high probability, separates relevant variables from noise variables. 3) We then show that the subsequent BIC-based role determination step, when applied to this consistent ranking, correctly recovers the true variable roles. A key component of the proof involves establishing consistency in parameter estimation. In particular, we derive a Restricted Strong Convexity (RSC)-type condition for the negative GMM log-likelihood. More precisely, let $ℒ_{N} (𝜶)$ denote the empirical objective function based on sample size $N$ , and let $𝜶^{⋆}$ be the true parameter. There exist constants $γ_{1}, γ_{2} > 0$ such that for every perturbation $𝚫$ in a restricted neighborhood of $𝜶^{⋆}$ , $ℒ_{N} (𝜶^{⋆} + 𝚫) - ℒ_{N} (𝜶^{⋆}) - ⟨ \nabla ℒ_{N} (𝜶^{⋆}), 𝚫 ⟩ \geq \frac{γ_{1}}{2} {∥ 𝚫_{𝝁} ∥}_{2}^{2} + \frac{γ_{2}}{2} {∥ 𝚫_{𝚺^{- 1}} ∥}_{F}^{2} - τ_{N} (𝚫),$ where $τ_{N} (𝚫)$ is a tolerance term that decays at rate $N^{- 1 / 2}$ . A complete and rigorous proof is provided in the Supplementary Material.

Implications of the theorems. In the framework of [13], the parameter $c$ in the BIC-based selection step (representing the number of consecutive non-positive BIC differences allowed before terminating inclusion into $𝕊$ or $𝕎$ ) governs the balance between false negatives (omitting true variables) and false positives (including irrelevant variables). Theorem 3 implies that for finite samples, choosing a moderately large $c$ , such as $c \approx \log (D)$ (e.g., 3 to 5, as commonly practiced), provides robustness against spurious BIC fluctuations caused by noise variables. In particular, if the gap in the $𝒪_{K} (d)$ scores between the last true signal and the first noise variable is sufficiently pronounced, the exact value of $c$ is not overly critical for ensuring asymptotic consistency, as long as it adequately accounts for statistical noise in the BIC differences sequence. However, an excessively large $c$ could lead to the inclusion of noise variables, especially if the variable ranking is imperfect.

5 Experiments

Our primary objective in this section is to evaluate the clustering quality, imputation accuracy, as well as variable-role recovery under incomplete data scenarios. For the simulated dataset, we compare our method with its direct predecessors in [37, 13], denoting them as Clustvarsel and Selvar respectively, and design each of them to be a pre-imputation+clustering pipeline with the imputation method to be Random Forest [10] using missRanger package [40, 63]. For [37], we use the algorithm in [59] with forward direction and headlong process, with rationale being explained in [13].

We also benchmark with VarSelLCM illustrated in [35] since the model itself also recasts the variable selection problem into a model-selection one. For this case study, we adopt the same real high-dimensional dataset on transcriptome as in [37, 39, 13] and interpret the result in comparison with previous findings in [37, 39]. To measure imputation accuracy, we use WNRMSE - a weighted version of NRMSE where the weights are deduced from the proportion of missingness in each group, while clustering performance is measured by ARI. The variable selection will be assessed by the frequency the model chooses the correct set of relevant variables, which are controlled in the simulation.

Simulated Dataset. We generate two synthetic data sets with sample size $n = 2000$ for variable/model selection with missing values treatment benchmarking. The first data set mimics [39] where each observation $𝒚_{n} \in ℝ^{7}$ is drawn from a four-component Gaussian mixture on the clustering block $𝕊 = {𝒚_{1}, 𝒚_{2}, 𝒚_{3}}$ . The components are equiprobable and have means ${(0, 0, 0)}^{^{'}}, {(- 6, 6, 0)}^{^{'}}, {(0, 0, 6)}^{^{'}}, {(- 6, 6, 6)}^{^{'}}$ with a diagonal covariance matrix $𝚺 = diag (6 \sqrt{2}, 1, 2) σ_{scale}^{2}$ . The redundant block $𝕌 = {𝒚_{4}, 𝒚_{5}}$ is generated by a linear regression $𝕌 = {(- 1, 2)}^{^{'}} + 𝐗_{S} [{(0.5, 2)}^{^{'}}, {(1, 0)}^{^{'}}] + ϵ$ , where $ϵ \sim 𝒩 (𝟎, 𝐑 (π / 𝟔) diag (𝟏, 𝟑) 𝐑 {(π / 𝟔)}^{^{'}})$ and $𝐑 (𝜽)$ denotes the usual planar rotation. The remaining $W$ variables are iid $𝒩 (0, 1)$ noise. In the second design, an observation $𝒚_{n} \in ℝ^{14}$ . The clustering block now consists of $(𝒚_{1}, 𝒚_{2})$ , drawn from an equiprobable four-component Gaussian mixture with means $(0, 0), (4, 0), (0, 2), (4, 2)$ and common covariance $0.5 𝐈_{𝟐}$ . Conditional on $(𝒚_{1}, 𝒚_{2})$ , the vector $𝒚_{3 : 14}$ follows a linear model $𝒚_{3 : 14} = 𝜶 + (𝒚_{1}, 𝒚_{2}) 𝜷 + 𝜺$ , where $𝜶$ , the $2 \times 12$ coefficient matrix $𝜷$ and the diagonal covariance $𝛀$ are varied across eight sub-scenarios: scenario $1$ contains only pure noise, scenarios $2 - 7$ introduce more regressors, and scenario $8$ adds intercept shifts together with three additional pure-noise variables $𝒚_{12 : 14}$ . In our setting, we consider $K \in {2, 3, 4}$ and every method except VarSelLCM is allowed to learn its own covariance structure. We vary the missing ratio ${0.05, 0.1, 0.2, 0.3, 0.5}$ under both MAR and MNAR patterns. For the second simulated dataset, we temporarily consider only scenario $8$ . Moreover, for Selvar and SelvarMNARz, we fix $c = 2$ and compute $𝑷_{k}$ with the spectral distance. Further details of the synthetic-data generation scheme are provided in the supplement. From Figure 1, we observe that SelvarMNARz delivers the highest ARI and the lowest WNRMSE across every missing-data level in both scenarios. Its performance declines only modestly as missingness increases, while the impute-then-cluster baselines deteriorate sharply, especially at 30% and 50% missingness under MNAR, pointing to the benefit of modeling the missing-data mechanism within the clustering algorithm. To statistically validate that our model exhibits a significantly slower performance decline under high missingness, we conducted one-sided Welch’s t-tests (Bonferroni-corrected) at the 50% MNAR level. As shown in Table 1, the ARI of SelvarMNARz is significantly greater than all baseline models ( $p < 0.001$ ) across 20 replications.

Table 1: Welch’s t-test for

ARI

(MNAR, 50% missingness,

α = 0.05

).

Model	Mean ARI	Std	Comparison	p-value	Signif.
SelvarMNARz	0.511	0.052	–	–	NA
Clustvarsel	0.363	0.088	vs. SelvarMNARz	$< 0.001$	$, , , $
Selvar	0.348	0.108	vs. SelvarMNARz	$< 0.001$	$, , , $
VarSelLCM	0.344	0.101	vs. SelvarMNARz	$< 0.001$	$, , , $

While all methods show a monotonic performance drop as missingness increases, these results confirm that the superior performance of SelvarMNARz is not a random artifact but a statistically significant improvement, demonstrating its robustness in the challenging missing data scenarios. Figure 2 shows that SelvarMNARz consistently recovers the true cluster component and the correct set of clustering variables, regardless of the missing-data mechanism or rate. On Dataset 2, every method succeeds under both MAR and MNAR, whereas on Dataset 1, the performance of Clustvarsel and VarSelLCM drops as missingness increases, leading to the omission of important variables. The contrast reflects the data-generating designs: Dataset 2 includes pure-noise variables $𝕎$ within the $𝕊 ℝ 𝕌 𝕎$ framework, whereas Dataset 1 aligns more closely with the assumptions behind Clustvarsel and VarSelLCM. Finally, workflows that rely on external imputation misidentify relevant variables once missingness exceeds 20% under either missing-data scenarios.

Refer to caption — Figure 1: Comparison of four models under MAR and MNAR mechanisms over 20 replications; for ARI/WNRMSE, higher/lower boxplots indicate better performance.

Table 2: Size of each mixture component and

R^{2}

obtained by regressing the

𝕌

-block on the

𝕊

-block within that cluster. Values extremely close to 1 (marked by ^∗) occur because the

𝕌

-block is nearly constant in those groups.

Cluster	# Genes	$𝑹^{𝟐}$
1	715	0.02
2	11	1.00^∗
3	6	1.00^∗
4	93	0.45
5	124	0.12
6	9	1.00^∗

Cluster	# Genes	$𝑹^{𝟐}$
7	36	0.68
8	13	1.00^∗
9	15	1.00^∗
10	29	0.59
11	71	0.43
12	24	0.91

Cluster	# Genes	$𝑹^{𝟐}$
13	46	0.57
14	19	1.00^∗
15	6	1.00^∗
16	16	1.00^∗
17	24	0.90
18	10	1.00^∗

Transcriptome Data. We re-examine the 1,267-gene Arabidopsis thaliana transcriptome dataset previously analyzed using SelvarClust and SelvarClustMV in [37, 39] to illustrate clustering with variable selection. Full experimental details are provided in the Supplementary Material; here, we summarize the settings and key findings. We fitted SelvarMNARz for $K \in {2, \dots, 20}$ clusters with $c = 5$ , spectral distance weights $𝑷_{k}$ , and a $π_{k} L C$ mixture structure as in [39]. For conciseness, we refer to Project 1 as P1, Project 2 as P2, and so on. As shown in Figure 3, the proposed framework partitions the transcriptome into 18 clusters and reaffirms P1-P4 as core drivers, while globally assigning P5-P7 as being explained by these drivers. This finding is consistent with [39], which also identifies P2 as relevant for clustering, supporting the hypothesis that iron signaling responses are critical for defining co-expression groups when a broader gene set (including entries with missing data) is considered. Additionally, all three methods agree that P5 is not a primary clustering factor and is more likely a correlated effect explained by broader cellular processes. However, a key departure lies in the reclassification of P6 and P7, previously identified as informative clustering drivers [37, 39], into a redundant $𝕌$ block, explained by P1-P4. This suggests that, after accounting for MNAR effects, variation in P6-P7 is largely predictable from the core axes P1-P4, and P5-P7 all measure late-stage stress outputs. Further analysis of cluster sizes and the per-cluster $R^{2}$ values, shown in Table 2, validates the selected variables when regressing P5-P7 on P1-P4 within each cluster. Six biologically coherent clusters (6, 7, 8, 10, 12, 18) exhibit $R^{2} > 0.60$ , indicating that late-stress variation is well encoded by P1-P4. Cluster 1, although large, is transcriptionally flat across both $𝕊$ and $𝕌$ ; its low $R^{2} = 0.02$ thus reflects a vanishing denominator rather than hidden structure. True residual structure (unexplained $𝕌$ signal) is concentrated in two groups: clusters 5 and 11, which are exactly where the 18th mixture component forms.

6 Conclusion, Limitations and Perspectives

We proposed a unified model-based clustering framework that jointly performs variable selection and handles MNAR data by combining adaptive penalization with explicit modeling of missingness-latent class dependencies. This enhances clustering flexibility and robustness in high-dimensional settings like transcriptomics. The method enjoys theoretical guarantees, including selection and asymptotic consistency, and shows strong empirical performance. However, it is currently limited to continuous data via Gaussian mixture models, making it less suitable for categorical or mixed-type variables. Extending the framework to categorical settings is a natural next step. For instance, Dean and Raftery [15] introduced variable selection for latent class models, later refined by Fop et al. [20] using a statistically grounded stepwise approach. Bontemps and Toussile [8] proposed a mixture of multivariate multinomial distributions with combinatorial variable selection and slope heuristics for penalty calibration. Building upon these, a future extension of our framework could incorporate discrete data distributions, adapt penalization accordingly, and model missingness mechanisms using latent class structures. Further directions include mixed-type data extensions and scalability improvements to accommodate larger, more heterogeneous datasets.

Acknowledgments

TrungTin Nguyen and Christopher Drovandi acknowledge funding from the Australian Research Council Centre of Excellence for the Mathematical Analysis of Cellular Systems (CE230100001). Christopher Drovandi was also supported by an Australian Research Council Future Fellowship (FT210100260). This paper was also supported by AISIA Research Lab, Vietnam. The authors would like to express their special thanks to the authors of [37, 39], in particular Professor Cathy Maugis-Rabusseau and Dr. Marie-Laure Martin, for providing the Arabidopsis thaliana transcriptome dataset.

References

[1] H. Akaike. A new look at the statistical model identification. IEEE transactions on automatic control, 19(6):716–723, 2003. ↩ 1 G.2
[2] E. S. Allman, C. Matias, and J. A. Rhodes. Identifiability of parameters in latent structure models with many observed variables. Annals of Statistics, 37(6A):3099–3132, 2009. ↩ D.1
[3] J. L. Andrews and P. D. McNicholas. Variable selection for clustering and classification. Journal of Classification, 31(2):136–153, 2014. ↩ 1 G.2
[4] I. Balelli, A. Sportisse, F. Cremonesi, P.-A. Mattei, and M. Lorenzi. Fed-miwae: Federated imputation of incomplete data via deep generative models. arXiv preprint arXiv:2304.08054, 2023. ↩ 1 G.3
[5] E. Barut, J. Fan, and A. Verhasselt. Conditional sure independence screening. Journal of the American Statistical Association, 111(515):1266–1277, 2016. ↩ 1
[6] J.-P. Baudry, C. Maugis, and B. Michel. Slope heuristics: overview and implementation. Statistics and Computing, 22(2):455–470, 2012. ↩ 2
[7] C. Biernacki, G. Celeux, and G. Govaert. Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7):719–725, 2000. ↩ 2
[8] D. Bontemps and W. Toussile. Clustering and variable selection for categorical multivariate data. Electronic Journal of Statistics, 7:2344–2371, Jan. 2013. ↩ 6 G.2
[9] L. Breiman. Better subset regression using the nonnegative garrote. Technometrics, 37(4):373–384, 1995. ↩ 1 G.2
[10] L. Breiman. Random forests. Machine Learning, 45:5–32, 2001. ↩ 5
[11] A. Casa, A. Cappozzo, and M. Fop. Group-wise shrinkage estimation in penalized model-based clustering. Journal of Classification, 39(3):648–674, 2022. ↩ 2 A
[12] G. Celeux and G. Govaert. Gaussian parsimonious clustering models. Pattern Recognition, 28(5):781–793, 1995. Publisher: Elsevier. ↩ 2
[13] G. Celeux, C. Maugis-Rabusseau, and M. Sedki. Variable selection in model-based clustering and discriminant analysis with a regularization approach. Advances in Data Analysis and Classification, 13:259–278, 2019. ↩ 1 2 3 4 5 A F.3 F.5 G.2
[14] M. C. Chong, H. D. Nguyen, and TrungTin Nguyen. Risk Bounds for Mixture Density Estimation on Compact Domains via the h-Lifted Kullback–Leibler Divergence. Transactions on Machine Learning Research, 2024. ↩ 2
[15] N. Dean and A. E. Raftery. Latent class analysis variable selection. Annals of the Institute of Statistical Mathematics, 62:11–35, 2010. ↩ 6 G.2
[16] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–38, 1977. ↩ 1 G.1
[17] D. Do, L. Do, S. A. McKinley, J. Terhorst, and X. Nguyen. Dendrogram of mixing measures: Learning latent hierarchy and model selection for finite mixture models. arXiv preprint arXiv:2403.01684, 2024. ↩ 2
[18] B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of Statistics, pages 407–451, 2004. ↩ 1 G.2
[19] J. Fan and J. Lv. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society Series B: Statistical Methodology, 70(5):849–911, 2008. ↩ 1
[20] M. Fop, K. M. Smart, and T. B. Murphy. Variable selection for latent class analysis with application to low back pain diagnosis. The Annals of Applied Statistics, pages 2080–2110, 2017. ↩ 6 G.2
[21] F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel. Mixture of expert posterior surrogates for approximate Bayesian computation. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique (SFdS), Lyon, France, June 2022. ↩ 2
[22] F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel. Summary statistics and discrepancy measures for approximate Bayesian computation via surrogate posteriors. Statistics and Computing, 32(5):85, Oct. 2022. ↩ 2
[23] R. Foygel and M. Drton. Extended Bayesian Information Criteria for Gaussian Graphical Models. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010. ↩ 2
[24] S. Frühwirth-Schnatter, C. Pamminger, A. Weber, and R. Winter-Ebmer. Labor market entry and earnings dynamics: Bayesian inference using mixtures-of-experts Markov chain clustering. Journal of Applied Econometrics, 27(7):1116–1137, 2012. ↩ 2
[25] C. R. Genovese and L. Wasserman. Rates of convergence for the Gaussian mixture sieve. The Annals of Statistics, 28(4):1105 – 1127, 2000. ↩ 2
[26] N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. The Annals of Statistics, 44(6):2726 – 2755, 2016. ↩ 2
[27] N. Ho and X. Nguyen. On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics, 10(1):271–307, 2016. ↩ 2
[28] A. E. Hoerl and R. W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55–67, 1970. ↩ 1 G.2
[29] A. Jasra, C. C. Holmes, and D. A. Stephens. Markov Chain Monte Carlo Methods and the Label Switching Problem in Bayesian Mixture Modeling. Statistical Science, 20(1):50 – 67, 2005. ↩ 1 G.1
[30] S. Kim, M. G. Tadesse, and M. Vannucci. Variable selection in clustering via dirichlet process mixture models. Biometrika, 93(4):877–893, 2006. ↩ 1
[31] J. Kwon and C. Caramanis. The em algorithm gives sample-optimality for learning mixtures of well-separated gaussians. In Conference on Learning Theory, pages 2425–2487. PMLR, 2020. ↩ F.4
[32] M. H. Law, M. A. Figueiredo, and A. K. Jain. Simultaneous feature selection and clustering using mixture models. IEEE transactions on pattern analysis and machine intelligence, 26(9):1154–1166, 2004. ↩ 1 G.2
[33] R. J. Little and D. B. Rubin. Statistical analysis with missing data. John Wiley & Sons, 2019. ↩ A
[34] P.-L. Loh and M. J. Wainwright. Regularized M-estimators with Nonconvexity: Statistical and Algorithmic Theory for Local Optima. Journal of Machine Learning Research, 16(19):559–616, 2015. ↩ A
[35] M. Marbac and M. Sedki. Variable selection for mixed data clustering: A model-based approach. arXiv preprint arXiv:1703.02293, 2017. ↩ 5 A
[36] P.-A. Mattei and J. Frellsen. MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4413–4423. PMLR, June 2019. ↩ 1 G.3
[37] C. Maugis, G. Celeux, and M.-L. Martin-Magniette. Variable selection for clustering with gaussian mixture models. Biometrics, 65(3):701–709, 2009. ↩ 2 5 ↑ B D.1 F.6
[38] C. Maugis, G. Celeux, and M.-L. Martin-Magniette. Variable selection in model-based clustering: A general variable role modeling. Computational Statistics & Data Analysis, 53(11):3872–3882, 2009. ↩ 1 2 3 4 A B D.1 D.2 G.2
[39] C. Maugis-Rabusseau, M.-L. Martin-Magniette, and S. Pelletier. Selvarclustmv: Variable selection approach in model-based clustering allowing for missing values. Journal de la Société Française de Statistique, 153(2):21–36, 2012. ↩ 1 2 3 5 ↑ F.6
[40] M. Mayer. missRanger: Fast Imputation of Missing Values, 2025. R package version 2.6.2, https://mayer79.github.io/missRanger/. ↩ 5
[41] R. Mazumder and T. Hastie. Exact covariance thresholding into connected components for large-scale graphical lasso. Journal of Machine Learning Research, 13(27):781–794, 2012. ↩ F.4
[42] G. J. McLachlan and K. E. Basford. Mixture Models: Inference and Applications to Clustering, volume 38. Marcel Dekker, 1988. ↩ 1
[43] G. J. McLachlan and D. Peel. Finite mixture models. John Wiley & Sons, 2004. ↩ 1
[44] S. Negahban and M. J. Wainwright. Restricted strong convexity and weighted matrix completion: Optimal bounds with noise. Journal of Machine Learning Research, 13(53):1665–1697, 2012. ↩ A D.3
[45] D. N. Nguyen and Z. Li. Joint learning of Gaussian graphical models in heterogeneous dependencies of high-dimensional transcriptomic data. In The 16th Asian Conference on Machine Learning (Conference Track), 2024. ↩ 2
[46] H. Nguyen, T. Nguyen, and N. Ho. Demystifying Softmax Gating in Gaussian Mixture of Experts. In Advances in Neural Information Processing Systems, Dec. 2023. ↩ 2
[47] H. Nguyen, T. Nguyen, K. Nguyen, and N. Ho. Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts. In S. Dasgupta, S. Mandt, and Y. Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 2683–2691. PMLR, May 2024. ↩ 2
[48] H. D. Nguyen, T. Nguyen, and F. Forbes. Bayesian Likelihood Free Inference using Mixtures of Experts. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2024. ↩ 2
[49] T. Nguyen, F. Chamroukhi, H. D. Nguyen, and G. J. McLachlan. Approximation of probability density functions via location-scale finite mixtures in Lebesgue spaces. Communications in Statistics - Theory and Methods, 52(14):5048–5059, 2023. ↩ 2
[50] T. Nguyen, D. N. Nguyen, H. D. Nguyen, and F. Chamroukhi. A non-asymptotic theory for model selection in high-dimensional mixture of experts via joint rank and variable selection. In AJCAI Australasian Joint Conference on Artificial Intelligence 2023, Dec. 2023. ↩ 2
[51] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and F. Forbes. A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models. Electronic Journal of Statistics, 16(2):4742 – 4822, 2022. ↩ 2
[52] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and G. J. McLachlan. Approximation by finite mixtures of continuous density functions that vanish at infinity. Cogent Mathematics & Statistics, 7(1):1750861, Jan. 2020. ↩ 2
[53] X. Nguyen. Convergence of latent mixing measures in finite and infinite mixture models. The Annals of Statistics, 41(1):370–400, 2013. ↩ 2
[54] A. E. Raftery and N. Dean. Variable selection for model-based clustering. Journal of the American Statistical Association, 101(473):168–178, 2006. ↩ 1 G.2
[55] A. Rakhlin, D. Panchenko, and S. Mukherjee. Risk bounds for mixture density estimation. ESAIM: PS, 9:220–229, 2005. ↩ 2
[56] D. B. Rubin. Inference and missing data. Biometrika, 63(3):581–592, 1976. ↩ A
[57] D. B. Rubin. An overview of multiple imputation. In Proceedings of the survey research methods section of the American statistical association, volume 79, page 84. Citeseer, 1988. ↩ 1 G.3
[58] G. Schwarz. Estimating the dimension of a model. The annals of statistics, pages 461–464, 1978. ↩ 1 2 G.2
[59] L. Scrucca and A. E. Raftery. clustvarsel: A package implementing variable selection for gaussian model-based clustering in r. Journal of Statistical Software, 84:1–28, 2018. ↩ 1 5 G.2
[60] W. Shen, S. T. Tokdar, and S. Ghosal. Adaptive Bayesian multivariate density estimation with Dirichlet mixtures. Biometrika, 100(3):623–640, 2013. ↩ 2
[61] C.-Y. Sin and H. White. Information criteria for selecting possibly misspecified parametric models. Journal of Econometrics, 71(1):207–225, 1996. ↩ 2
[62] A. Sportisse, M. Marbac, F. Laporte, G. Celeux, C. Boyer, J. Josse, and C. Biernacki. Model-based clustering with missing not at random data. Statistics and Computing, 34(4):135, 2024. ↩ 1 3 A F.2 F.5 G.3
[63] D. J. Stekhoven and P. Bühlmann. Missforest—non-parametric missing value imputation for mixed-type data. Bioinformatics, 28(1):112–118, 2012. ↩ 1 5 G.3
[64] M. Stephens. Bayesian analysis of mixture models with an unknown number of components—an alternative to reversible jump methods. The Annals of Statistics, 28(1):40 – 74, 2000. ↩ 1
[65] M. Stephens. Dealing with label switching in mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 62(4):795–809, 2000. ↩ G.1
[66] N. Städler, P. Bühlmann, and S. van de Geer. $l_{1}$ -penalization for mixture regression models. TEST, 19(2):209–256, Aug. 2010. ↩ A
[67] M. G. Tadesse, N. Sha, and M. Vannucci. Bayesian variable selection in clustering high-dimensional data. Journal of the American Statistical Association, 100(470):602–617, 2005. ↩ 1
[68] H. Teicher. Identifiability of finite mixtures. The Annals of Mathematical Statistics, 34(4):1265–1269, December 1963. ↩ D.1
[69] T. Thai, T. Nguyen, D. Do, N. Ho, and C. Drovandi. Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures. arXiv preprint arXiv:2505.13052, 2025. ↩ 2
[70] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267–288, 1996. ↩ 1 A G.2
[71] S. Van Buuren and K. Groothuis-Oudshoorn. mice: Multivariate imputation by chained equations in r. Journal of statistical software, 45:1–67, 2011. ↩ 1 G.3
[72] J. Westerhout, T. Nguyen, X. Guo, and H. D. Nguyen. On the Asymptotic Distribution of the Minimum Empirical Risk. In Forty-first International Conference on Machine Learning, 2024. ↩ 2
[73] D. M. Witten, J. H. Friedman, and N. Simon. New insights and faster computations for the graphical lasso. Journal of Computational and Graphical Statistics, 20(4):892–900, 2011. ↩ F.4
[74] Y. Zhao and Q. Long. Multiple imputation in the presence of high-dimensional data. Statistical Methods in Medical Research, 25(5):2021–2035, 2016. ↩ 1 G.3
[75] H. Zhou, W. Pan, and X. Shen. Penalized model-based clustering with unconstrained covariance matrices. Electronic Journal of Statistics, 3(none):1473 – 1496, 2009. Publisher: Institute of Mathematical Statistics and Bernoulli Society. ↩ 1 2 3 A G.2
[76] M. Zhou et al. Global convergence of gradient em for over-parameterized gaussian mixtures. arXiv preprint arXiv:2506.06584, 2025. ↩ F.4
[77] H. Zou. The adaptive lasso and its oracle properties. Journal of the American Statistical Association, 101(476):1418–1429, 2006. ↩ 2

Supplementary Materials for
“A Unified Framework for Variable Selection in Model-Based Clustering with Missing Not at Random”

In this supplementary material, we first provide additional details on the main challenges as well as the technical and computational contributions in Appendix A, with the aim of enhancing the reader’s understanding of our key theoretical and methodological developments. Next, we summarize the definitions and theoretical results for the $𝕊 ℝ$ and $𝕊 ℝ 𝕌 𝕎$ models in Appendix C. We then present the regularity assumptions and technical proofs for the main theoretical results in Appendices C and D, respectively. Finally, we include additional experimental results and further discussion of related work in Appendices G and F.

Appendix A More Details on Main Challenges and Contributions

Model-based clustering with variable selection, particularly using the sophisticated SRUW framework [38], offers a powerful approach to understanding data structure by not only grouping data but also by assigning distinct functional roles to variables. However, extending this rich framework to handle missing data, especially MNAR, and to perform variable selection efficiently in high-dimensional settings presents substantial theoretical and practical challenges. We address some difficulties as follows:

Integrating MNAR inside model-based clustering with variable selection: Unlike MAR or MCAR data, MNAR mechanisms are generally non-ignorable, meaning the missing data process must be explicitly modeled alongside the data distribution to avoid biased parameter estimates and incorrect clustering [33, 56]. Modeling this joint distribution significantly increases model complexity and the risk of misspecification. While various MNAR models exist (see [35, 62]), integration into complex variable selection clustering frameworks like SRUW, along with proving identifiability and estimator consistency, remains a frontier. Specifically, the MNARz mechanism (where missingness depends only on latent class membership) offers a tractable yet meaningful way to handle informative missingness, but its properties within a structured model like SRUW require careful elucidation.

Variable selection with SRUW and missing data in high-dimensional regime: The SRUW model’s strength lies in its detailed variable role specification. However, determining these roles from data is a combinatorial model selection problem. Original SRUW procedures often rely on stepwise algorithms that become computationally prohibitive as the number of variables $D$ increases. Introducing missing data further complicates parameter estimation within each candidate model. Penalized likelihood methods, common in high-dimensional regression (e.g., LASSO [70, 75, 11]), offer a promising avenue for simultaneous parameter estimation and variable selection. However, adapting these to the GMM likelihood and then to the structured SRUW model with an integrated MNARz mechanism requires establishing theoretical guarantees such as parameter estimation consistency and ranking/selection consistency for the chosen penalized objective. Some notable, nontrivial hurdles are appropriate curvature properties like RSC for non-convex log-likelihood loss functions in high dimensions or the cone conditions to ensure sparse recovery.

Consistency of multi-stage procedures: Practical algorithms for variable selection tasks often involve multiple stages (e.g., initial ranking/variable screening, then detailed role assignment). Proving the consistency of such a multi-stage procedure requires advanced tools to handle.

Being aware of these difficulties, we attempt to make several contributions to address the aforementioned challenges, particularly focusing on providing a theoretical foundation for a two-stage variable selection procedure within the SRUW-MNARz framework:

1.

Though the extension of the reinterpretation result from [62] is straightforward, this significantly simplifies the analysis and provides a way to handle identifiability of SRUW under MNAR missingness, which potentially facilitates algorithmic design.
2.

Building on the MAR reinterpretation and existing results for SRUW data model identifiability and MNARz mechanism identifiability, we establish conditions under which the full SRUW-MNARz model parameters (including SRUW structure, GMM parameters, regression coefficients, and MNARz probabilities) are identifiable from the observed data. This is important since identifiability is a prerequisite for consistent parameter estimation.
3.

To underpin the variable ranking stage, which uses a LASSO-like penalized GMM, we provide a rigorous analysis of the penalized GMM estimator based on [66, 44, 34]: First, we establish a non-asymptotic sup-norm bound on the gradient of the GMM negative log-likelihood at the true parameters, which is crucial for selecting the appropriate magnitude for regularization parameters. Subsequently, we formally state the RSC condition needed for the GMM loss function, and the error vector of the penalized GMM estimator lies in a specific cone, which is essential for sparse recovery. These settings provide the necessary high-dimensional statistical guarantees for the penalized GMM used in the ranking stage, ensuring that its estimated sparse means are reliable.
4.

Leveraging the parameter consistency of the penalized GMM, we prove that the variable ranking score $𝒪_{K} (j)$ can consistently distinguish between truly relevant variables (for clustering means) and irrelevant variables. This involves establishing precise “signal strength” conditions; thereby, validating the first step of the two-stage variable selection procedure in [13].
5.

For the main theorem on selection consistency, the process combines ranking consistency with an analysis of the BIC-based stepwise decisions for assigning roles. We show that under appropriate conditions on local BIC decision error rates and the choice of the stopping parameter $c$ , the entire two-stage procedure consistently recovers the true variable partition. This provides the first (to our knowledge) formal consistency proof for such a two-stage variable selection in model-based clustering (SRUW) that incorporates an initial LASSO-like ranking, especially in the context of MNAR pattern. It bridges the theoretical gap between the computationally intensive and a more practical high-dimensional strategy.
6.

While full optimality of $c$ is not derived, we provide a condition for choosing $c$ to achieve a target false positive rate in relevant variable selection, given the single-decision error probability. This moves beyond purely heuristic choices for $c$ in [13], paving the way for more adaptive or theoretically grounded choices of $c$ in practice. This offers a more principled approach to selecting this single hyperparameter in the algorithm, enhancing the reliability of the selection procedure.

Appendix B Preliminary of $𝕊 ℝ$ and $𝕊 ℝ 𝕌 𝕎$ Models

This part summarizes definitions and theoretical results of the $𝕊 ℝ$ and $𝕊 ℝ 𝕌 𝕎$ models in [37, 38]. We start with the following two definitions:

Definition 1 ( $𝕊 ℝ$ Model for Complete Data).

Let $𝐲_{n} \in ℝ^{D}$ be the $n$ -th observation, for $n = 1, \dots, N$ . A model $ℳ_{𝕊 ℝ} = (K, m, 𝕊, ℝ)$ is defined by:

1.

$K$ : The number of mixture components (clusters).
2.

$m$ : The covariance structure of the GMM for variables in $𝕊$ .
3.

$𝕊 \subseteq {1, \dots, D}$ : The non-empty set of relevant clustering variables.
4.

$ℝ \subseteq 𝕊$ : The subset of relevant variables in $𝕊$ used to explain the irrelevant variables $𝕊^{c} = {1, \dots, D} ∖ 𝕊$ via linear regression.

The parameters of the model are $𝛉 = (𝛂, 𝐚, 𝛃, 𝛀)$ , where $𝛂$ are the GMM parameters for variables in $𝕊$ . $𝐚$ is the intercept vector, $𝛃$ is the matrix of regression coefficients for explaining $𝐲^{𝕊^{c}}$ from $𝐲^{ℝ}$ , and $𝛀$ is the covariance matrix of the regression component. The complete data likelihood for one observation $𝐲_{n}$ is:

f ​ (𝒚_{n}; K, m, 𝕊, ℝ, 𝜽) = f_{clust} ​ (𝒚_{n}^{𝕊}; K, m, 𝜶) ​ f_{reg} ​ (𝒚_{n}^{𝕊^{c}}; 𝒂 + 𝒚_{n}^{ℝ} ​ 𝜷, 𝛀)

where $𝐲_{n}^{𝕊}$ denotes the sub-vector of $𝐲_{n}$ corresponding to variables in $𝕊$ , and similarly for $𝐲_{n}^{𝕊^{c}}$ and $𝐲_{n}^{ℝ}$ . $f_{clust}$ is a K-component GMM density, and $f_{reg}$ is a multivariate Gaussian density.

Importantly, $𝕊 ℝ$ model can be equivalently written as a single K-component GMM for the full data vector $𝐲_{n} \in ℝ^{D}$ [37]:

f ​ (𝒚_{n}; K, m, 𝕊, ℝ, 𝜽) = \sum_{k = 1}^{K} π_{k} ​ Φ ​ (𝒚_{n}; 𝝂_{k}, 𝚫_{k})

(8)

where $Φ (\cdot; 𝛎, 𝚫)$ is the multivariate Gaussian density with mean $𝛎$ and covariance $𝚫$ . The parameters $(𝛎_{k}, 𝚫_{k})$ are constructed from the original $𝕊 ℝ$ parameters $(α_{k}, 𝐚, 𝛃, 𝛀)$ as follows: Let $𝚲$ be a $| 𝕊 | \times | 𝕊^{c} |$ matrix derived from $𝛃$ (which is $| ℝ | \times | 𝕊^{c} |$ ). For a variable $j \in 𝕊$ and $l \in 𝕊^{c}$ , $Λ_{j l} = β_{j^{'} l}$ if the $j$ -th variable of $𝕊$ is the $j^{'}$ -th variable of $ℝ$ , and $Λ_{j l} = 0$ if the $j$ -th variable of $𝕊$ is in $𝕊 ∖ ℝ$ . Then, for each component $k$ :

•

The mean vector $𝝂_{k} \in ℝ^{D}$ has elements:

$ν_{k j} = {\begin{matrix} μ_{k j} & if variable j \in 𝕊 \\ {(𝒂 + 𝝁_{k}^{𝕊} 𝚲)}_{j} & if variable j \in 𝕊^{c} \end{matrix}$

where $𝝁_{k}^{𝕊}$ is the mean of the $k$ -th component for variables in $𝕊$ .
•

The covariance matrix $𝚫_{k} \in ℝ^{D \times D}$ has blocks:

$Δ_{k, j l} = {\begin{matrix} Σ_{k, j l} & if j \in 𝕊, l \in 𝕊 \\ {(𝚺_{k} 𝚲)}_{j l} & if j \in 𝕊, l \in 𝕊^{c} \\ {(𝚲^{T} 𝚺_{k})}_{j l} & if j \in 𝕊^{c}, l \in 𝕊 \\ {(𝛀 + 𝚲^{T} 𝚺_{k} 𝚲)}_{j l} & if j \in 𝕊^{c}, l \in 𝕊^{c} \end{matrix}$

Definition 2 ( $𝕊 ℝ 𝕌 𝕎$ Model for Complete Data).

Let $𝐲_{n} \in ℝ^{D}$ be the $n$ -th observation. An $𝕊 ℝ 𝕌 𝕎$ model $ℳ_{𝕊 ℝ 𝕌 𝕎} = (K, m, r, l, 𝕍)$ is defined by:

1.

$K$ : The number of mixture components.
2.

$m$ : The covariance structure of the GMM for variables in $𝕊$ .
3.

$r$ : The form of the covariance matrix $𝛀$ for the regression of $𝕌$ on $ℝ$ .
4.

$l$ : The form of the covariance matrix $𝚪$ for the independent variables $𝕎$ .
5.
$𝕍 = (𝕊, ℝ, 𝕌, 𝕎)$ : A partition of the $D$ variables, where
- •
  
  $𝕊$ : Non-empty set of relevant clustering variables.
- •
  
  $ℝ \subseteq 𝕊$ : Subset of $𝕊$ regressing variables in $𝕌$ . ( $ℝ = \emptyset$ if $𝕌 = \emptyset$ ).
- •
  
  $𝕌$ : Set of irrelevant variables explained by $ℝ$ .
- •
  
  $𝕎$ : Set of irrelevant variables independent of all variables in $𝕊$ .
- •
  
  $𝕊 \cup 𝕌 \cup 𝕎 = {1, \dots, D}$ and these sets are disjoint.

The parameters are $𝛉 = (𝛂, 𝐚, 𝛃, 𝛀, 𝛄, 𝚪)$ , where $𝛂$ is the GMM parameters. The complete data density for one observation $𝐲_{n}$ is:

f ​ (𝒚_{n}; K, m, r, l, 𝕍, 𝜽) = f_{clust} ​ (𝒚_{n}^{𝕊}; K, m, 𝜶) ​ f_{reg} ​ (𝒚_{n}^{𝕌}; 𝒂 + 𝒚_{n}^{ℝ} ​ 𝜷, 𝛀) ​ f_{indep} ​ (𝒚_{n}^{𝕎}; 𝜸, 𝚪)

This can be written as a K-component GMM for the full data vector $𝐲_{n} \in ℝ^{D}$ :

f ​ (𝒚_{n}; K, m, r, l, 𝕍, 𝜽) = \sum_{k = 1}^{K} π_{k} ​ Φ ​ (𝒚_{n}; {\tilde{𝝂}}_{k}, {\tilde{𝚫}}_{k})

(9)

where ${\tilde{𝛎}}_{k}$ and ${\tilde{𝚫}}_{k}$ are constructed from $𝛉$ . Let $(𝛎_{k}^{(𝕊, 𝕌)}, 𝚫_{k}^{(𝕊, 𝕌)})$ be the mean and covariance for the $(𝕊, 𝕌)$ part derived as in the $𝕊 ℝ$ model (where $𝕌$ plays the role of $𝕊_{𝕊 ℝ}^{c}$ ). Then:

	${\tilde{𝝂}}_{k}$	$= {({(𝝂_{k}^{(𝕊, 𝕌)})}^{T}, 𝜸^{T})}^{T}$
	${\tilde{𝚫}}_{k}$	$= (\begin{matrix} 𝚫_{k}^{(𝕊, 𝕌)} & 𝟎 \\ 𝟎 & 𝚪 \end{matrix})$

due to the independence of $𝕎$ from $𝕊$ and $𝕌$ (given the cluster $k$ , which is implicit in the construction of $𝛎_{k}^{(𝕊, 𝕌)}$ using $𝛍_{k}^{𝕊}$ ).

Similar to the global GMM representation of $𝕊 ℝ$ model under MAR, we can obtain such a global form of $𝕊 ℝ 𝕌 𝕎$ model under the same missingness pattern, which is illustrated by the following claim.

Proposition 1.

Under MAR, $𝕊 ℝ 𝕌 𝕎$ has a natural global GMM representation with appropriate parameters.

Proof of Proposition 1.

Let $𝒚_{n} = (𝒚_{n}^{𝕊}, 𝒚_{n}^{𝕌}, 𝒚_{n}^{𝕎})$ be the complete data vector for observation $n \in {1, \dots, N}$ , partitioned according to the $𝕊 ℝ 𝕌 𝕎$ model structure $𝕍 = (𝕊, ℝ, 𝕌, 𝕎)$ . The parameters are $𝜽 = (𝜶, 𝒂, 𝜷, 𝛀, 𝜸, 𝚪)$ , where $𝜶 = (π_{1}, \dots, π_{k}, 𝝁_{1}, \dots, 𝝁_{K}, 𝚺_{1}, \dots, 𝚺_{K})$ .

The complete-data density for a single observation $𝒚_{n}$ , given its membership to cluster $k$ (denoted by $z_{i k} = 1$ ), is:

f ​ (𝒚_{n} ∣ z_{i ​ k} = 1, 𝕍, 𝜽) = f_{clust} ​ (𝒚_{n}^{𝕊}; 𝝁_{k}, 𝚺_{k}) \cdot f_{reg} ​ (𝒚_{n}^{𝕌}; 𝒂 + 𝒚_{n}^{ℝ} ​ 𝜷, 𝛀) \cdot f_{indep} ​ (𝒚_{n}^{𝕎}; 𝜸, 𝚪)

The marginal complete-data density for $𝒚_{n}$ ,by summing over latent cluster memberships is:

f ​ (𝒚_{n}; 𝕍, 𝜽) = \sum_{k = 1}^{K} π_{k} ​ [f_{clust} ​ (𝒚_{n}^{𝕊}; 𝝁_{k}, 𝚺_{k}) \cdot f_{reg} ​ (𝒚_{n}^{𝕌}; 𝒂 + 𝒚_{n}^{ℝ} ​ 𝜷, 𝛀) \cdot f_{indep} ​ (𝒚_{n}^{𝕎}; 𝜸, 𝚪)]

(10)

Due to the independence of $𝒚_{n}^{𝕎}$ from $(𝒚_{n}^{𝕊}, 𝒚_{n}^{𝕌})$ given the cluster $k$ and its parameters $𝜸, 𝚪$ are not $k$ -specific, we can rewrite the term inside the sum as follows: suppose that $f_{k} (𝒚_{n}^{𝕊}, 𝒚_{n}^{𝕌}) = Φ (𝒚_{n}^{𝕊}; 𝝁_{k}, 𝚺_{k}) \cdot Φ (𝒚_{n}^{𝕌}; 𝒂 + 𝒚_{n}^{ℝ} 𝜷, 𝛀)$ . This is the density of $(𝒚_{n}^{𝕊}, 𝒚_{n}^{𝕌})$ given cluster $k$ . Then, Equation 10 becomes:

f ​ (𝒚_{n}; 𝕍, 𝜽) = (\sum_{k = 1}^{K} π_{k} ​ f_{k} ​ (𝒚_{n}^{𝕊}, 𝒚_{n}^{𝕌})) \cdot f_{indep} ​ (𝒚_{n}^{𝕎}; 𝜸, 𝚪)

The term $Φ (𝒚_{n}^{𝕎}; 𝜸, 𝚪)$ factors out of the sum over $k$ because $𝜸$ and $𝚪$ are not indexed by $k$ .

Let $𝒚_{n} = (𝒚_{n}^{o}, 𝒚_{n}^{m})$ be the partition of $𝒚_{n}$ into observed and missing parts for observation $n$ . The missing parts are $𝒚_{n}^{m} = (𝒚_{n}^{𝕊 \cap 𝕄_{n}}, 𝒚_{n}^{𝕌 \cap 𝕄_{n}}, 𝒚_{n}^{𝕎 \cap 𝕄_{n}})$ , where $𝕄_{n}$ denotes the set of missing variable indices for observation $n$ . The observed-data density for observation $n$ is obtained by integrating $f (𝒚_{n}; 𝕍, 𝜽)$ over $𝒚_{n}^{m}$ :

	$f (𝒚_{n}^{o}; 𝕍, 𝜽)$	$= \int f (𝒚_{n}^{o}, 𝒚_{n}^{m}; 𝕍, 𝜽) 𝑑 𝒚_{n}^{m}$
		$= \int [(\sum_{k = 1}^{K} π_{k} f_{k} (𝒚_{n}^{𝕊 \cap 𝕆_{n}}, 𝒚_{n}^{𝕊 \cap 𝕄_{n}}, 𝒚_{n}^{𝕌 \cap 𝕆_{n}}, 𝒚_{n}^{𝕌 \cap 𝕄_{n}}))$
		$\times f_{indep} (𝒚_{n}^{𝕎 \cap 𝕆_{n}}, 𝒚_{n}^{𝕎 \cap 𝕄_{n}}; 𝜸, 𝚪)] d 𝒚_{n}^{𝕊 \cap 𝕄_{n}} d 𝒚_{n}^{𝕌 \cap 𝕄_{n}} d 𝒚_{n}^{𝕎 \cap 𝕄_{n}}$

where $𝕆_{n}$ denotes the set of observed variable indices for observation $n$ . We can swap the sum and the integral:

	$f (𝒚_{n}^{o}; 𝕍, 𝜽)$	$= \sum_{k = 1}^{K} π_{k} \int [f_{k} (𝒚_{n}^{𝕊 \cap 𝕆_{n}}, 𝒚_{n}^{𝕊 \cap 𝕄_{n}}, 𝒚_{n}^{𝕌 \cap 𝕆_{n}}, 𝒚_{n}^{𝕌 \cap 𝕄_{n}})$
		$\times f_{indep} (𝒚_{n}^{𝕎 \cap 𝕆_{n}}, 𝒚_{n}^{𝕎 \cap 𝕄_{n}}; 𝜸, 𝚪)] d 𝒚_{n}^{𝕊 \cap 𝕄_{n}} d 𝒚_{n}^{𝕌 \cap 𝕄_{n}} d 𝒚_{n}^{𝕎 \cap 𝕄_{n}}$

Since $f_{k} (\cdot)$ only involves variables in $𝕊$ and $𝕌$ , and $f_{clust} (\cdot; 𝜸, 𝚪)$ only involves variables in $𝕎$ , and these sets are disjoint, the integration can be separated:

	$f (𝒚_{n}^{o}; 𝕍, 𝜽) = \sum_{k = 1}^{K} π_{k}$	$[\int f_{k} (𝒚_{n}^{𝕊 \cap 𝕆_{n}}, 𝒚_{n}^{𝕊 \cap 𝕄_{n}}, 𝒚_{n}^{𝕌 \cap 𝕆_{n}}, 𝒚_{n}^{𝕌 \cap 𝕄_{n}}) 𝑑 𝒚_{n}^{𝕊 \cap 𝕄_{n}} 𝑑 𝒚_{n}^{𝕌 \cap 𝕄_{n}}]$
		$\times [\int f_{indep} (𝒚_{n}^{𝕎 \cap 𝕆_{n}}, 𝒚_{n}^{𝕎 \cap 𝕄_{n}}; 𝜸, 𝚪) 𝑑 𝒚_{n}^{𝕎 \cap 𝕄_{n}}]$

Let $f_{k} (𝒚_{n}^{(𝕊, 𝕌) \cap 𝕆_{n}}) = \int f_{k} (𝒚_{n}^{𝕊 \cap 𝕆_{n}}, 𝒚_{n}^{𝕊 \cap 𝕄_{n}}, 𝒚_{n}^{𝕌 \cap 𝕆_{n}}, 𝒚_{n}^{𝕌 \cap 𝕄_{n}}) 𝑑 𝒚_{n}^{𝕊 \cap 𝕄_{n}} 𝑑 𝒚_{n}^{𝕌 \cap 𝕄_{n}}$ . This is the marginal density of the observed parts of $(𝕊, 𝕌)$ variables for component $k$ . It is Gaussian, $Φ (𝒚_{n}^{(𝕊, 𝕌) \cap 𝕆_{n}}; 𝝂_{k, o}^{(𝕊, 𝕌) (n)}, 𝚫_{k, o o}^{(𝕊, 𝕌) (n)})$ , where these parameters are derived from the complete-data $𝕊 ℝ$ parameters for the $(𝕊, 𝕌)$ part. Let $Φ (𝒚_{n}^{𝕎 \cap 𝕆_{n}}; 𝜸_{o}^{(n)}, 𝚪_{o o}^{(n)}) = \int Φ (𝒚_{n}^{𝕎 \cap 𝕆_{n}}, 𝒚_{n}^{𝕎 \cap 𝕄_{n}}; 𝜸, 𝚪) 𝑑 𝒚_{n}^{𝕎 \cap 𝕄_{n}}$ . This is the marginal density of the observed parts of $𝕎$ variables. Then,

f ​ (𝒚_{n}^{o}; 𝕍, 𝜽) = (\sum_{k = 1}^{K} π_{k} ​ f_{clust} ​ (𝒚_{n}^{(𝕊, 𝕌) \cap 𝕆_{n}}; 𝝂_{k, o}^{(𝕊, 𝕌) ​ (n)}, 𝚫_{k, o ​ o}^{(𝕊, 𝕌) ​ (n)})) \cdot f_{indep} ​ (𝒚_{n}^{𝕎 \cap 𝕆_{n}}; 𝜸_{o}^{(n)}, 𝚪_{o ​ o}^{(n)})

(11)

From Equation 11, we can get

\prod_{n = 1}^{N} (\sum_{k = 1}^{K} π_{k} ​ f_{clust} ​ (𝒚_{n}^{o}; {\tilde{𝝂}}_{k, o}^{(n)}, {\tilde{𝚫}}_{k, o ​ o}^{(n)}))

by allowing $f_{indep} (𝒚_{n}^{𝕎 \cap 𝕆_{n}}; 𝜸_{o}^{(n)}, 𝚪_{o o}^{(n)})$ to be “absorbed” into the sum over $k$ . So, we can write:

f ​ (𝒚_{n}^{o}; 𝕍, 𝜽)

= \sum_{k = 1}^{K} (π_{k} ​ f_{clust} ​ (𝒚_{n}^{(𝕊, 𝕌) \cap 𝕆_{n}}; 𝝂_{k, o}^{(𝕊, 𝕌) ​ (n)}, 𝚫_{k, o ​ o}^{(𝕊, 𝕌) ​ (n)}) \cdot f_{indep} ​ (𝒚_{n}^{𝕎 \cap 𝕆_{n}}; 𝜸_{o}^{(n)}, 𝚪_{o ​ o}^{(n)}))

Since the variables in $(𝕊, 𝕌)$ and $𝕎$ are conditionally independent given $k$ , the product of their marginal observed densities is the marginal observed density of their union. Let $𝒚_{n}^{o} = (𝒚_{n}^{(𝕊, 𝕌) \cap 𝕆_{n}}, 𝒚_{n}^{𝕎 \cap 𝕆_{n}})$ . Let ${\tilde{𝝂}}_{k, o}^{(n)} = {({(𝝂_{k, o}^{(𝕊, 𝕌) (n)})}^{T}, {(𝜸_{o}^{(n)})}^{T})}^{T}$ . Let ${\tilde{𝚫}}_{k, o o}^{(n)} = (\begin{matrix} 𝚫_{k, o o}^{(𝕊, 𝕌) (n)} & 𝟎 \\ 𝟎 & 𝚪_{o o}^{(n)} \end{matrix})$ . Then, $f_{clust} (𝒚_{n}^{(𝕊, 𝕌) \cap 𝕆_{n}}; 𝝂_{k, o}^{(𝕊, 𝕌) (n)}, 𝚫_{k, o o}^{(𝕊, 𝕌) (n)}) \cdot f_{indep} (𝒚_{n}^{𝕎 \cap 𝕆_{n}}; 𝜸_{o}^{(n)}, 𝚪_{o o}^{(n)}) = f_{clust} (𝒚_{n}^{o}; {\tilde{𝝂}}_{k, o}^{(n)}, {\tilde{𝚫}}_{k, o o}^{(n)})$ .

Therefore, the observed-data density for a single observation $n$ is:

f ​ (𝒚_{n}^{o}; 𝕍, 𝜽) = \sum_{k = 1}^{K} π_{k} ​ f_{clust} ​ (𝒚_{n}^{o}; {\tilde{𝝂}}_{k, o}^{(n)}, {\tilde{𝚫}}_{k, o ​ o}^{(n)})

(12)

And for $N$ i.i.d. observations, the total observed-data likelihood is:

\prod_{n = 1}^{N} f ​ (𝒚_{n}^{o}; K, m, r, l, 𝕍, 𝜽) = \prod_{n = 1}^{N} (\sum_{k = 1}^{K} π_{k} ​ f_{clust} ​ (𝒚_{n}^{o}; {\tilde{𝝂}}_{k, o}^{(n)}, {\tilde{𝚫}}_{k, o ​ o}^{(n)}))

(13)

∎

When the parameters meet some distinct properties, $𝕊 ℝ 𝕌 𝕎$ under complete-data can achieve identifiability, as shown in Theorem 1.

Fact 1 (Theorem 1 in [38]).

Let $𝚯_{(K, m, r, l, 𝕍)}$ be a subset of the parameter set $Υ_{(K, m, r, l, 𝕍)}$ such that elements $𝛉 = (𝛂, 𝐚, 𝛃, 𝛀, 𝛄, 𝚪)$

•

contain distinct couples $(μ_{k}, Σ_{k})$ fulfilling $\forall s \subseteq 𝕊, \exists (k, k^{'}), 1 \leq k < k^{'} \leq K;$

$μ_{k, \bar{s} | s} \neq μ_{k^{'}, \bar{s} | s} or Σ_{k, \bar{s} | s} \neq Σ_{k^{'}, \bar{s} | s} or Σ_{k, \bar{s} \bar{s} | s} \neq Σ_{k^{'}, \bar{s} \bar{s} | s}$ (14)

where $\bar{s}$ denotes the complement in $𝕊$ of any nonempty subset $s$ of $𝕊$
•
if $𝕌 \neq \emptyset$ ,
- –
  
  for all variables $j$ of $ℝ$ , there exists a variable $u$ of $𝕌$ such that the restriction $𝜷_{u j}$ of the regression coefficient matrix $𝜷$ associated to $j$ and $u$ is not equal to zero.
- –
  
  for all variables $u$ of $𝕌$ , there exists a variable $j$ of $ℝ$ such that $𝜷_{u j} \neq 0$ .
•

parameters $Ω$ and $τ$ exactly respect the forms $r$ and $l$ respectively. They are both diagonal matrices with at least two different eigenvalues if $r = [L B]$ and $l = [L B]$ and $Ω$ has at least a non-zero entry outside the main diagonal if $r = [L C]$ .

Let $(K, m, r, l, 𝕍)$ and $(K^{*}, m^{*}, r^{*}, l^{*}, 𝕍^{*})$ be two models. If there exist $𝛉 \in 𝚯_{(K, m, r, l, 𝕍)}$ and $𝛉^{*} \in 𝚯_{(K^{*}, m^{*}, r^{*}, l^{*}, 𝕍^{*})}$ such that

f (\cdot | K, m, r, l, 𝕍, 𝜽) = f (\cdot | K^{*}, m^{*}, r^{*}, l^{*}, 𝕍^{*}, 𝜽^{*})

then $(K, m, r, l, 𝕍) = (K^{*}, m^{*}, r^{*}, l^{*}, 𝕍^{*})$ and $𝛉 = 𝛉^{*}$ (up to a permutation of mixture components).

Appendix C Regular Assumptions for Main Theoretical Results

For a fixed tuple of $(K_{0}, m_{0}, r_{0}, l_{0}, 𝕍_{0})$ , we can write $f (𝒚; 𝜽)$ instead of $f (𝒚; K_{0}, m_{0}, 𝕊_{0}, ℝ_{0}, 𝜽)$ or $f (𝒚; 𝕍, 𝜽)$ for short notation.

Assumption 1 ( $𝕊 ℝ$ ).

There exists a unique $(K_{0}, m_{0}, 𝕊_{0}, ℝ_{0})$ such that

h = f ​ (\cdot; 𝜽_{(K_{0}, m_{0}, 𝕊_{0}, ℝ_{0})}^{*})

for some parameter value $𝛉^{*}$ , and the pair $(K_{0}, m_{0})$ is assumed known. (In the following, we omit explicit notation of the dependence on $(K_{0}, m_{0})$ for brevity.)

Assumption 2.

The vectors $𝛉_{(𝕊, ℝ)}^{*}$ and ${\hat{𝛉}}_{(𝕊, ℝ)}$ belong to a compact subset

𝚯_{(𝕊, ℝ)}^{'} \subset 𝚯_{(𝕊, ℝ)}

defined by

𝚯_{(𝕊, ℝ)}^{'} = 𝒫_{K_{0} - 1} \times ℬ ​ {(η, | 𝕊 |)}^{K_{0}} \times 𝒟_{| 𝕊 |}^{K_{0}} \times ℬ ​ (ρ, | 𝕊^{c} |) \times ℬ ​ (ρ, | ℝ |, | 𝕊^{c} |) \times 𝒟_{| 𝕊^{c} |}

where the components are defined as follows:

•

$𝒫_{K - 1} = {(π_{1}, \dots, π_{K}) \in {[0, 1]}^{K} : \sum_{k = 1}^{K} π_{k} = 1}$ (set of proportions);
•

$ℬ (η, r) = {x \in ℝ^{r} : ‖ x ‖ \leq η}$ with $‖ x ‖ = \sqrt{\sum_{i = 1}^{r} x_{i}^{2}}$ ;
•

$ℬ (ρ, q, r) = {A \in ℳ_{q \times r} (ℝ) : ‖ | A | ‖ \leq ρ}$ , with norm

$‖ | A | ‖ = sup_{y \in ℝ^{r}, ‖ y ‖ = 1} ‖ A y ‖;$
•

$𝒟_{r}$ is the set of $r \times r$ positive definite matrices with eigenvalues in $[s_{m}, s_{M}]$ (with $0 < s_{m} < s_{M}$ )

Assumption 3.

The optimal parameter $𝛉_{(𝕊_{0}, ℝ_{0})}^{*}$ is an interior point of $𝚯_{(𝕊_{0}, ℝ_{0})}^{'}$ .

The assumptions can be extended to the $𝕊 ℝ 𝕌 𝕎$ framework. We define the optimal parameter as $𝜽_{𝕍}^{*}$ , and the maximum likelihood estimator as ${\hat{𝜽}}_{𝕍}$ . In this modified framework, the assumptions become:

Assumption 4 ( $𝕊 ℝ 𝕌 𝕎$ ).

There exists a unique $(K_{0}, m_{0}, r_{0}, l_{0}, 𝕍_{0})$ such that

h = f ​ (\cdot; 𝜽_{(K_{0}, m_{0}, r_{0}, l_{0}, 𝕍_{0})}^{⋆})

for some $𝛉^{⋆}$ , and the model $(K_{0}, m_{0}, r_{0}, l_{0}, 𝕍_{0})$ is assumed known.

Assumption 5.

Vectors $𝛉_{𝕍}^{*}$ and ${\hat{𝛉}}_{𝕍}$ belong to the compact subspace $𝚯_{𝕍}^{'}$ of the compact set $𝚯_{𝕍}$ , defined by:

𝚯_{𝕍}^{'}

= 𝒫_{K_{0} - 1} \times ℬ ​ {(η, | 𝕊 |)}^{K_{0}} \times 𝒟_{| 𝕊 |}^{K_{0}} \times ℬ ​ (ρ, | 𝕌 |) \times ℬ ​ (ρ, | ℝ |, | 𝕌 |) \times 𝒟_{| 𝕌 |} \times ℬ ​ (η, | 𝕎 |) \times 𝒟_{| 𝕎 |} .

Assumption 6.

The optimal parameter $𝛉_{𝕍}^{*}$ is an interior point of $𝚯_{𝕍}^{'}$ .

Appendix D Proof of Main Results

Throughout the proofs, we denote the full parameter vector of the $K$ -component Gaussian Mixture Model (GMM) by $𝜶 = (𝝅, {𝝁_{k}}_{k = 1}^{K}, {𝚿_{k}}_{k = 1}^{K})$ , where $𝝅$ are the mixing proportions, $𝝁_{k} \in ℝ^{D}$ are the component means for $D$ variables, and $𝚿_{k}$ are the component precision matrices (inverse of covariance matrices $𝚺_{k}$ ). The true parameter vector is denoted by $𝜶^{*}$ . The negative single-observation log-likelihood is $ℓ_{1} (𝒚_{n}; 𝜶) = - \ln f_{clust} (𝒚_{n}; 𝜶)$ , and its sample average is $ℓ_{N} (𝜶)$ . The score function components are $S_{j} (𝒚; 𝜶^{*}) = - {\frac{\partial ℓ_{1} (𝒚; 𝜶)}{\partial α_{j}} |}_{𝜶 = 𝜶^{*}}$ . The Fisher Information Matrix (FIM) is $𝑰 (𝜶^{*})$ , and the empirical Hessian is $𝑯_{N} (𝜶)$ . The set of true non-zero penalized parameters in $𝜶^{*}$ is $𝕊_{0}$ , with cardinality $s_{0}$ . The error vector is $𝚫 = \hat{𝜶} - 𝜶^{*}$ . For ranking, $𝒪_{K} (j)$ is the ranking score for variable $j$ based on a grid of regularization parameters $𝒢_{λ}$ . Standard notations like $𝔼 [\cdot]$ for expectation, $∥ \cdot ∥_{1}, ∥ \cdot ∥_{2}, ∥ \cdot ∥_{\infty}$ for $L_{1}, L_{2}, L_{\infty}$ norms. Other abbreviations include KKT (Karush-Kuhn-Tucker), RSC (Restricted Strong Convexity), and SRUW (for the variable role framework).

D.1 Proof of Theorem 1

Proof Sketch. The proof involves two main steps. First, we establish that equality of observed-data likelihoods under MAR implies equality of the parameters of an equivalent full-data GMM representation for the $𝕊 ℝ 𝕌 𝕎$ model. Second, we apply the identifiability arguments for the complete-data $𝕊 ℝ 𝕌 𝕎$ model similar to [38], which itself relies on the identifiability of the $𝕊 ℝ$ model [37]

Proof.

Let $ℳ_{1} = (K, m, r, l, 𝕍, 𝜽)$ and $ℳ_{2} = (K^{*}, m^{*}, r^{*}, l^{*}, 𝕍^{*}, 𝜽^{*})$ be two $𝕊 ℝ 𝕌 𝕎$ models with $ℳ_{1}, ℳ_{2} \in 𝚯_{(K, m, r, l, 𝕍)}$ . The complete-data density for an observation $𝒚_{n}$ under $ℳ_{1}$ and Equation 12 is given by:

f ​ (𝒚_{n}^{o}; K, m, r, l, 𝕍, 𝜽, 𝑴_{n}) = \sum_{k = 1}^{K} p_{k} ​ f_{clust} ​ (𝒚_{n}^{o}; {\tilde{𝝂}}_{k, o}^{(n)}, {\tilde{𝚫}}_{k, o ​ o}^{(n)}),

Analogously, we can derive the same expression for $ℳ_{2}$ . Consider the specific missingness pattern $𝑴_{0}$ where no data are missing for observation $n$ . For this pattern, $𝒚_{n}^{o} = 𝒚_{n}$ , ${\tilde{𝝂}}_{k, o}^{(n)} = {\tilde{𝝂}}_{k}$ , and ${\tilde{𝚫}}_{k, o o}^{(n)} = {\tilde{𝚫}}_{k}$ . Under the premise of the theorem, if the equality of observed-data likelihoods for pattern $𝑴_{0}$ , we get:

\sum_{k = 1}^{K} π_{k} ​ f_{clust} ​ (𝒚_{n}; {\tilde{𝝂}}_{k}, {\tilde{𝚫}}_{k}) = \sum_{k^{'} = 1}^{K^{*}} p_{k^{'}}^{*} ​ f_{clust} ​ (𝒚_{n}; {\tilde{𝝂}}_{k^{'}}^{*}, {\tilde{𝚫}}_{k^{'}}^{*}), \forall 𝒚_{n} \in ℝ^{D} .

From theorem 1 in [38], since $𝕊 ℝ 𝕌 𝕎$ are identifiable under complete-data scenario this implies $K = K^{*}$ , and up to a permutation of component labels:

π_{k} = π_{k}^{*}, {\tilde{𝝂}}_{k} = {\tilde{𝝂}}_{k}^{*}, and {\tilde{𝚫}}_{k} = {\tilde{𝚫}}_{k}^{*} for all ​ k = 1, \dots, K .

(15)

For any $𝒚 \in ℝ^{D}$ :

f_{reg} ​ (𝒚^{𝕌}; r, 𝒂 + 𝒚^{ℝ} ​ 𝜷, 𝛀) ​ f_{indep} ​ (𝒚^{𝕎}; l, 𝜸, 𝚪) = f_{clust} ​ (𝒚^{𝕌 \cup 𝕎}; \overset{ˇ}{𝒂} + 𝒚^{ℝ} ​ \overset{ˇ}{𝜷}, \overset{ˇ}{𝛀})

where $𝕊^{c} = 𝕌 \cup 𝕎$ , $\overset{ˇ}{𝒂} = {({(𝒂)}^{T}, {(𝜸)}^{T})}^{T}$ , $\overset{ˇ}{𝜷} = (\begin{matrix} 𝜷 \\ 𝟎 \end{matrix})$ (with $| 𝟎 | = | 𝕎 | \times | ℝ |$ ), and $\overset{ˇ}{𝛀} = blockdiag (𝛀, 𝚪)$ . Here, $blockdiag (𝛀, 𝚪)$ is the block diagonal matrix created by aligning the input matrices $𝛀, 𝚪$ . The $𝕊 ℝ 𝕌 𝕎$ density can be expressed as an equivalent $𝕊 ℝ$ model density:

	$f (𝒚; K, m, r, l, 𝕍, 𝜽)$	$= f_{clust} (𝒚^{𝕊}; K, m, 𝜶) f_{clust} (𝒚^{𝕊^{c}}; \overset{ˇ}{𝒂} + 𝒚^{ℝ} \overset{ˇ}{𝜷}, \overset{ˇ}{𝛀})$
		$= f_{𝕊 ℝ -eq} (𝒚; K, m, 𝕊, ℝ, 𝜽_{𝕊 ℝ -eq})$

where $𝜽_{𝕊 ℝ -eq} = (𝜶, \overset{ˇ}{𝒂}, \overset{ˇ}{𝜷}, \overset{ˇ}{𝛀})$ . From Section D.1, the equivalent $𝕊 ℝ$ densities for the two models, where the second model is denoted with ^∗) must be equal:

f_{𝕊 ​ ℝ -eq} ​ (𝒚; K, m, 𝕊, ℝ, 𝜽_{𝕊 ​ ℝ -eq}) = f_{𝕊 ​ ℝ -eq} ​ (𝒚; K^{*}, m^{*}, 𝕊^{*}, ℝ^{*}, 𝜽_{𝕊 ​ ℝ -eq}^{*})

By the identifiability of the $𝕊 ℝ$ model [37] and three conditions in Theorem 1 , we deduce: $K = K^{*}$ , $m = m^{*}$ , $𝕊 = 𝕊^{*}$ , $ℝ = ℝ^{*}$ , $𝜶 = 𝜶^{*}$ , $\overset{ˇ}{𝒂} = {\overset{ˇ}{𝒂}}^{*}$ , $\overset{ˇ}{𝜷} = {\overset{ˇ}{𝜷}}^{*}$ , and $\overset{ˇ}{𝛀} = {\overset{ˇ}{𝛀}}^{*}$ .

Since $𝕊 = 𝕊^{*}$ and $𝕊^{c} = 𝕊^{* c}$ , supposed that, for contradiction, that $𝕌^{*} \cap 𝕎 \neq \emptyset$ . Let $j \in 𝕌^{*} \cap 𝕎$ . Since $j \in 𝕎$ , for all $q \in 𝕌^{*}$ the rows of ${(\overset{ˇ}{𝜷})}_{q j}$ are $𝟎^{T}$ . Additionally, since $j \in 𝕌^{*}$ , there exists some $q \in ℝ * = ℝ$ such that ${({\overset{ˇ}{𝜷}}^{*})}_{q j} \neq 𝟎^{T}$ . However, we have established $\overset{ˇ}{𝜷} = {\overset{ˇ}{𝜷}}^{*}$ . This is a contradiction: ${(\overset{ˇ}{𝜷})}_{q j}$ must be simultaneously $𝟎^{T}$ and non-zero for all $q \in s R$ . Therefore, $𝕌^{*} \cap 𝕎 = \emptyset$ . By symmetry, $𝕌 \cap 𝕎^{*} = \emptyset$ . Given $𝕌 \cup 𝕎 = 𝕌^{*} \cup 𝕎^{*}$ , the disjoint conditions imply $𝕌 \subseteq 𝕌^{*}$ and $𝕌^{*} \subseteq 𝕌$ , hence $𝕌 = 𝕌^{*}$ . Consequently, $𝕎 = 𝕎^{*}$ . This establishes $𝕍 = 𝕍^{*}$ . The parameter equalities $𝒂 = 𝒂^{*}, 𝜷 = 𝜷^{*}, 𝜸 = 𝜸^{*}, 𝛀 = 𝛀^{*}, 𝚪 = 𝚪^{*}$ and $r = r^{*}, l = l^{*}$ then follow directly from comparing the components of $\overset{ˇ}{𝒂} = {\overset{ˇ}{𝒂}}^{*}$ , $\overset{ˇ}{𝜷} = {\overset{ˇ}{𝜷}}^{*}$ , $\overset{ˇ}{𝛀} = {\overset{ˇ}{𝛀}}^{*}$ , condition 3 in Theorem 1. Thus, the SRUW parameters under MAR are identifiable.

Now, we extend this to the full model which includes the MNARz mechanism parameters $𝝍 = {ψ_{k}}_{k = 1}^{K}$ . The full set of parameters for the MNARz-SRUW model is $(𝜽, 𝝍)$ . Suppose that the two models $ℳ_{A}$ and $ℳ_{B}$ produce the same observed-data likelihood for all observed data $(𝒚_{n}^{o}, 𝒄_{n})$ :

L_{MNARz-SRUW} ​ (𝒚_{n}^{o}, 𝒄_{n}; 𝜽_{A}^{data}, 𝝍_{A}^{MNARz}) = L_{MNARz-SRUW} ​ (𝒚_{n}^{o}, 𝒄_{n}; 𝜽_{B}^{data}, 𝝍_{B}^{MNARz}) \forall (𝒚_{n}^{o}, 𝒄_{n}),

By Theorem 5, the equality of observed-data likelihoods under MNARz-SRUW:

L_{MNARz-SRUW} ​ (𝒚_{n}^{o}, 𝒄_{n}; 𝜽_{A}^{data}, 𝝍_{A}^{MNARz}) = L_{MNARz-SRUW} ​ (𝒚_{n}^{o}, 𝒄_{n}; 𝜽_{B}^{data}, 𝝍_{B}^{MNARz})

for all $(𝒚_{n}^{o}, 𝒄_{n})$ implies the equality of the likelihoods for the augmented observed data ${\tilde{𝒚}}_{n}^{o} = (𝒚_{n}^{o}, 𝒄_{n})$ under the MAR interpretation where $f (𝒄_{n} | z_{n k} = 1; ψ_{k})$ is treated as part of the component density:

		$\sum_{k = 1}^{K_{A}} {(π_{A})}_{k} f_{k} (𝒚_{n}^{o} \| z_{n k} = 1; {(θ_{A}^{SRUW})}_{k}) f (𝒄_{n} \| z_{n k} = 1; {(ψ_{A})}_{k})$
		$= \sum_{k^{'} = 1}^{K_{B}} {(π_{B})}_{k^{'}} f_{k^{'}} (𝒚_{n}^{o} \| z_{n k^{'}} = 1; {(θ_{B}^{SRUW})}_{k^{'}}) f (𝒄_{n} \| z_{n k^{'}} = 1; {(ψ_{B})}_{k^{'}})$		(16)

for all $(𝒚_{n}^{o}, 𝒄_{n})$ . Let $g_{k} (𝒚_{n}^{o}, 𝒄_{n}; {(θ_{A})}_{k}, {(ψ_{A})}_{k}) = f_{k} (𝒚_{n}^{o} | z_{n k} = 1; {(θ_{A}^{SRUW})}_{k}) f (𝒄_{n} | z_{n k} = 1; {(ψ_{A})}_{k})$ . This means two finite mixture models on the augmented data $(𝒚_{n}^{o}, 𝒄_{n})$ are identical. If $\sum_{k = 1}^{K_{A}} {(π_{A})}_{k} g_{k} (\cdot) = \sum_{k^{'} = 1}^{K_{B}} {(π_{B})}_{k^{'}} g_{k^{'}}^{'} (\cdot)$ , and the family of component densities ${g_{k}}$ (and ${g_{k^{'}}^{'}}$ ) is identifiable and satisfies certain linear independence conditions ([68], [2]), then $K_{A} = K_{B} = K$ , and after a permutation of labels, ${(π_{A})}_{k} = {(π_{B})}_{k}$ and $g_{k} (\cdot) = g_{k}^{'} (\cdot)$ for all $k$ . Then, we need to ensure the family of component densities $h_{k} (𝒚_{n}^{o}, 𝒄_{n}; θ_{k}, ψ_{k}) = f_{k} (𝒚_{n}^{o} | z_{n k} = 1; θ_{k}) f (𝒄_{n} | z_{n k} = 1; ψ_{k})$ meets such conditions. Hence, we further assume:

1.

The parameter sets $𝜽^{SRUW}$ and $𝝍^{MNARz}$ are functionally independent
2.

The support of $𝒚_{n}$ and $𝒄_{n}$ is rich enough such that equalities holding for all $(𝒚_{n}, 𝒄_{n})$ imply functional equalities. This implies there is sufficient variability of $𝒚_{n}$ and $𝒄_{n}$ .

These assumptions hold because $ψ$ parametrizes only $f (𝒄_{n} | 𝒛)$ and $θ$ only $f (𝒚_{n}^{o} | 𝒛)$ ; hence there is no functional overlap. Applying those assumptions to Section D.1, we conclude that $K_{A} = K_{B} = K$ , and up to a permutation of labels:

{(π_{A})}_{k} = {(π_{B})}_{k} for all ​ k,

(17)

and

\begin{matrix} f_{k} ​ (𝒚_{n}^{o} | z_{n ​ k} = 1; {(θ_{A}^{SRUW})}_{k}) ​ f ​ (𝒄_{n} | z_{n ​ k} = 1; {(ψ_{A})}_{k}) \\ = \\ f_{k} ​ (𝒚_{n}^{o} | z_{n ​ k} = 1; {(θ_{B}^{SRUW})}_{k}) ​ f ​ (𝒄_{n} | z_{n ​ k} = 1; {(ψ_{B})}_{k}) \end{matrix}

(18)

for all $(𝒚_{n}^{o}, 𝒄_{n})$ and for each $k = 1, \dots, K$ .

Now we need to deduce that ${(θ_{A}^{SRUW})}_{k} = {(θ_{B}^{SRUW})}_{k}$ and ${(ψ_{A})}_{k} = {(ψ_{B})}_{k}$ . Equation 18 can be written as:

A_{1} ​ (𝒚_{n}^{o}) ​ B_{1} ​ (𝒄_{n}) = A_{2} ​ (𝒚_{n}^{o}) ​ B_{2} ​ (𝒄_{n})

where $A_{1} (𝒚_{n}^{o}) = f_{k} (𝒚_{n}^{o} | z_{n k} = 1; {(θ_{A}^{SRUW})}_{k})$ , $B_{1} (𝒄_{n}) = f (𝒄_{n} | z_{n k} = 1; {(ψ_{A})}_{k})$ . By defined assumptions, this equality holding for all $𝒚_{n}^{o}, 𝒄_{n}$ implies a relationship between these functions. Since $𝒚_{n}^{o}$ and $𝒄_{n}$ can vary independently ¹¹1Since, for each component $k$ , the joint component density factorizes as $f_{k} (𝒚_{n}^{o} | z_{n k} = 1; θ_{k}) f (𝒄_{n} | z_{n k} = 1; ψ_{k})$ and both factors are positive on sets of positive measure, pick $c_{0}$ in that common positive-measure set; integrating $A_{1} (𝒚_{n}^{o}) B_{1} (c_{0}) = A_{2} (𝒚_{n}^{o}) B_{2} (c_{0})$ over $𝒚_{n}^{o}$ gives $B_{1} (c_{0}) = B_{2} (c_{0})$ , hence $A_{1} = A_{2}$ , and then $B_{1} = B_{2}$ . and the parameters ${(θ_{A})}_{k}, {(ψ_{A})}_{k}$ are functionally independent, if $\int A_{1} B_{1} 𝑑 𝒚^{o} 𝑑 𝒄 = \int A_{2} B_{2} 𝑑 𝒚^{o} 𝑑 𝒄 = 1$ , and $A_{1} B_{1} = A_{2} B_{2}$ pointwise, and assuming $A_{1}, B_{1}, A_{2}, B_{2}$ are non-zero almost everywhere on their support: Pick a $𝒄_{n}^{0}$ such that $f (𝒄_{n}^{0} | k; {(ψ_{A})}_{k}) \neq 0$ and $f (𝒄_{n}^{0} | k; {(ψ_{B})}_{k}) \neq 0$ . Then for this fixed $𝒄_{n}^{0}$ :

f_{k} ​ (𝒚_{n}^{o} | z_{n ​ k} = 1; {(θ_{A}^{SRUW})}_{k}) \cdot C_{A} = f_{k} ​ (𝒚_{n}^{o} | z_{n ​ k} = 1; {(θ_{B}^{SRUW})}_{k}) \cdot C_{B}

where $C_{A} = f (𝒄_{n}^{0} | k; {(ψ_{A})}_{k})$ and $C_{B} = f (𝒄_{n}^{0} | k; {(ψ_{B})}_{k})$ . Since both $f_{k} (𝒚_{n}^{o} | \dots)$ are conditional densities , this implies $C_{A} = C_{B}$ and thus

f_{k} ​ (𝒚_{n}^{o} | z_{n ​ k} = 1; {(θ_{A}^{SRUW})}_{k}) = f_{k} ​ (𝒚_{n}^{o} | z_{n ​ k} = 1; {(θ_{B}^{SRUW})}_{k}) for all ​ 𝒚_{n}^{o} .

By identifiability of SRUW under MAR, this implies that the structural SRUW parameters $(m_{A}, r_{A}, l_{A}, 𝕍_{A})$ must equal $(m_{B}, r_{B}, l_{B}, 𝕍_{B})$ and the parameters ${(θ_{A}^{SRUW})}_{k} = {(θ_{B}^{SRUW})}_{k}$ .

Since $C_{A} = C_{B}$ means $f (𝒄_{n}^{0} | k; {(ψ_{A})}_{k}) = f (𝒄_{n}^{0} | k; {(ψ_{B})}_{k})$ , and this must hold for all $𝒄_{n}^{0}$ (by choosing different fixed $𝒄_{n}^{0}$ or by varying $𝒚_{n}^{o}$ first), this implies:

f ​ (𝒄_{n} | k; {(ψ_{A})}_{k}) = f ​ (𝒄_{n} | k; {(ψ_{B})}_{k}) for all ​ 𝒄_{n} .

By idenitifiability of MNARz, this implies ${(ψ_{A})}_{k} = {(ψ_{B})}_{k}$ .

Thus, we have $K_{A} = K_{B} = K$ , and up to a common permutation of labels for $k = 1, \dots, K$ :

	${(π_{A})}_{k} = {(π_{B})}_{k}$
	$(m_{A}, r_{A}, l_{A}, 𝕍_{A}) = (m_{B}, r_{B}, l_{B}, 𝕍_{B})$
	${(θ_{A}^{SRUW})}_{k} = {(θ_{B}^{SRUW})}_{k}$
	${(ψ_{A})}_{k} = {(ψ_{B})}_{k}$

This implies the full set of model specifications and parameters for $ℳ_{A}$ and $ℳ_{B}$ is identical. Therefore, the SRUW model under the specified MNARz mechanism is identifiable. ∎

D.2 Proof of Theorem 2

Proof Sketch.

Suppose that the observations are from a parametric model $𝒫_{𝚯} = {f (\cdot; 𝜽) : 𝜽 \in 𝚯}$ , and assume that the distributions in $𝒫_{𝚯}$ are dominated by a common $σ$ -finite measure $υ$ with respect to which they have probability density functions $f (\cdot; 𝜽)$ .

Now, we need to prove the consistency of the sample KL, which will lead to the consistency of the BIC criterion in both the $𝕊 ℝ$ and $𝕊 ℝ 𝕌 𝕎$ models under the MAR mechanism. First, we prove the consistency of the sample KL in the $𝕊 ℝ$ model:

Proposition 2.

Under Assumption 1 and 2, for all $(𝕊, ℝ)$ ,

\frac{1}{N} ​ \sum_{n = 1}^{N} \ln (\frac{h ​ (𝒚_{n}^{o})}{f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{(𝕊, ℝ)})}) ​ \to_{n \to \infty}^{𝑃} ​ D_{KL} ​ [h, f ​ (\cdot; 𝜽_{(𝕊, ℝ)}^{*})] .

Proof.

Let $𝕆 \subset {1, \dots, D}$ denote the indices of the observed components.

Recall that the full covariance matrix for the component $k$ , $𝚫_{k}$ , is built as:

Δ_{k, j ​ l} = {\begin{matrix} Σ_{k, j ​ l}, & j, l \in 𝕊, \\ {(𝚺_{k} ​ 𝚲)}_{j ​ l}, & j \in 𝕊, l \in 𝕊^{c}, \\ {(𝚲^{T} ​ 𝚺_{k})}_{j ​ l}, & j \in 𝕊^{c}, l \in 𝕊, \\ {(𝛀 + 𝚲^{T} ​ 𝚺_{k} ​ 𝚲)}_{j ​ l}, & j, l \in 𝕊^{c} . \end{matrix}

For indices $j, l \in 𝕊$ : As before,

Δ_{k, j ​ l} = Σ_{k, j ​ l} .

Thus, eigenvalues of $Δ_{k, 𝕊 𝕊}$ are in $[s_{m}, s_{M}]$ .

For $j \in 𝕊, l \in 𝕊^{c}$ :

Δ_{k, j ​ l} = {(𝚺_{k} ​ 𝚲)}_{j ​ l} .

The norm of the product satisfies

∥ 𝚺_{k} ​ 𝚲 ∥ \leq ∥ 𝚺_{k} ∥ \cdot ∥ 𝚲 ∥ .

Since $𝚺_{k} \in 𝒟_{| 𝕊 |}$ ,

∥ 𝚺_{k} ∥ = λ_{\max} ​ (𝚺_{k}) \leq s_{M} .

Given $𝜷 \in ℬ (ρ, | ℝ |, | 𝕊^{c} |)$ ,

∥ 𝚲 ∥ \leq ρ .

Thus,

∥ 𝚺_{k} ​ 𝚲 ∥ \leq s_{M} ​ ρ .

This bounds the norm of the $(𝕊 \times 𝕊^{c})$ block of $𝚫_{k}$ .

For $j, l \in 𝕊^{c}$ :

Δ_{k, j ​ l} = {(𝛀 + 𝚲^{T} ​ 𝚺_{k} ​ 𝚲)}_{j ​ l} .

We can bound the eigenvalues of this sum using the spectral norm:

∥ 𝛀 + 𝚲^{T} ​ 𝚺_{k} ​ 𝚲 ∥ \leq ∥ 𝛀 ∥ + ∥ 𝚲^{T} ​ 𝚺_{k} ​ 𝚲 ∥ .

Since $𝛀 \in 𝒟_{| 𝕊^{c} |}$ ,

∥ 𝛀 ∥ = λ_{\max} ​ (𝛀) \leq s_{M} .

Next,

∥ 𝚲^{T} ​ 𝚺_{k} ​ 𝚲 ∥ \leq ∥ 𝚲^{T} ∥ \cdot ∥ 𝚺_{k} ∥ \cdot ∥ 𝚲 ∥ = {∥ 𝚲 ∥}^{2} \cdot ∥ 𝚺_{k} ∥ \leq ρ^{2} ​ s_{M} .

Thus,

∥ 𝛀 + 𝚲^{T} ​ 𝚺_{k} ​ 𝚲 ∥ \leq s_{M} + ρ^{2} ​ s_{M} = s_{M} ​ (1 + ρ^{2}) .

Given the above bounds, there exist refined constants ${\tilde{s}}_{m} > 0$ and

{\tilde{s}}_{M} = \max {s_{M} ​ (1 + ρ^{2}), s_{M} ​ ρ, s_{M}} = s_{M} ​ (1 + ρ^{2})

such that

{\tilde{s}}_{m} ​ I ⪯ 𝚫_{k} ⪯ {\tilde{s}}_{M} ​ I .

Note that ${\tilde{s}}_{M}$ will depend on the sizes of the blocks, and it is finite being dependent on $s_{M}$ , $ρ$ , and the structure of $𝛀$ , $𝜷$ , and $𝚺_{k}$ .

Since $𝚫_{k, o o}$ is a principal submatrix of $𝚫_{k}$ and $𝚫_{k}$ is a symmetric matrix, by the interlacing theorem:

{\tilde{s}}_{m} \leq λ_{\min} ​ (𝚫_{k, o ​ o}) \leq λ_{\max} ​ (𝚫_{k, o ​ o}) \leq {\tilde{s}}_{M} .

Thus,

{\tilde{s}}_{m} ​ I_{| 𝕆 |} ⪯ 𝚫_{k, o ​ o} ⪯ {\tilde{s}}_{M} ​ I_{| 𝕆 |},

and

{(𝚫_{k, o ​ o})}^{- 1} ⪯ \frac{1}{{\tilde{s}}_{m}} ​ I_{| 𝕆 |} .

We are now moving to the main proof. By hypothesis, the true observed-data density is given by

h ​ (𝒚^{o}) = \sum_{k = 1}^{K} π_{k} ​ Φ ​ (𝒚^{o}; 𝝂_{k, o}^{*}, Δ_{k, o ​ o}^{*})

and if we can show that

𝔼_{𝒚^{o}} ​ [| \ln h ​ (𝒚^{o}) |] < \infty .

then by the LLN,

\frac{1}{N} ​ \sum_{n = 1}^{N} \ln (h ​ (𝒚_{n}^{o})) ​ \to_{n \to \infty}^{𝑃} ​ 𝔼_{𝒚^{o}} ​ [\ln h ​ (𝒚^{o})] .

Moreover, the observed-data likelihood under the model is

f ​ (𝒚^{o}; 𝜽) = \sum_{k = 1}^{K} π_{k} ​ Φ ​ (𝒚^{o}; 𝝂_{k, o}, 𝚫_{k, o ​ o}) .

and we later show in Proposition 3 that

\frac{1}{N} ​ \sum_{n = 1}^{N} \ln (f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{(𝕊, ℝ)})) ​ \to_{n \to \infty}^{𝑃} ​ 𝔼_{𝒚^{o}} ​ [\ln f ​ (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*})] .

To do so, we must verify that the class of functions

ℱ_{(𝕊, ℝ)} = {𝒚^{o} \mapsto \ln f ​ (𝒚^{o}; 𝜽) : 𝜽 \in 𝚯_{(𝕊, ℝ)}^{'}}

satisfies the conditions for uniform convergence under Assumption 2. In particular, by Assumption 2, the parameter space $𝚯_{(𝕊, ℝ)}^{'}$ is compact, and for every $𝒚^{o} \in ℝ^{| 𝕆 |}$ the mapping

𝜽 \mapsto \ln f ​ (𝒚^{o}; 𝜽)

is continuous. Next, we verify that there is an $h$ -integrable envelope function $F \in ℱ_{(𝕊, ℝ)}$ . Recalling that for $𝒚^{o} \in ℝ^{| 𝕆 |}$ , the Gaussian component density is

Φ ​ (𝒚^{o}; 𝝂_{k, o}, 𝚫_{k, o ​ o}) = {(2 ​ π)}^{- \frac{| 𝕆 |}{2}} ​ {| 𝚫_{k, o ​ o} |}^{- \frac{1}{2}} ​ \exp (- \frac{1}{2} ​ {(𝒚^{o} - 𝝂_{k, o})}^{T} ​ 𝚫_{k, o ​ o}^{- 1} ​ (𝒚^{o} - 𝝂_{k, o})) .

We can derive the bounds for this as follows.

For the upper bound, since $λ_{\min} (𝚫_{k, o o}) \geq {\tilde{s}}_{m}$ ,

{| 𝚫_{k, o ​ o} |}^{- \frac{1}{2}} \leq {({\tilde{s}}_{m})}^{- \frac{| 𝕆 |}{2}} .

With $\exp (- \cdot) \leq 1$ and $𝚫_{k, o o}$ is positive definite

Φ ​ (𝒚^{o}; 𝝂_{k, o}, 𝚫_{k, o ​ o}) \leq {(2 ​ π)}^{- \frac{| 𝕆 |}{2}} ​ {({\tilde{s}}_{m})}^{- \frac{| 𝕆 |}{2}} .

Summing over $k$ and using $\sum_{k = 1}^{K} π_{k} = 1$ ,

f ​ (𝒚^{o}; 𝜽) \leq {(2 ​ π ​ {\tilde{s}}_{m})}^{- \frac{| 𝕆 |}{2}} .

Taking logarithms:

\ln f ​ (𝒚^{o}; 𝜽) \leq - \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{m}) .

For the lower bound, for each $k$ ,

\ln Φ ​ (𝒚^{o}; 𝝂_{k, o}, 𝚫_{k, o ​ o}) = - \frac{| 𝕆 |}{2} ​ \ln (2 ​ π) - \frac{1}{2} ​ \ln | 𝚫_{k, o ​ o} | - \frac{1}{2} ​ {(𝒚^{o} - 𝝂_{k, o})}^{T} ​ 𝚫_{k, o ​ o}^{- 1} ​ (𝒚^{o} - 𝝂_{k, o}) .

Since $λ_{\max} (𝚫_{k, o o}) \leq {\tilde{s}}_{M}$ ,

\ln | 𝚫_{k, o ​ o} | \leq | 𝕆 | ​ \ln ({\tilde{s}}_{M}),

and $𝚫_{k, o o}^{- 1} ⪯ \frac{1}{{\tilde{s}}_{m}} I$ ,

{(𝒚^{o} - 𝝂_{k, o})}^{T} ​ 𝚫_{k, o ​ o}^{- 1} ​ (𝒚^{o} - 𝝂_{k, o}) \leq \frac{{‖ 𝒚^{o} - 𝝂_{k, o} ‖}^{2}}{{\tilde{s}}_{m}} .

Using ${‖ 𝒚^{o} - 𝝂_{k, o} ‖}^{2} \leq 2 ({‖ 𝒚^{o} ‖}^{2} + {‖ 𝝂_{k, o} ‖}^{2})$ ,

{(𝒚^{o} - 𝝂_{k, o})}^{T} ​ 𝚫_{k, o ​ o}^{- 1} ​ (𝒚^{o} - 𝝂_{k, o}) \leq \frac{2 ​ ({‖ 𝒚^{o} ‖}^{2} + {‖ 𝝂_{k, o} ‖}^{2})}{{\tilde{s}}_{m}} .

Given the construction of the mean vector $𝝂_{k}$ for each component $k$ :

ν_{k ​ j} = {\begin{matrix} μ_{k ​ j}, & if ​ j \in 𝕊, \\ {(a + 𝝁_{k} ​ Λ)}_{j}, & if ​ j \in 𝕊^{c}, \end{matrix}

and knowing that $a$ belongs to the closed ball $ℬ (ρ, 1, | 𝕊^{c} |)$ -a set of $1 \times | 𝕊^{c} |$ matrices with norm bounded by $ρ$ -we seek to derive a uniform bound for $𝝂_{k}$ . For indices $j \in 𝕊$ , $ν_{k j} = μ_{k j}$ , and since the parameter space $𝚯_{V}^{'}$ is compact, the cluster means $μ_{k j}$ are bounded by some constant $η > 0$ . For indices $j \in 𝕊^{c}$ , we have $ν_{k j} = {(a + 𝝁_{k} Λ)}_{j} = a_{j} + {(𝝁_{k} Λ)}_{j}$ . Using the elementary inequality ${(u + v)}^{2} \leq 2 (u^{2} + v^{2})$ , it follows that

{∥ ν_{k ​ j} ∥}^{2} \leq 2 ​ ({∥ a_{j} ∥}^{2} + {∥ {(𝝁_{k} ​ Λ)}_{j} ∥}^{2}) .

Given that $a$ lies in $ℬ (ρ, 1, | 𝕊^{c} |)$ , ${∥ a_{j} ∥}^{2} \leq ρ^{2}$ for each $j \in 𝕊^{c}$ . In addition, the term ${(𝝁_{k} Λ)}_{j}$ can be bounded by ${‖ 𝝁_{k} ‖}_{2} {‖ Λ_{\cdot j} ‖}_{2}$ , where $Λ_{\cdot j}$ is the $j$ -th column of $Λ$ . With $𝝁_{k}$ belonging to a compact set $ℬ (η, | 𝕊 |)$ , ${∥ 𝝁_{k} ∥}^{2} \leq η^{2}$ , and because $Λ$ is derived from bounded parameters (including $𝜷$ with norm being bounded by $ρ^{2}$ ), each column $Λ_{\cdot j}$ is bounded in norm by $ρ^{2}$ . Consequently, $| {(𝝁_{k} Λ)}_{j} | \leq ρ^{2} + η^{2} ρ^{2} = ρ^{2} (1 + η^{2})$ for each $j \in 𝕊^{c}$ . Combining these results, for any $j \in 𝕊^{c}$ ,

{∥ ν_{k ​ j} ∥}^{2} \leq 2 ​ ρ^{2} ​ (1 + η^{2}) .

Thus, all entries of $𝝂_{k}$ , irrespective of whether they correspond to indices in $𝕊$ or $𝕊^{c}$ , are bounded by a constant that depends on $η$ , $ρ$ . Hence,

	${‖ 𝝂_{k} ‖}^{2}$	$= \sum_{j} {‖ 𝝂_{k, j} ‖}^{2}$
		$\leq \sum_{j} 2 ρ^{2} (1 + η^{2})$
		$= 2 D ρ^{2} (1 + η^{2})$

uniformly for all $k$ .

Consider again the Gaussian density for the observed variables

\ln Φ ​ (𝒚^{o}; 𝝂_{k, o}, 𝚫_{k, o ​ o}) = - \frac{| 𝕆 |}{2} ​ \ln (2 ​ π) - \frac{1}{2} ​ \ln | 𝚫_{k, o ​ o} | - \frac{1}{2} ​ {(𝒚^{o} - 𝝂_{k, o})}^{T} ​ 𝚫_{k, o ​ o}^{- 1} ​ (𝒚^{o} - 𝝂_{k, o}) .

To control the quadratic term in the exponent, we note that

{‖ 𝒚^{o} - 𝝂_{k, o} ‖}^{2} \leq 2 ​ ({‖ 𝒚^{o} ‖}^{2} + {‖ 𝝂_{k, o} ‖}^{2}) .

Given that $‖ 𝝂_{k} ‖ = \sum_{j \in 𝕆} {∥ ν_{k j} ∥}^{2} \leq 2 | 𝕆 | ρ^{2} (1 + η^{2})$ is uniformly bounded, it follows that

{‖ 𝒚^{o} - 𝝂_{k, o} ‖}^{2} \leq 2 ​ ({‖ 𝒚^{o} ‖}^{2} + 2 ​ | 𝕆 | ​ ρ^{2} ​ (1 + η^{2})) .

This bound ensures that the quadratic form ${(𝒚^{o} - 𝝂_{k, o})}^{T} 𝚫_{k, o o}^{- 1} (𝒚^{o} - 𝝂_{k, o})$ remains finite. The uniform boundedness of $𝝂_{k}$ thus allows us to assert the existence of an integrable envelope function $F (𝒚^{o})$ that uniformly bounds $| \ln f (𝒚^{o}; 𝜽) |$ for all $𝜽$ within the compact parameter space $𝚯_{V}^{'}$ .

Thus,

\ln Φ ​ (𝒚^{o}; 𝝂_{k, o}, 𝚫_{k, o ​ o}) \geq - \frac{| 𝕆 |}{2} ​ \ln (2 ​ π) - \frac{| 𝕆 |}{2} ​ \ln ({\tilde{s}}_{M}) - \frac{1}{2} \cdot \frac{2 ​ ({‖ 𝒚^{o} ‖}^{2} + 2 ​ | 𝕆 | ​ ρ^{2} ​ (1 + η^{2}))}{{\tilde{s}}_{m}} .

Simplifying,

\ln Φ ​ (𝒚^{o}; 𝝂_{k, o}, 𝚫_{k, o ​ o}) \geq - \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{M}) - \frac{∥ 𝒚^{o} ∥^{2} + 2 | 𝕆 | ρ^{2} (1 + η^{2}))}{{\tilde{s}}_{m}} .

Using Jensen’s inequality over the mixture:

\ln f ​ (𝒚^{o}; 𝜽) \geq \sum_{k = 1}^{K} π_{k} ​ (- \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{M}) - \frac{∥ 𝒚^{o} ∥^{2} + 2 | 𝕆 | ρ^{2} (1 + η^{2}))}{{\tilde{s}}_{m}}) .

Since $\sum_{k = 1}^{K} π_{k} = 1$ ,

\ln f ​ (𝒚^{o}; 𝜽) \geq - \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{M}) - \frac{∥ 𝒚^{o} ∥^{2} + 2 | 𝕆 | ρ^{2} (1 + η^{2}))}{{\tilde{s}}_{m}} .

Combining the refined upper and lower bounds:

- \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{M}) - \frac{∥ 𝒚^{o} ∥^{2} + 2 | 𝕆 | ρ^{2} (1 + η^{2}))}{{\tilde{s}}_{m}} \leq \ln f ​ (𝒚^{o}; 𝜽_{(𝕊, ℝ)}) \leq - \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{m}) .

As a final note, these bounds also rely on the eigenvalue constraints on $𝚺_{k}$ , $𝛀$ , and the norm constraint on $𝜷$ .

Now we prove that the envelope function $F$ , which is related to the upper and lower bounds above, is $h$ -integrable. The true density $h (𝒚)$ corresponds to a Gaussian mixture model $f (𝒚; 𝜽_{(𝕊_{0}, ℝ_{0})}^{*})$ with parameters in a compact set. The observed-data density for $𝒚^{o}$ , obtained by marginalizing over the missing components, is given by

h ​ (𝒚^{o}) = \sum_{k = 1}^{K} π_{k} ​ Φ ​ (𝒚^{o}; 𝝂_{k, o}^{*}, Δ_{k, o ​ o}^{*}) .

To verify that the envelope function $F$ is $h$ -integrable, we need to show

\int {‖ 𝒚^{o} ‖}^{2} ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} < \infty .

We proceed by examining the second moment of the observed components. First, observe that

	$\int {‖ 𝒚^{o} ‖}^{2} h (𝒚^{o}) 𝑑 𝒚^{o}$	$= \sum_{k = 1}^{K} π_{k} \int {‖ 𝒚^{o} ‖}^{2} Φ (𝒚^{o}; 𝝂_{k, o}^{}, Δ_{k, o o}^{}) 𝑑 𝒚^{o} .$
		$\leq 2 {‖ 𝝂_{k, o}^{} ‖}^{2} + 2 tr (Δ_{k, o o}^{}) .$

by using Lemma. By the compactness of the parameter space $𝚯_{𝕍}$ and since $\sum_{k = 1}^{K} π_{k} = 1$ and the bounds derived earlier, we have

\int {‖ 𝒚^{o} ‖}^{2} ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} \leq | 𝕆 | ​ ρ^{2} ​ (1 + η^{2}) + 2 ​ s_{M} ​ | 𝕊_{0, o} | < \infty

Therefore, $F$ is h-integrable, i.e.,

\int F ​ (𝒚^{o}) ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} < \infty,

since $\int {‖ 𝒚^{o} ‖}^{2} h (𝒚^{o}) 𝑑 𝒚^{o} < \infty$ .

Finally, because $| \ln (h (𝒚^{o})) | \leq F (𝒚^{o})$ , it implies

𝔼 ​ [| \ln (h ​ (𝒚^{o})) |] = \int | \ln (h ​ (𝒚^{o})) | ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} \leq \int F ​ (𝒚^{o}) ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} < \infty .

Hence, the envelope function $F$ is $h$ -integrable and $𝔼 [| \ln (h (𝒚^{o})) |] < \infty$ . Thus, we can apply the law of large numbers to obtain the consistency of the sample KL divergence. ∎

Proposition 3.

Assume that

1.

$(𝒚_{1}^{o}, \dots, 𝒚_{n}^{o})$ are i.i.d. observed vectors with unknown density $h$ .
2.

$𝚯$ is a compact metric space.
3.

$𝜽 \in 𝚯 \mapsto \ln [f (𝒚^{o}; 𝜽)]$ is continuous for every $𝒚^{o} \in ℝ^{| 𝕆 |}$ .
4.

$F$ is an envelope function of $ℱ := {\ln [f (\cdot; 𝜽)]; 𝜽 \in 𝚯}$ which is $h$ -integrable.
5.

$𝜽^{*} = \underset{𝜽 \in 𝚯}{argmax} D_{KL} [h, f (\cdot; 𝜽)]$
6.

$\hat{𝜽} = \underset{𝜽 \in 𝚯}{argmax} \sum_{n = 1}^{N} \ln f (𝒚_{n}; 𝜽)$ .

Then, as $n \to \infty$ ,

\frac{1}{N} ​ \sum_{n = 1}^{N} \ln f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{(𝕊, ℝ)}) ​ \to_{n \to \infty}^{𝑃} ​ 𝔼_{𝒚^{o}} ​ [\ln f ​ (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*})] .

Proof.

We consider the following inequality:

| 𝔼 ​ [\ln f ​ (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*})] - \frac{1}{N} ​ \sum_{n = 1}^{N} \ln f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{(𝕊, ℝ)}) |

\leq | 𝔼 ​ [\ln f ​ (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*})] - 𝔼 ​ [\ln f ​ (𝒚^{o}; {\hat{𝜽}}_{(𝕊, ℝ)})] | + sup_{𝜽 \in 𝚯_{(𝕊, ℝ)}} | 𝔼 ​ [\ln f ​ (𝒚^{o}; 𝜽)] - \frac{1}{N} ​ \sum_{n = 1}^{N} \ln f ​ (𝒚_{n}^{o}; 𝜽) | .

By the definition of $𝜽_{(𝕊, ℝ)}^{*}$ , we have

𝔼 ​ [\ln f ​ (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*})] - 𝔼 ​ [\ln f ​ (𝒚^{o}; {\hat{𝜽}}_{(𝕊, ℝ)})] \geq 0 .

Note that,

	$𝔼 [\ln f (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*})] - 𝔼 [\ln f (𝒚^{o}; {\hat{𝜽}}_{(𝕊, ℝ)})]$	$= \underset{I}{\underset{⏟}{𝔼 [\ln f (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{})] - \frac{1}{N} \sum_{n = 1}^{N} \ln f (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{})}}$
		$+ \underset{I I}{\underset{⏟}{\frac{1}{N} \sum_{n = 1}^{N} \ln f (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*}) - \frac{1}{N} \sum_{n = 1}^{N} \ln f (𝒚^{o}; {\hat{𝜽}}_{(𝕊, ℝ)})}}$
		$- \underset{I I I}{\underset{⏟}{\frac{1}{N} \sum_{n = 1}^{N} \ln f (𝒚^{o}; {\hat{𝜽}}_{(𝕊, ℝ)}) + 𝔼 [\ln f (𝒚^{o}; {\hat{𝜽}}_{(𝕊, ℝ)})}}]$

For term $I I$ , sine $\hat{𝜽}$ maximizes the empirical log-likelihood,

\frac{1}{N} \sum_{n = 1}^{N} \ln (f (X_{n} | {\hat{𝜽}}_{(𝕊, ℝ)}) \geq \frac{1}{N} \sum_{n = 1}^{N} \ln (f (X_{n} | 𝜽_{(𝕊, ℝ)}^{*}),

which implies that $I I \leq 0$ . Therefore,

𝔼 ​ [\ln f ​ (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*})] - 𝔼 ​ [\ln f ​ (𝒚^{o}; {\hat{𝜽}}_{(𝕊, ℝ)})] \leq | I | + | I ​ I ​ I | .

Moreover, since both $| I | and | I I I |$ are bounded by the uniform deviation:

	$\| I \| \leq sup_{𝜽 \in 𝚯_{(𝕊, ℝ)}} \| 𝔼 [\ln f (𝒚^{o}; 𝜽)] - \frac{1}{N} \sum_{n = 1}^{N} \ln f (𝒚_{n}^{o}; 𝜽) \|$
	and
	$\| I I \| \leq sup_{𝜽 \in 𝚯_{(𝕊, ℝ)}} \| 𝔼 [\ln f (𝒚^{o}; 𝜽)] - \frac{1}{N} \sum_{n = 1}^{N} \ln f (𝒚_{n}^{o}; 𝜽) \|$

Hence,

	$𝔼 [\ln f (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*})] - 𝔼 [\ln f (𝒚^{o}; {\hat{𝜽}}_{(𝕊, ℝ)})]$	$\leq \| I \| + \| I I I \|$
		$\leq 2 sup_{𝜽 \in 𝚯_{(𝕊, ℝ)}} \| 𝔼 [\ln f (𝒚^{o}; 𝜽)] - \frac{1}{N} \sum_{n = 1}^{N} \ln f (𝒚_{n}^{o}; 𝜽) \| .$

Hence, the left-hand side is bounded by three times the uniform deviation:

| 𝔼 ​ [\ln f ​ (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*})] - \frac{1}{N} ​ \sum_{n = 1}^{N} \ln f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{(𝕊, ℝ)}) | \leq 3 ​ sup_{𝜽 \in 𝚯_{(𝕊, ℝ)}} | 𝔼 ​ [\ln f ​ (𝒚^{o}; 𝜽)] - \frac{1}{N} ​ \sum_{n = 1}^{N} \ln f ​ (𝒚_{n}^{o}; 𝜽) | .

By the argument in the Proposition 2-namely, using the compactness of $𝚯_{(𝕊, ℝ)}$ , the continuity of $𝜽 \mapsto \ln f (\cdot; 𝜽)$ , and the existence of an $h$ -integrable envelope $F$ -the class $ℱ_{(𝕊, ℝ)}$ is P-Glivenko-Cantelli by applying Example 19.8 in van der Vaart (1998) to conclude on the finiteness of bracketing numbers of $ℱ$ under the assumptions. In particular,

sup_{𝜽 \in 𝚯_{(𝕊, ℝ)}} | 𝔼 ​ [\ln f ​ (𝒚^{o}; 𝜽)] - \frac{1}{N} ​ \sum_{n = 1}^{N} \ln f ​ (𝒚_{n}^{o}; 𝜽) | ​ \to_{n \to \infty}^{𝑃} ​ 0 .

Therefore,

\frac{1}{N} ​ \sum_{n = 1}^{N} \ln f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{(𝕊, ℝ)}) ​ \to_{n \to \infty}^{𝑃} ​ 𝔼 ​ [\ln f ​ (𝒚^{o}; 𝜽_{(𝕊, ℝ)}^{*})] .

which concludes the proof of Proposition 2. ∎

Now, we prove the consistency of the sample KL in the $𝕊 ℝ 𝕌 𝕎$ model:

Proposition 4.

Under Assumption 4 and 5, for all $𝕍$ ,

\frac{1}{N} ​ \sum_{n = 1}^{N} \ln (\frac{h ​ (𝒚_{n}^{o})}{f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{𝕍})}) ​ \to_{n \to \infty}^{𝑃} ​ D_{KL} ​ [h, f ​ (\cdot; 𝜽_{𝕍}^{*})] .

Proof.

We carry the proof in the same manner as in the $𝕊 ℝ$ model. Let $𝕍 = (𝕊, ℝ, 𝕌, 𝕎)$ and $𝕆 \subset {1, \dots, D}$ denotes the observed variables. We still want to apply the Proposition 3 with the new family

ℱ_{(𝕍)} ≔ {\ln [f (\cdot | 𝜽)]; 𝜽 \in 𝚯_{𝕍}^{'}}

similarly to the proof of Proposition 2 to achieve

\frac{1}{N} ​ \sum_{n = 1}^{N} \ln (f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{𝕍})) ​ \to_{n \to \infty}^{𝑃} ​ 𝔼_{𝒚^{o}} ​ [\ln f ​ (𝒚^{o}; 𝜽_{𝕍}^{*})] .

To do so, we must initially verify that the class of functions

ℱ_{(𝕍)} = {𝒚^{o} \mapsto \ln f ​ (𝒚^{o}; 𝜽) : 𝜽 \in 𝚯_{𝕍}^{'}}

satisfies the conditions for uniform convergence under Assumption 5. In particular, by Assumption 5, the parameter space $𝚯_{𝕍}^{'}$ is compact, and for every $𝒚^{o} \in ℝ^{| 𝕆 |}$ the mapping

𝜽 \mapsto \ln f ​ (𝒚^{o}; 𝜽)

is continuous. Next, we verify that there is an $h$ -integrable envelope function $F \in ℱ_{(𝕍)}$ . Recalling that for $𝒚^{o} \in ℝ^{| 𝕆 |}$ and a given component $k$ , the current Gaussian component density is

Φ ​ (𝒚^{o}; {\tilde{𝝂}}_{k, o}, {\tilde{𝚫}}_{k, o ​ o}) = {(2 ​ π)}^{- \frac{| 𝕆 |}{2}} ​ {| {\tilde{𝚫}}_{k, o ​ o} |}^{- \frac{1}{2}} ​ \exp (- \frac{1}{2} ​ {(𝒚^{o} - {\tilde{𝝂}}_{k, o})}^{T} ​ {\tilde{𝚫}}_{k, o ​ o}^{- 1} ​ (𝒚^{o} - {\tilde{𝝂}}_{k, o})) .

We will bound the density function as usual. From Proposition 2, we know that there exists ${\tilde{s}}_{m} > 0$ and ${\tilde{s}}_{M}^{'} = \max {s_{M} (1 + ρ^{2}), s_{M} ρ, s_{M}} = s_{M} (1 + ρ^{2})$ such that

{\tilde{s}}_{m} ​ I ⪯ 𝚫_{k} ⪯ {\tilde{s}}_{M} ​ I .

Thus,

{\tilde{s}}_{m} ​ I_{| 𝕆 |} ⪯ 𝚫_{k, o ​ o} ⪯ {\tilde{s}}_{M} ​ I_{| 𝕆 |},

and

{(𝚫_{k, o ​ o})}^{- 1} ⪯ \frac{1}{{\tilde{s}}_{m}} ​ I_{| 𝕆 |} .

Given that $𝚪 \in 𝒟_{| 𝕎 |}$ and is independent of any relevant variables, we have that the principal sub-matrix $𝚪_{o o}$ is bounded by $s_{M}$

λ_{\max} ​ (𝚪_{o ​ o}) \leq s_{M} .

Since ${\tilde{𝚫}}_{k, o o}$ is block-diagonal, its eigenvalues are the union of the eigenvalues of $𝚫_{k, o o}$ and $𝚪_{o o}$ . Hence,

λ_{\max} ​ ({\tilde{𝚫}}_{k, o ​ o}) \leq \max {s_{M} ​ (1 + ρ^{2}), s_{M}} = s_{M} ​ (1 + ρ^{2}) .

We define the upper bound by

{\tilde{s}}_{M} := s_{M} ​ (1 + ρ^{2}) .

The lower bound of the structured block, together with the lower bound on the independent block, implies that the covariance ${\tilde{𝚫}}_{k, o o}$ satisfies

{\tilde{s}}_{m} ​ I ⪯ {\tilde{𝚫}}_{k, o ​ o} ⪯ {\tilde{s}}_{M} ​ I .

Because ${\tilde{𝚫}}_{k, o o}^{- 1} ⪯ \frac{1}{{\tilde{s}}_{m}} I$ , it follows that

{(𝒚^{o} - {\tilde{𝝂}}_{k, o})}^{T} ​ {\tilde{𝚫}}_{k, o ​ o}^{- 1} ​ (𝒚^{o} - {\tilde{𝝂}}_{k, o}) \leq \frac{{‖ 𝒚^{o} - {\tilde{𝝂}}_{k, o} ‖}^{2}}{{\tilde{s}}_{m}} .

Using the elementary inequality

{‖ 𝒚^{o} - {\tilde{𝝂}}_{k, o} ‖}^{2} \leq 2 ​ ({‖ 𝒚^{o} ‖}^{2} + {‖ {\tilde{𝝂}}_{k, o} ‖}^{2}),

we obtain

{(𝒚^{o} - {\tilde{𝝂}}_{k, o})}^{T} ​ {\tilde{𝚫}}_{k, o ​ o}^{- 1} ​ (𝒚^{o} - {\tilde{𝝂}}_{k, o}) \leq \frac{2 ​ ({‖ 𝒚^{o} ‖}^{2} + {‖ {\tilde{𝝂}}_{k, o} ‖}^{2})}{{\tilde{s}}_{m}} .

Now we process the bound for ${\tilde{𝝂}}_{k, o}$ . Recalling that if we denote $D_{𝕊 𝕌}$ denote the number of coordinates in the structured part (i.e. $𝕊 \cup 𝕌$ ); then

{‖ 𝝂_{k, o} ‖}^{2} = \sum_{j \in o} {∥ ν_{k ​ j} ∥}^{2} \leq D_{𝕊 ​ 𝕌, o} ​ ρ^{2} ​ (1 + η^{2})

is uniformly bounded and ${∥ γ_{o} ∥}^{2} \leq η^{2}$ since $γ$ belongs to the compact set $ℬ (η, | 𝕎 |)$ , it follows that

{‖ {\tilde{𝝂}}_{k, o} ‖}^{2} \leq D_{𝕊 ​ 𝕌, o} ​ ρ^{2} ​ {(1 + η)}^{2} + η^{2},

We deduce that

\ln Φ ​ (𝒚^{o}; {\tilde{𝝂}}_{k, o}, {\tilde{𝚫}}_{k, o ​ o}) \geq - \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{M}) - \frac{{‖ 𝒚^{o} ‖}^{2} + D_{𝕊 ​ 𝕌, o} ​ ρ^{2} ​ (1 + η^{2}) + η^{2}}{{\tilde{s}}_{m}} .

Using Jensen’s inequality over the mixture and $\sum_{k = 1}^{K} π_{k} = 1$ , we have

	$\ln f (𝒚^{o}; 𝜽)$	$\geq \sum_{k = 1}^{K} π_{k} (- \frac{\| 𝕆 \|}{2} \ln (2 π {\tilde{s}}_{M}) - \frac{{‖ 𝒚^{o} ‖}^{2} + D_{𝕊 𝕌, o} ρ^{2} (1 + η^{2}) + η^{2}}{{\tilde{s}}_{m}}) .$
		$\geq - \frac{\| 𝕆 \|}{2} \ln (2 π {\tilde{s}}_{M}) - \frac{{‖ 𝒚^{o} ‖}^{2} + D_{𝕊 𝕌, o} ρ^{2} (1 + η^{2}) + η^{2}}{{\tilde{s}}_{m}}$

Therefore, each family’s member in $ℱ_{(𝕍)}$ is bounded by

- \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{M}) - \frac{{‖ 𝒚^{o} ‖}^{2} + D_{𝕊 ​ 𝕌, o} ​ ρ^{2} ​ (1 + η^{2}) + η^{2}}{{\tilde{s}}_{m}} \leq \ln f ​ (𝒚^{o}; 𝜽_{(𝕍)}) \leq - \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{m}) .

Now we prove that $F$ is $h$ -integrable. The true observed-data density under the $𝕊 ℝ 𝕌 𝕎$ model is given by

h ​ (𝒚^{o}) = \sum_{k = 1}^{K} π_{k} ​ Φ ​ (𝒚^{o}; {\tilde{𝝂}}_{k, o}^{*}, {\tilde{Δ}}_{k, o ​ o}^{*}),

where the “ $*$ ” indicates that the parameters are the true ones and the density is that of a Gaussian mixture with parameters in a compact set. In our derivation we have shown that, for any $𝜽$ in the family $ℱ_{(𝕍)}$ , the log-density satisfies

- \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{M}) - \frac{{‖ 𝒚^{o} ‖}^{2} + D_{𝕊 ​ 𝕌, o} ​ ρ^{2} ​ (1 + η^{2}) + η^{2}}{{\tilde{s}}_{m}} \leq \ln f ​ (𝒚^{o}; 𝜽) \leq - \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{m}) .

In other words, every function $\ln f (𝒚^{o}; 𝜽)$ is bounded in absolute value by

F ​ (𝒚^{o}) = \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{M}) + \frac{{‖ 𝒚^{o} ‖}^{2} + D_{𝕊 ​ 𝕌, o} ​ ρ^{2} ​ (1 + η^{2}) + η^{2}}{{\tilde{s}}_{m}} .

Hence, for all $𝜽$ in the compact parameter space,

| \ln f ​ (𝒚^{o}; 𝜽) | \leq F ​ (𝒚^{o}) .

To verify that $F$ is $h$ -integrable, we must show that

\int F ​ (𝒚^{o}) ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} < \infty .

Since the envelope function $F$ is of the form

F ​ (𝒚^{o}) = C_{0} + \frac{1}{{\tilde{s}}_{m}} ​ {‖ 𝒚^{o} ‖}^{2},

with

C_{0} = \frac{| 𝕆 |}{2} ​ \ln (2 ​ π ​ {\tilde{s}}_{M}) + \frac{D_{𝕊 ​ 𝕌, o} ​ ρ^{2} ​ (1 + η^{2}) + η^{2}}{{\tilde{s}}_{m}},

it is clear that

\int F ​ (𝒚^{o}) ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} \leq C_{0} + \frac{1}{{\tilde{s}}_{m}} ​ \int {‖ 𝒚^{o} ‖}^{2} ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} .

Because $h (𝒚^{o})$ is a Gaussian mixture with parameters in a compact set, standard properties of Gaussian mixtures guarantee that the second moment is finite, that is,

\int {‖ 𝒚^{o} ‖}^{2} ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} < \infty .

Thus,

\int F ​ (𝒚^{o}) ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} < \infty .

Finally, since for every $𝒚^{o}$ we have

| \ln h ​ (𝒚^{o}) | \leq F ​ (𝒚^{o}),

It follows that

𝔼 ​ [| \ln h ​ (𝒚^{o}) |] = \int | \ln h ​ (𝒚^{o}) | ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} \leq \int F ​ (𝒚^{o}) ​ h ​ (𝒚^{o}) ​ 𝑑 𝒚^{o} < \infty .

Therefore, the envelope function $F$ is $h$ -integrable and $𝔼 [| \ln (h (𝒚^{o})) |] < \infty$ , and consequently, we can apply the law of large numbers and uniform convergence to conclude the proof. ∎

Proof of Theorem 2.

Define the BIC score of a variable partition $𝕍 = (𝕊, ℝ, 𝕌, 𝕎)$ by

BIC ​ (𝕍) = 2 ​ ℓ_{N} ​ ({\hat{𝜽}}_{𝕍}) - Ξ_{(𝕍)} ​ \log N,

where $Ξ_{(𝕍)}$ is the number of free parameters. Let $𝕍_{0}$ be the true partition and set $Δ BIC (𝕍) = BIC (𝕍_{0}) - BIC (𝕍)$ .

Let $𝕍_{1} = {𝕍 \neq 𝕍_{0} : D_{KL} [h, f (\cdot; 𝜽_{𝕍}^{⋆})] > 0}$ . The Identifiability Theorem 1 implies every $𝕍 \neq 𝕍_{0}$ belongs to $𝕍_{1}$ . For $𝕍 \in 𝕍_{1}$ ,

Δ ​ BIC ​ (𝕍) = 2 ​ N ​ [\frac{1}{N} ​ \sum_{n = 1}^{N} \ln (\frac{f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{𝕍_{0}})}{h ​ (𝒚_{n}^{o})}) - \frac{1}{N} ​ \sum_{n = 1}^{N} \ln (\frac{f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{𝕍})}{h ​ (𝒚_{n}^{o})})] + [Ξ_{(𝕍)} - Ξ_{(𝕍_{0})}] ​ \log N

(19)

To prove the theorem, we will prove that:

\forall 𝕍 \in 𝕍_{1}, ℙ ​ [Δ ​ BIC ​ (𝕍) < 0] ​ \underset{N \to \infty}{\to} ​ 0

Denoting $𝕄_{N} (𝕍) = \frac{1}{N} \sum_{n = 1}^{N} \ln (\frac{f (𝒚_{n}^{o}; {\hat{𝜽}}_{𝕍})}{h (𝒚_{n}^{o})}), M (𝕍) = - D_{KL} [h, f (\cdot; 𝜽_{𝕍}^{*})]$ , from Equation 19, we get:

		$ℙ (Δ BIC (𝕍) < 0)$
	$=$	$ℙ (2 N (𝕄_{N} (𝕍_{0}) - 𝕄_{N} (𝕍)) + [Ξ_{(𝕍)} - Ξ_{(𝕍_{0})}] \log N < 0)$
	$=$	$ℙ (𝕄_{N} (𝕍_{0}) - M (𝕍_{0}) + M (𝕍_{0}) - M (𝕍) + M (𝕍) - 𝕄_{N} (𝕍) + \frac{[Ξ_{(𝕍)} - Ξ_{(𝕍_{0})}] \log N}{2 N} < 0)$
	$\leq$	$ℙ (𝕄_{N} (𝕍_{0}) - M (𝕍_{0}) > ϵ) + ℙ (M (𝕍) - 𝕄_{N} (𝕍) > ϵ) + ℙ (M (𝕍_{0}) - M (𝕍) + \frac{[Ξ_{(𝕍)} - Ξ_{(𝕍_{0})}] \log N}{2 N} < 2 ϵ)$

From Proposition 4, we know that

\frac{1}{N} ​ \sum_{n = 1}^{N} \ln (\frac{h ​ (𝒚_{n}^{o})}{f ​ (𝒚_{n}^{o}; {\hat{𝜽}}_{𝕍})}) ​ \to_{n \to \infty}^{𝑃} ​ D_{KL} ​ [h, f ​ (\cdot; 𝜽_{𝕍}^{*})] .

This leads to $𝕄_{N} (𝕍) \to_{n \to \infty}^{𝑃} M (𝕍)$ . Similar to the proof for the $𝕊 ℝ 𝕌 𝕎$ model in Maugis [38], we prove the theorem. ∎

D.3 Proof of Theorem 3

Throughout this proof, we analyze the estimator defined by minimizing the negative log-likelihood penalized. Accordingly, although we denote by

ℓ ​ (𝒚; 𝜶) = \ln f ​ (𝒚; 𝜶)

the usual log-likelihood of a single observation, we consistently work with its negative $- ℓ$ as the optimization objective. To streamline notation, we therefore use the term “score” to mean the gradient of the negative log-likelihood:

S_{j} ​ (𝒚_{n}; 𝜶) := - \frac{\partial ℓ ​ (𝒚_{n}; 𝜶)}{\partial α_{j}} .

This convention differs by a minus sign from the standard statistical score, but it ensures that all gradients, Hessians, and Fisher information matrices below are taken with respect to the minimized objective $- ℓ_{N}$ .

We start with presenting some useful lemmas before proving our main theorem. These lemmas establish fundamental properties of the penalized GMM estimators and the variable ranking procedure.

Lemma 1 (Score Function Components).

Let $f_{clust} (𝐲_{n}; 𝛂) = \sum_{m = 1}^{K} π_{m} Φ (𝐲_{n}; 𝛍_{m}, 𝚺_{m})$ be the p.d.f of a $K$ -component GMM for an observation $𝐲_{n}$ , where $𝛂 = (𝛑, {𝛍_{k}}_{k = 1}^{K}, {𝚺_{k}}_{k = 1}^{K})$ represents the GMM parameters. Let $𝚿_{k} = 𝚺_{k}^{- 1}$ be the precision matrix for component $k$ . The log-likelihood for observation $𝐲_{n}$ is $ℓ (𝐲_{n}; 𝛂) = \ln f_{clust} (𝐲_{n}; 𝛂)$ . The responsibility for component $k$ given observation $𝐲_{n}$ and parameters $𝛂$ is $t_{k} (𝐲_{n}; 𝛂) = \frac{π_{k} Φ (𝐲_{n}; 𝛍_{k}, 𝚺_{k})}{f_{clust} (𝐲_{n}; 𝛂)}$ . The score components, $S_{j} (𝐲_{n}; 𝛂^{*})$ , where $α_{j}$ is a parameter in $𝛂$ and $𝛂^{*}$ are the true parameter values, are given as follows:

1.

Mixing proportions $π_{k}$ : When parameterizing $π_{K} = 1 - \sum_{j = 1}^{K - 1} π_{j}$ , the score component for $π_{k}$ , $k \in {1, \dots, K - 1}$ , is:

$S_{π_{k}} (𝒚_{n}; 𝜶^{*}) = \frac{t_{k} (𝒚_{n}; 𝜶^{*})}{π_{K}^{*}} - \frac{t_{k} (𝒚_{n}; 𝜶^{*})}{π_{k}^{*}}$
2.

Mean parameters ${(𝝁_{k})}_{d}$ : Let ${(𝝁_{k})}_{d}$ be the $d$ -th component of the mean vector $𝝁_{k}$ . The score component is:

$S_{{(𝝁_{k})}_{d}} (𝒚_{n}; 𝜶^{*}) = - t_{k} (𝒚_{n}; 𝜶^{*}) {(𝚿_{k}^{*} (𝒚_{n} - 𝝁_{k}^{*}))}_{d}$
3.

Precision matrix parameters ${(Ψ_{k})}_{r s}$ : Let ${(𝚿_{k})}_{r s}$ be the element $(r, s)$ of the symmetric precision matrix $𝚿_{k}$ . The score component is:

$S_{{(𝚿_{k})}_{r s}} (𝒚_{n}; 𝜶^{*}) = - t_{k} (𝒚_{n}; 𝜶^{*}) \frac{(2 - δ_{r s})}{2} ({(𝚺_{k}^{*})}_{r s} - {(𝒚_{n} - 𝝁_{k}^{*})}_{r} {(𝒚_{n} - 𝝁_{k}^{*})}_{s})$

where $δ_{r s}$ is the Kronecker delta.

Proof of Lemma 1.

Let $Φ_{m} (𝒚_{n}; 𝜶) = Φ (𝒚_{n}; 𝝁_{m}, 𝚺_{m})$ . The log-likelihood for a single observation $𝒚_{n}$ is $ℓ (𝒚_{n}; 𝜶) = \ln (\sum_{m = 1}^{K} π_{m} Φ_{m} (𝒚_{n}; 𝜶))$ . The score component for a generic parameter $α_{j}$ is $S_{j} (𝒚_{n}; 𝜶^{*}) = - {\frac{\partial ℓ (𝒚_{n}; 𝜶)}{\partial α_{j}} |}_{𝜶 = 𝜶^{*}}$ .

Score components for mixing proportions $π_{k}$ . We parameterize $π_{K} = 1 - \sum_{j = 1}^{K - 1} π_{j}$ . For $k \in {1, \dots, K - 1}$ :

	$\frac{\partial ℓ (𝒚_{n}; 𝜶)}{\partial π_{k}}$	$= \frac{1}{f_{clust} (𝒚_{n}; 𝜶)} \frac{\partial}{\partial π_{k}} (\sum_{m = 1}^{K - 1} π_{m} Φ_{m} (𝒚_{n}; 𝜶) + (1 - \sum_{j = 1}^{K - 1} π_{j}) Φ_{K} (𝒚_{n}; 𝜶))$
		$= \frac{Φ_{k} (𝒚_{n}; 𝜶) - Φ_{K} (𝒚_{n}; 𝜶)}{f_{clust} (𝒚_{n}; 𝜶)}$

Since $t_{m} (𝒚_{n}; 𝜶) = \frac{π_{m} Φ_{m} (𝒚_{n}; 𝜶)}{f_{clust} (𝒚_{n}; 𝜶)}$ , we have $\frac{Φ_{m} (𝒚_{n}; 𝜶)}{f_{clust} (𝒚_{n}; 𝜶)} = \frac{t_{m} (𝒚_{n}; 𝜶)}{π_{m}}$ . Thus,

\frac{\partial ℓ ​ (𝒚_{n}; 𝜶)}{\partial π_{k}} = \frac{t_{k} ​ (𝒚_{n}; 𝜶)}{π_{k}} - \frac{t_{k} ​ (𝒚_{n}; 𝜶)}{π_{K}}

The score component at $𝜶^{*}$ is:

S_{π_{k}} ​ (𝒚_{n}; 𝜶^{*}) = - (\frac{t_{k} ​ (𝒚_{n}; 𝜶^{*})}{π_{k}^{*}} - \frac{t_{k} ​ (𝒚_{n}; 𝜶^{*})}{π_{K}^{*}}) = \frac{t_{k} ​ (𝒚_{n}; 𝜶^{*})}{π_{K}^{*}} - \frac{t_{k} ​ (𝒚_{n}; 𝜶^{*})}{π_{k}^{*}} .

Score components for mean parameters ${(μ_{k})}_{d}$ . Let ${(𝝁_{k})}_{d}$ be the $d$ -th component of $𝝁_{k}$ .

	$\frac{\partial ℓ (𝒚_{n}; 𝜶)}{\partial {(𝝁_{k})}_{d}}$	$= \frac{1}{f_{clust} (𝒚_{n}; 𝜶)} \frac{\partial}{\partial {(𝝁_{k})}_{d}} (\sum_{m = 1}^{K} π_{m} Φ_{m} (𝒚_{n}; 𝜶))$
		$= \frac{π_{k}}{f_{clust} (𝒚_{n}; 𝜶)} \frac{\partial Φ_{k} (𝒚_{n}; 𝜶)}{\partial {(𝝁_{k})}_{d}}$

Since $\frac{\partial Φ_{k}}{\partial {(𝝁_{k})}_{d}} = Φ_{k} \frac{\partial \ln Φ_{k}}{\partial {(𝝁_{k})}_{d}}$ , and for $Φ_{k}$ , $\ln Φ_{k} (𝒚_{n}; 𝜶) = C_{k} - \frac{1}{2} {(𝒚_{n} - 𝝁_{k})}^{⊤} 𝚿_{k} (𝒚_{n} - 𝝁_{k})$ , we have:

\frac{\partial \ln Φ_{k} ​ (𝒚_{n}; 𝜶)}{\partial {(𝝁_{k})}_{d}} = {(𝚿_{k} ​ (𝒚_{n} - 𝝁_{k}))}_{d}

Substituting this back:

\frac{\partial ℓ ​ (𝒚_{n}; 𝜶)}{\partial {(𝝁_{k})}_{d}} = \frac{π_{k} ​ Φ_{k} ​ (𝒚_{n}; 𝜶)}{f_{clust} ​ (𝒚_{n}; 𝜶)} ​ {(𝚿_{k} ​ (𝒚_{n} - 𝝁_{k}))}_{d} = t_{k} ​ (𝒚_{n}; 𝜶) ​ {(𝚿_{k} ​ (𝒚_{n} - 𝝁_{k}))}_{d}

The score component at $𝜶^{*}$ is:

S_{{(𝝁_{k})}_{d}} ​ (𝒚_{n}; 𝜶^{*}) = - t_{k} ​ (𝒚_{n}; 𝜶^{*}) ​ {(𝚿_{k}^{*} ​ (𝒚_{n} - 𝝁_{k}^{*}))}_{d} .

Score components for precision matrix parameters ${(𝚿_{k})}_{r s}$ . Let ${(𝚿_{k})}_{r s}$ be an element of the symmetric precision matrix $𝚿_{k}$ .

\frac{\partial ℓ ​ (𝒚_{n}; 𝜶)}{\partial {(𝚿_{k})}_{r ​ s}} = \frac{π_{k}}{f_{clust} ​ (𝒚_{n}; 𝜶)} ​ \frac{\partial Φ_{k} ​ (𝒚_{n}; 𝜶)}{\partial {(𝚿_{k})}_{r ​ s}} = t_{k} ​ (𝒚_{n}; 𝜶) ​ \frac{1}{π_{k}} ​ π_{k} ​ \frac{\partial \ln Φ_{k} ​ (𝒚_{n}; 𝜶)}{\partial {(𝚿_{k})}_{r ​ s}}

The log-density of a single Gaussian component is $\ln Φ_{k} (𝒚_{n}; 𝜶) = C_{k}^{'} + \frac{1}{2} \ln det (𝚿_{k}) - \frac{1}{2} {(𝒚_{n} - 𝝁_{k})}^{⊤} 𝚿_{k} (𝒚_{n} - 𝝁_{k})$ . For a symmetric matrix $𝚿_{k}$ , the derivative with respect to an element ${(𝚿_{k})}_{r s}$ (using the “symmetric” derivative convention where $\frac{\partial}{\partial X_{r s}}$ means varying $X_{r s}$ and $X_{s r}$ simultaneously if $r \neq s$ ) is:

	$\frac{\partial \ln det (𝚿_{k})}{\partial {(𝚿_{k})}_{r s}}$	$= (2 - δ_{r s}) {(𝚿_{k}^{- 1})}_{r s} = (2 - δ_{r s}) {(𝚺_{k})}_{r s}$
	$\frac{\partial {(𝒚_{n} - 𝝁_{k})}^{⊤} 𝚿_{k} (𝒚_{n} - 𝝁_{k})}{\partial {(𝚿_{k})}_{r s}}$	$= (2 - δ_{r s}) {(𝒚_{n} - 𝝁_{k})}_{r} {(𝒚_{n} - 𝝁_{k})}_{s}$

Therefore,

\frac{\partial \ln Φ_{k} ​ (𝒚_{n}; 𝜶)}{\partial {(𝚿_{k})}_{r ​ s}} = \frac{1}{2} ​ (2 - δ_{r ​ s}) ​ {(𝚺_{k})}_{r ​ s} - \frac{1}{2} ​ (2 - δ_{r ​ s}) ​ {(𝒚_{n} - 𝝁_{k})}_{r} ​ {(𝒚_{n} - 𝝁_{k})}_{s}

So,

\frac{\partial ℓ ​ (𝒚_{n}; 𝜶)}{\partial {(𝚿_{k})}_{r ​ s}} = t_{k} ​ (𝒚_{n}; 𝜶) ​ \frac{(2 - δ_{r ​ s})}{2} ​ ({(𝚺_{k})}_{r ​ s} - {(𝒚_{n} - 𝝁_{k})}_{r} ​ {(𝒚_{n} - 𝝁_{k})}_{s})

The score component at $𝜶^{*}$ is:

S_{{(𝚿_{k})}_{r ​ s}} ​ (𝒚_{n}; 𝜶^{*}) = - t_{k} ​ (𝒚_{n}; 𝜶^{*}) ​ \frac{(2 - δ_{r ​ s})}{2} ​ ({(𝚺_{k}^{*})}_{r ​ s} - {(𝒚_{n} - 𝝁_{k}^{*})}_{r} ​ {(𝒚_{n} - 𝝁_{k}^{*})}_{s}) .

∎

For the two lemmas below, we will use the following assumptions:

Assumption 7 (Identifiability & Smoothness).

The GMM density $f_{clust} (𝐲; 𝛂)$ is identifiable. $ℓ_{1} (𝐲; 𝛂)$ . is three times continuously differentiable w.r.t. $𝛂$ in an open ball $𝔹 (𝛂^{*}, r_{0})$ around $𝛂^{*}$ .

Assumption 8 (Compact Parameter Space & True Parameter Properties).

$𝜶^{*}$ is an interior point of a compact set $𝚯_{𝕍}^{'} \subset 𝔹 (𝛂^{*}, r_{0})$ . This implies:

•

Mixing proportions: $π_{k}^{*} \geq π_{\min} > 0$ for all $k = 1, \dots, K$ , for some constant $π_{\min} \in (0, 1 / K]$ .
•

Means: ${‖ 𝝁_{k}^{*} ‖}_{2} \leq η < \infty$ for all $k$ .
•

Covariance and Precision Matrices: The covariance matrices $𝚺_{k}^{*}$ have eigenvalues $λ (𝚺_{k}^{*})$ such that $0 < s_{m} \leq λ (𝚺_{k}^{*}) \leq s_{M} < \infty$ . Consequently, for the precision matrices $𝚿_{k}^{*} = {(𝚺_{k}^{*})}^{- 1}$ , their eigenvalues $λ (𝚿_{k}^{*})$ satisfy $0 < 1 / s_{M} \leq λ (𝚿_{k}^{*}) \leq 1 / s_{m} < \infty$ . We define $σ_{\min}^{2} = s_{m}$ , $σ_{\max}^{2} = s_{M}$ . And for precision matrices, $θ_{\min} = 1 / s_{M}$ , $θ_{\max} = 1 / s_{m}$ .

All $𝛂 \in 𝚯_{𝕍}^{'}$ satisfy these bounds. Let $L_{3}$ be an upper bound on the norm of the third derivative tensor of $ℓ_{1} (𝐲; 𝛂)$ for $𝛂 \in 𝚯_{𝕍}^{'}$ , such that $𝔼 [L_{3} (𝐲)] < \infty$ .

Assumption 9 (Data Distribution).

Observations $𝐲_{1}, \dots, 𝐲_{N}$ are i.i.d. from $f_{clust} (𝐲; 𝛂^{*})$ . For each $𝐲_{n}$ , there exists a latent class variable $z_{n} \in {1, \dots, K}$ with $P (z_{n} = k) = π_{k}^{*}$ , such that $𝐲_{n} | z_{n} = k \sim 𝒩 (𝛍_{k}^{*}, 𝚺_{k}^{*})$ .

Assumption 10 (Bounded Posteriors).

For $𝛂 \in 𝚯_{𝕍}^{'}$ , the posterior probabilities $t_{k} (𝐲; 𝛂) = \frac{π_{k} ϕ (𝐲 | 𝛍_{k}, 𝚺_{k})}{\sum_{j = 1}^{K} π_{j} ϕ (𝐲 | 𝛍_{j}, 𝚺_{j})}$ satisfy $0 < t_{k} (𝐲; 𝛂) \leq 1$ .

The empirical Hessian is $𝑯_{N} (𝜶) = \nabla^{2} ℓ_{N} (𝜶)$ . The Fisher Information Matrix is $𝑰 (𝜶^{*}) = 𝔼_{𝜶^{*}} [\nabla^{2} ℓ (𝒚; 𝜶^{*})]$ . Moreover, we have the following definition of the true support of penalized parameters:

Definition 3 (True Support of Penalized Parameters $𝕊_{0}$ ).

Let $𝛂^{*} = (𝛑^{*}, {𝛍_{k}^{*}}_{k = 1}^{K}, {𝚿_{k}^{*}}_{k = 1}^{K})$ be the true GMM parameter vector. Consider the penalty

P ​ (𝜶) = λ ​ \sum_{k = 1}^{K} {‖ 𝝁_{k} ‖}_{1} + ρ ​ \sum_{k = 1}^{K} {‖ 𝚿_{k} ‖}_{1} .

We define the true support set $𝕊_{0}$ as the collection of indices corresponding to nonzero parameters in $𝛂^{*}$ that are subject to penalization:

𝕊_{0} = 𝕊_{𝝁}^{*} \cup 𝕊_{𝚿}^{*}, s_{0} = | 𝕊_{0} | = s_{𝝁}^{*} + s_{𝚿}^{*} .

Here:

1.

Support of means

$𝕊_{𝝁}^{*} = {(k, d) : 1 \leq k \leq K, 1 \leq d \leq p, {(𝝁_{k}^{*})}_{d} \neq 0}, s_{𝝁}^{*} = | 𝕊_{𝝁}^{*} | .$
2.

Support of precision matrices

$𝕊_{𝚿}^{*} = {(k, r, s) : 1 \leq k \leq K, 1 \leq r, s \leq p, {(𝚿_{k}^{*})}_{r s} \neq 0}, s_{𝚿}^{*} = | 𝕊_{𝚿}^{*} | .$

Assumption 11 (Restricted Eigenvalue Condition).

For any parameter increment vector $𝚫$ indexed compatibly with $𝛂$ , we $𝚫_{𝕊_{0}}$ for its restriction to indices in $𝕊_{0}$ and $𝚫_{{(𝕊_{0})}^{c}}$ for its complement. For $c_{0} \geq 1$ , define the cone

𝒞 ​ (c_{0}, 𝕊_{0}) = {𝚫 : {∥ 𝚫_{{(𝕊_{0})}^{c}} ∥}_{1} \leq c_{0} ​ {∥ 𝚫_{𝕊_{0}} ∥}_{1}} .

Assume the FIM $𝐈 (𝛂^{*})$ is positive definite, then there exists $κ_{I} > 0$ such that, for all $𝚫 \in 𝒞 (c_{0}, 𝕊_{0})$ ,

𝚫^{⊤} ​ 𝑰 ​ (𝜶^{*}) ​ 𝚫 \geq κ_{I} ​ {‖ 𝚫 ‖}_{2}^{2} .

Remark 1.

The constant $c_{0}$ is chosen to match the cone where the estimation error $\hat{𝛂} - 𝛂^{*}$ lies. A sufficient choice is $c_{0} \geq \frac{A_{0} + 1}{A_{0} - 1}$ when the regularization level satisfies $λ \geq A_{0} {‖ \nabla ℓ_{N} (𝛂^{*}) ‖}_{\infty}$ with $A_{0} > 1$ .

Assumption 12 (Uniform Hessian Concentration).

et $d_{α} = (K - 1) + K D + K D (D + 1) / 2$ denote the number of free parameters in $𝛂$ . There exist constants $C_{H} > 0$ , $c_{1}^{H}, c_{2}^{H} > 0$ and a radius $δ_{R} > 0$ such that, for $N ≳ s_{0} \ln d_{α}$ , with probability at least $1 - c_{1}^{H} d_{α}^{- c_{2}^{H}}$ ,

sup_{\begin{matrix} \tilde{𝜶} \in 𝔹 ​ (𝜶^{*}, δ_{R}) \cap 𝚯_{𝕍}^{'} \\ 𝚫 \in 𝒞 ​ (c_{0}, 𝕊_{0}), {‖ 𝚫 ‖}_{2} = 1 \end{matrix}} | 𝚫^{⊤} ​ (𝑯_{N} ​ (\tilde{𝜶}) - 𝑰 ​ (\tilde{𝜶})) ​ 𝚫 | \leq C_{H} ​ \sqrt{\frac{s_{0} ​ \ln d_{α}}{N}} .

The constant $C_{H}$ depends on the bounds in Assumption 8 and on moment bounds for derivatives of $ℓ$ . The radius can be taken as $δ_{R} ≍ \sqrt{s_{0} \ln d_{α} / N}$ .

Lemma 2 (Gradient Bound).

Let $𝛂^{*} = (𝛑^{*}, 𝛍_{1}^{*}, \dots, 𝛍_{K}^{*}, 𝚿_{1}^{*}, \dots, 𝚿_{K}^{*})$ be the true parameter vector of a $K$ -component GMM with $D$ -dimensional components. Suppose Assumptions 7-10 hold. Let $d_{α}$ be defined in Assumption 12. Then there exist absolute constants $C_{1}, C_{2} > 0$ and $C_{g} > 0$ such that, for any $N \geq 1$ ,

ℙ ​ ({‖ \nabla ℓ_{N} ​ (𝜶^{*}) ‖}_{\infty} \leq C_{g} ​ \sqrt{\frac{\ln d_{α}}{N}}) \geq 1 - C_{1} ​ d_{α}^{- C_{2}},

where $ℓ_{N} (𝛂) = \frac{1}{N} \sum_{n = 1}^{N} ℓ (𝐲_{n}; 𝛂)$ is the empirical negative log-likelihood (recall $ℓ (𝐲; 𝛂) = - \ln f_{clust} (𝐲; 𝛂)$ ). An explicit choice is $C_{g} = \sqrt{2 (C_{2} + 1) / c_{0}} ν_{\max}$ , where $c_{0}$ is an absolute constant in Bernstein’s inequality (e.g., $c_{0} = 1 / 2$ ). This bound holds provided $N \geq C_{B} \ln d_{α}$ , with

C_{B} = \frac{2 ​ (C_{2} + 1) ​ ν_{\max}^{2}}{c_{0} ​ {(\min_{j : b_{j} \neq 0} (𝔼 ​ [S_{j ​ n}^{2}] / b_{j}))}^{2}},

$ν_{\max}^{2} = \max_{j} 𝔼 [S_{j n}^{2}]$ , and $b_{j}$ is the sub-exponential scale of $S_{j n}$ ( $b_{j} = 0$ for sub-Gaussian components).

Proof of Lemma 2.

By Assumptions 7 and 9 (regularity and correct specification),

𝔼_{𝜶^{*}} ​ [S_{j} ​ (𝒚_{n}; 𝜶^{*})] = 0 for all ​ j = 1, \dots, d_{α}

Let $ν_{j}^{2} = 𝔼 [S_{j} {(𝒚_{n}; 𝜶^{*})}^{2}]$ be the variance of the score evaluated at $α_{j}^{*}$ . We bound variances and tail parameters for each parameter block.

Mixing proportions $S_{π_{k}} (y_{n}; α^{*})$ . By Lemma 1,

S_{π_{k}} ​ (𝒚_{n}; 𝜶^{*}) = \frac{t_{K} ​ (𝒚_{n}; 𝜶^{*})}{π_{K}^{*}} - \frac{t_{k} ​ (𝒚_{n}; 𝜶^{*})}{π_{k}^{*}} .

By Assumption 10, $0 < t_{m} (𝒚_{n}; 𝜶^{*}) \leq 1$ , and by Assumption 8, $π_{m}^{*} \geq π_{\min} > 0$ ; hence $| S_{π_{k}} (𝒚_{n}; 𝜶^{*}) | \leq 2 / π_{\min}$ . Thus $S_{π_{k}}$ is bounded sub-Gaussian with $ν_{π_{k}}^{2} = 𝔼 [S_{π_{k}}^{2}] \leq {(2 / π_{\min})}^{2}$ and $b_{π_{k}} = 0$ .

Mean parameters $S_{{(μ_{k})}_{d}} (y_{n}; α^{*})$ . From Lemma 1,

S_{{(𝝁_{k})}_{d}} ​ (𝒚_{n}; 𝜶^{*}) = - t_{k} ​ (𝒚_{n}; 𝜶^{*}) ​ {(𝚿_{k}^{*} ​ (𝒚_{n} - 𝝁_{k}^{*}))}_{d} .

Let $U_{n k d} := {(𝚿_{k}^{*} (𝒚_{n} - 𝝁_{k}^{*}))}_{d}$ . Conditionally on $z_{n} = ℓ$ , we have $𝒚_{n} = 𝝁_{ℓ}^{*} + ε_{ℓ}$ with $ε_{ℓ} \sim 𝒩 (𝟎, 𝚺_{ℓ}^{*})$ . Then

Var ​ (U_{n ​ k ​ d} ∣ z_{n} = ℓ) = e_{d}^{⊤} ​ 𝚿_{k}^{*} ​ 𝚺_{ℓ}^{*} ​ 𝚿_{k}^{*} ​ e_{d} \leq {‖ 𝚿_{k}^{*} ‖}_{2}^{2} ​ {‖ 𝚺_{ℓ}^{*} ‖}_{2} \leq θ_{\max}^{2} ​ σ_{\max}^{2},

and

| 𝔼 [U_{n ​ k ​ d} ∣ z_{n} = ℓ] | = | e_{d}^{⊤} 𝚿_{k}^{*} (𝝁_{ℓ}^{*} - 𝝁_{k}^{*}) | \leq ∥ 𝚿_{k}^{*} ∥_{2} ∥ 𝝁_{ℓ}^{*} - 𝝁_{k}^{*} ∥_{2} \leq θ_{\max} (2 η) .

Hence $𝔼 [U_{n k d}^{2}] \leq θ_{\max}^{2} (σ_{\max}^{2} + 4 η^{2})$ . Since $| t_{k} | \leq 1$ ,

ν_{{(𝝁_{k})}_{d}}^{2} = 𝔼 ​ [S_{{(𝝁_{k})}_{d}}^{2}] \leq 𝔼 ​ [U_{n ​ k ​ d}^{2}] \leq θ_{\max}^{2} ​ (σ_{\max}^{2} + 4 ​ η^{2}) .

Moreover, $U_{n k d}$ is (conditionally) Gaussian and hence sub-Gaussian; with the bounded multiplier $t_{k}$ , $S_{{(𝝁_{k})}_{d}}$ is sub-Gaussian, so $b_{{(𝝁_{k})}_{d}} = 0$ .

Precision entries $S_{{(𝚿_{k})}_{r s}} (y_{n}; α^{*})$ . From Lemma 1,

S_{{(𝚿_{k})}_{r ​ s}} ​ (𝒚_{n}; 𝜶^{*}) = - t_{k} ​ (𝒚_{n}; 𝜶^{*}) ​ \frac{(2 - δ_{r ​ s})}{2} ​ ({(𝚺_{k}^{*})}_{r ​ s} - {(𝒚_{n} - 𝝁_{k}^{*})}_{r} ​ {(𝒚_{n} - 𝝁_{k}^{*})}_{s}) .

Let $V_{n k r s} := {(𝚺_{k}^{*})}_{r s} - {(𝒚_{n} - 𝝁_{k}^{*})}_{r} {(𝒚_{n} - 𝝁_{k}^{*})}_{s}$ . Each ${(𝒚_{n} - 𝝁_{k}^{*})}_{r}$ is sub-Gaussian with parameter $≲ σ_{\max} + η$ , so the product ${(𝒚_{n} - 𝝁_{k}^{*})}_{r} {(𝒚_{n} - 𝝁_{k}^{*})}_{s}$ is sub-exponential; hence $V_{n k r s}$ is sub-exponential. Since $| t_{k} | \leq 1$ , $S_{{(𝚿_{k})}_{r s}}$ is sub-exponential. Thus there exist constants $C_{Ψ, ν}, C_{Ψ, b} > 0$ such that

ν_{{(𝚿_{k})}_{r ​ s}}^{2} = 𝔼 ​ [S_{{(𝚿_{k})}_{r ​ s}}^{2}] \leq C_{Ψ, ν} ​ {(σ_{\max}^{2} + η^{2})}^{2}, b_{{(𝚿_{k})}_{r ​ s}} \leq C_{Ψ, b} ​ (σ_{\max}^{2} + η^{2}) .

Union bound. Let $ν_{\max}^{2} = \max_{j} 𝔼 [S_{j n}^{2}]$ and $b_{\max} = \max_{j} {b_{j} : b_{j} \neq 0}$ ; by the above bounds these depend only on $(π_{\min}, η, s_{m}, s_{M})$ . By Bernstein’s inequality, for each coordinate $j$ and any $t > 0$ ,

ℙ ​ (| \frac{1}{N} ​ \sum_{n = 1}^{N} S_{j ​ n} | \geq t) \leq 2 ​ \exp (- c_{0} ​ N ​ \min (\frac{t^{2}}{ν_{j}^{2}}, \frac{t}{b_{j}})),

with the convention that if $b_{j} = 0$ then $\min (\cdot) = t^{2} / ν_{j}^{2}$ . Set $t_{N} = C_{g} \sqrt{\frac{\ln d_{α}}{N}}$ and assume $N \geq C_{B} \ln d_{α}$ so that $t_{N} \leq \min_{j : b_{j} \neq 0} ν_{j}^{2} / b_{j}$ . Then for all $j$ ,

ℙ ​ (| \frac{1}{N} ​ \sum_{n = 1}^{N} S_{j ​ n} | \geq t_{N}) \leq 2 ​ \exp (- \frac{c_{0} ​ N ​ t_{N}^{2}}{2 ​ ν_{j}^{2}}) \leq 2 ​ \exp (- \frac{c_{0} ​ N ​ t_{N}^{2}}{2 ​ ν_{\max}^{2}}) = 2 ​ d_{α}^{- \frac{c_{0} ​ C_{g}^{2}}{2 ​ ν_{\max}^{2}}} .

A union bound over $j = 1, \dots, d_{α}$ yields

ℙ ​ ({‖ \nabla ℓ_{N} ​ (𝜶^{*}) ‖}_{\infty} \geq t_{N}) \leq 2 ​ d_{α}^{1 - \frac{c_{0} ​ C_{g}^{2}}{2 ​ ν_{\max}^{2}}} .

Choosing $C_{1} = 2$ and $C_{g}$ so that $\frac{c_{0} C_{g}^{2}}{2 ν_{\max}^{2}} = C_{2} + 1$ gives the claim, i.e., $C_{g} = \sqrt{2 (C_{2} + 1) / c_{0}} ν_{\max}$ . Finally, the condition $N \geq C_{B} \ln d_{α}$ is guaranteed by taking

C_{B} = \frac{C_{g}^{2}}{{(\min_{j : b_{j} \neq 0} ν_{j}^{2} / b_{j})}^{2}} = \frac{2 ​ (C_{2} + 1) ​ ν_{\max}^{2}}{c_{0} ​ {(\min_{j : b_{j} \neq 0} (𝔼 ​ [S_{j ​ n}^{2}] / b_{j}))}^{2}} .

∎

Lemma 3 (Parameter Consistency for Penalized GMM Estimator).

Let $\hat{𝛂}$ be any local minimizer of the penalized negative average log-likelihood

Q ​ (𝜶) = ℓ_{N} ​ (𝜶) + P ​ (𝜶),

where

ℓ_{N} ​ (𝜶) = - \frac{1}{N} ​ \sum_{n = 1}^{N} \ln f_{clust} ​ (𝒚_{n}; 𝜶), P ​ (𝜶) = λ ​ \sum_{k = 1}^{K} {‖ 𝝁_{k} ‖}_{1} + ρ ​ \sum_{k = 1}^{K} {‖ 𝚿_{k} ‖}_{1} .

Let $𝛂^{*}$ be the true GMM parameter vector, and $𝕊_{0}$ be the true support of the penalized parameters in $𝛂^{*}$ , with sparsity $s_{0} = | 𝕊_{0} |$ . Let $d_{α}$ be the total number of free parameters in $𝛂$ as defined in Assumptions 12.

Suppose Assumptions 7-12 hold and the gradient bound in Lemma 2 is satisfied. Choose the regularization parameters $λ = ρ = λ_{chosen}$ , where

λ_{chosen} = A_{0} ​ C_{g} ​ \sqrt{\frac{\ln d_{α}}{N}}

for a constant $A_{0} > 1$ (e.g. $A_{0} = 3$ ). Then, provided $N ≳ s_{0} \ln d_{α}$ , with probability at least $1 - C_{1}^{grad} d_{α}^{- C_{2}^{grad}} - c_{1}^{H} d_{α}^{- c_{2}^{H}}$ :

1.

( $L_{2}$ -norm consistency):

${‖ \hat{𝜶} - 𝜶^{*} ‖}_{2} \leq \frac{2 (A_{0} + 1) C_{g}}{κ_{I}} \sqrt{\frac{s_{0} \ln d_{α}}{N}} .$
2.

( $L_{1}$ -norm consistency):

${‖ \hat{𝜶} - 𝜶^{*} ‖}_{1} \leq \frac{4 A_{0} (A_{0} + 1) C_{g}}{(A_{0} - 1) κ_{I}} s_{0} \sqrt{\frac{\ln d_{α}}{N}} .$

Here $C_{g}$ is the gradient bound constant from Lemma 2, and $κ_{I}$ is the restricted eigenvalue constant from Assumption 11. The factor $A_{0}$ is a user-chosen constant controlling the regularization strength.

Proof of Lemma 3.

Let $\hat{𝜶}$ be a minimizer of the penalized negative log-likelihood $Q (𝜶) = ℓ_{N} (𝜶) + P (𝜶)$ , where $P (𝜶) = λ \sum_{k} {‖ 𝝁_{k} ‖}_{1} + ρ \sum_{k} {‖ 𝚿_{k} ‖}_{1}$ and $ℓ_{N} (𝜶) = \frac{1}{N} \sum_{n = 1}^{N} ℓ (𝒚_{n}; 𝜶)$ with $ℓ (𝒚; 𝜶) = - \ln f_{clust} (𝒚; 𝜶)$ . Since $\hat{𝜶}$ minimizes $Q$ , we have

ℓ_{N} ​ (\hat{𝜶}) - ℓ_{N} ​ (𝜶^{*}) \leq P ​ (𝜶^{*}) - P ​ (\hat{𝜶}) .

(20)

Let $𝚫 = \hat{𝜶} - 𝜶^{*}$ and assume $\hat{𝜶} \in 𝚯_{𝕍}^{'}$ so that $𝜶^{*} + 𝚫 \in 𝚯_{𝕍}^{'}$ . A second-order Taylor expansion gives

ℓ_{N} ​ (\hat{𝜶}) - ℓ_{N} ​ (𝜶^{*}) = ⟨ \nabla ℓ_{N} ​ (𝜶^{*}), 𝚫 ⟩ + \frac{1}{2} ​ 𝚫^{⊤} ​ 𝑯_{N} ​ (𝜶^{*} + t_{0} ​ 𝚫) ​ 𝚫

for some $t_{0} \in (0, 1)$ . Write $\tilde{𝜶} = 𝜶^{*} + t_{0} 𝚫$ . We lower bound $\frac{1}{2} 𝚫^{⊤} 𝑯_{N} (\tilde{𝜶}) 𝚫$ by decomposing

𝚫^{⊤} ​ 𝑯_{N} ​ (\tilde{𝜶}) ​ 𝚫

= 𝚫^{⊤} ​ 𝑰 ​ (𝜶^{*}) ​ 𝚫 + 𝚫^{⊤} ​ (𝑰 ​ (\tilde{𝜶}) - 𝑰 ​ (𝜶^{*})) ​ 𝚫 + 𝚫^{⊤} ​ (𝑯_{N} ​ (\tilde{𝜶}) - 𝑰 ​ (\tilde{𝜶})) ​ 𝚫 .

From this point, all bounds are stated for arbitrary $𝚫$ in the cone $𝒞 (c_{0}, 𝕊_{0})$ ; the fact that the actual error $\hat{𝜶} - 𝜶^{*}$ lies in $𝒞 (c_{0}, 𝕊_{0})$ will be established later by Lemma 4.

By Assumption 11, for $𝚫 \in 𝒞 (c_{0}, 𝕊_{0})$ ,

𝚫^{⊤} ​ 𝑰 ​ (𝜶^{*}) ​ 𝚫 \geq κ_{I} ​ {‖ 𝚫 ‖}_{2}^{2} .

By Assumption 7 and compactness (Assumption 8), $𝜶 \mapsto 𝑰 (𝜶) = 𝔼 [\nabla^{2} ℓ (𝒚; 𝜶)]$ is Lipschitz on $𝔹 (𝜶^{*}, r_{0}) \cap 𝚯_{𝕍}^{'}$ in spectral norm: there exists $L_{I} > 0$ such that

{‖ 𝑰 ​ (\tilde{𝜶}) - 𝑰 ​ (𝜶^{*}) ‖}_{2} \leq L_{I} ​ {‖ \tilde{𝜶} - 𝜶^{*} ‖}_{2} = L_{I} ​ t_{0} ​ {‖ 𝚫 ‖}_{2} \leq L_{I} ​ {‖ 𝚫 ‖}_{2},

and therefore

| 𝚫^{⊤} ​ (𝑰 ​ (\tilde{𝜶}) - 𝑰 ​ (𝜶^{*})) ​ 𝚫 | \leq {‖ 𝑰 ​ (\tilde{𝜶}) - 𝑰 ​ (𝜶^{*}) ‖}_{2} ​ {‖ 𝚫 ‖}_{2}^{2} \leq L_{I} ​ {‖ 𝚫 ‖}_{2}^{3} \leq L_{I} ​ δ_{R} ​ {‖ 𝚫 ‖}_{2}^{2},

provided ${‖ 𝚫 ‖}_{2} \leq δ_{R}$ . By Assumption 12, if ${‖ 𝚫 ‖}_{2} \leq δ_{R}$ and $𝚫 \in 𝒞 (c_{0}, 𝕊_{0})$ , then with probability at least $1 - c_{1}^{H} d_{α}^{- c_{2}^{H}}$ ,

| 𝚫^{⊤} ​ (𝑯_{N} ​ (\tilde{𝜶}) - 𝑰 ​ (\tilde{𝜶})) ​ 𝚫 | \leq C_{H} ​ \sqrt{\frac{s_{0} ​ \ln d_{α}}{N}} ​ {‖ 𝚫 ‖}_{2}^{2} .

Combining the three representations, for $𝚫 \in 𝒞 (c_{0}, 𝕊_{0})$ with ${‖ 𝚫 ‖}_{2} \leq δ_{R}$ ,

\frac{1}{2} ​ 𝚫^{⊤} ​ 𝑯_{N} ​ (\tilde{𝜶}) ​ 𝚫 \geq \frac{1}{2} ​ (κ_{I} - L_{I} ​ {‖ 𝚫 ‖}_{2} - C_{H} ​ \sqrt{\frac{s_{0} ​ \ln d_{α}}{N}}) ​ {‖ 𝚫 ‖}_{2}^{2} .

Choose $δ_{R}$ and $N$ so that $L_{I} δ_{R} \leq κ_{I} / 4$ and $C_{H} \sqrt{\frac{s_{0} \ln d_{α}}{N}} \leq κ_{I} / 4$ (e.g., $N ≳ (C_{H}^{2} / κ_{I}^{2}) s_{0} \ln d_{α}$ ). Then, with the same probability,

\frac{1}{2} ​ 𝚫^{⊤} ​ 𝑯_{N} ​ (\tilde{𝜶}) ​ 𝚫 \geq \frac{κ_{I}}{4} ​ {‖ 𝚫 ‖}_{2}^{2} .

Hence we obtain the restricted strong convexity (RSC) inequality on the cone:

ℓ_{N} ​ (𝜶^{*} + 𝚫) - ℓ_{N} ​ (𝜶^{*}) - ⟨ \nabla ℓ_{N} ​ (𝜶^{*}), 𝚫 ⟩ \geq \frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2}, for all ​ 𝚫 \in 𝒞 ​ (c_{0}, 𝕊_{0}), {‖ 𝚫 ‖}_{2} \leq δ_{R},

(21)

where $κ_{L} = κ_{I} / 2 > 0$ .

Stochastic term. By Lemma 2, if $N \geq C_{B} \ln d_{α}$ , then with probability at least $1 - C_{1}^{grad} d_{α}^{- C_{2}^{grad}}$ ,

{‖ \nabla ℓ_{N} ​ (𝜶^{*}) ‖}_{\infty} \leq C_{g} ​ \sqrt{\frac{\ln d_{α}}{N}} .

By Hölder’s inequality $(⟨ x, y ⟩ <= {∥ x ∥}_{\infty} {∥ y ∥}_{1})$ ,

| ⟨ \nabla ℓ_{N} ​ (𝜶^{*}), 𝚫 ⟩ | \leq {‖ \nabla ℓ_{N} ​ (𝜶^{*}) ‖}_{\infty} ​ {‖ 𝚫 ‖}_{1} \leq C_{g} ​ \sqrt{\frac{\ln d_{α}}{N}} ​ {‖ 𝚫 ‖}_{1} .

(22)

Penalty difference. Let $𝚫_{𝝁_{k}} = {\hat{𝝁}}_{k} - 𝝁_{k}^{*}$ and $𝚫_{𝚿_{k}} = {\hat{𝚿}}_{k} - 𝚿_{k}^{*}$ , and denote by $𝕊_{𝝁}^{*}$ and $𝕊_{𝚿}^{*}$ the true supports (Definition 3). Using the $L_{1}$ -triangle inequality on supports: for any vector $𝒂, 𝒃$ and support $𝕊$ of $𝒂$ : ${‖ 𝒂 ‖}_{1} - {‖ 𝒃 ‖}_{1} \leq {‖ 𝒂_{𝕊} - 𝒃_{𝕊} ‖}_{1} - {‖ 𝒂_{{(𝕊)}^{c}} - 𝒃_{{(𝕊)}^{c}} ‖}_{1} + 2 {‖ 𝒂_{{(𝕊)}^{c}} ‖}_{1}$ . Since $𝒂_{{(𝕊)}^{c}} = 𝟎$ :

{‖ 𝝁_{k}^{*} ‖}_{1} - {‖ {\hat{𝝁}}_{k} ‖}_{1} \leq {‖ 𝚫_{𝝁_{k}, 𝕊_{𝝁}^{*}} ‖}_{1} - {‖ 𝚫_{𝝁_{k}, {(𝕊_{𝝁}^{*})}^{c}} ‖}_{1}, {‖ 𝚿_{k}^{*} ‖}_{1} - {‖ {\hat{𝚿}}_{k} ‖}_{1} \leq {‖ 𝚫_{𝚿_{k}, 𝕊_{𝚿}^{*}} ‖}_{1} - {‖ 𝚫_{𝚿_{k}, {(𝕊_{𝚿}^{*})}^{c}} ‖}_{1} .

Summing over $k$ and writing $𝕊_{0} = 𝕊_{𝝁}^{*} \cup 𝕊_{𝚿}^{*}$ ,

P ​ (𝜶^{*}) - P ​ (\hat{𝜶})

\leq λ ​ ({‖ 𝚫_{𝝁, 𝕊_{0}} ‖}_{1} - {‖ 𝚫_{𝝁, {(𝕊_{0})}^{c}} ‖}_{1}) + ρ ​ ({‖ 𝚫_{𝚿, 𝕊_{0}} ‖}_{1} - {‖ 𝚫_{𝚿, {(𝕊_{0})}^{c}} ‖}_{1}) .

(23)

Combining. With probability at least $1 - C_{1}^{grad} d_{α}^{- C_{2}^{grad}} - c_{1}^{H} d_{α}^{- c_{2}^{H}}$ , combining Equation 21, Equation 22, and Equation 23 with Equation 20 yields, for all $𝚫 \in 𝒞 (c_{0}, 𝕊_{0})$ with ${‖ 𝚫 ‖}_{2} \leq δ_{R}$ ,

\frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2} \leq C_{g} ​ \sqrt{\frac{\ln d_{α}}{N}} ​ {‖ 𝚫 ‖}_{1} + λ ​ ({‖ 𝚫_{𝝁, 𝕊_{0}} ‖}_{1} - {‖ 𝚫_{𝝁, {(𝕊_{0})}^{c}} ‖}_{1}) + ρ ​ ({‖ 𝚫_{𝚿, 𝕊_{0}} ‖}_{1} - {‖ 𝚫_{𝚿, {(𝕊_{0})}^{c}} ‖}_{1}) .

(24)

Invoking the inequality Equation 29 from Lemma 4 (the cone lemma, proved via KKT and the gradient bound, thus independent of RSC),

\frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2} \leq (1 + \frac{1}{A_{0}}) ​ \sum_{j \in 𝕊_{0}} λ_{j} ​ | Δ_{j} | - (1 - \frac{1}{A_{0}}) ​ \sum_{j \in {(𝕊_{0})}^{c}} λ_{j} ​ | Δ_{j} | .

Since the second term is non-positive (as $A_{0} > 1$ and $λ_{j} | Δ_{j} | \geq 0$ ), we can drop it to get an upper bound:

\frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2} \leq (1 + \frac{1}{A_{0}}) ​ \sum_{j \in 𝕊_{0}} λ_{j} ​ | Δ_{j} | .

Taking the common regularization level $λ_{chosen} = λ = ρ = A_{0} C_{g} \sqrt{\frac{\ln d_{α}}{N}}$ with $A_{0} > 1$ and note that $\sum_{j \in 𝕊_{0}} λ_{j} | Δ_{j} | = λ_{chosen} {‖ 𝚫_{𝕊_{0}} ‖}_{1}$ , we obtain

\frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2} \leq (1 + \frac{1}{A_{0}}) ​ λ_{chosen} ​ {‖ 𝚫_{𝕊_{0}} ‖}_{1} .

Using the Cauchy-Schwarz inequality ${‖ 𝚫_{𝕊_{0}} ‖}_{1} \leq \sqrt{s_{0}} {‖ 𝚫_{𝕊_{0}} ‖}_{2}$ , and since ${‖ 𝚫_{𝕊_{0}} ‖}_{2} \leq {‖ 𝚫 ‖}_{2}$ :

\frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2} \leq λ_{chosen} ​ (1 + \frac{1}{A_{0}}) ​ \sqrt{s_{0}} ​ {‖ 𝚫 ‖}_{2} .

If ${‖ 𝚫 ‖}_{2} \neq 0$ , we can divide by ${‖ 𝚫 ‖}_{2}$ :

{‖ 𝚫 ‖}_{2} \leq \frac{2 ​ λ_{chosen}}{κ_{L}} ​ (1 + \frac{1}{A_{0}}) ​ \sqrt{s_{0}} .

Substituting $λ_{chosen} = A_{0} C_{g} \sqrt{\frac{\ln d_{α}}{N}}$ :

	${‖ \hat{𝜶} - 𝜶^{*} ‖}_{2}$	$\leq \frac{2 A_{0} C_{g}}{κ_{L}} (1 + \frac{1}{A_{0}}) \sqrt{s_{0}} \sqrt{\frac{\ln d_{α}}{N}}$
		$= \frac{2 (A_{0} + 1) C_{g}}{κ_{L}} \sqrt{\frac{s_{0} \ln d_{α}}{N}} .$

This proves the $L_{2}$ -rate with constant $C_{L 2} = \frac{2 (A_{0} + 1) C_{g}}{κ_{L}}$ .

For the $L_{1}$ -rate, Lemma 4 yields the cone bound ${‖ 𝚫_{{(𝕊_{0})}^{c}} ‖}_{1} \leq C_{cone} {‖ 𝚫_{𝕊_{0}} ‖}_{1}$ with $C_{cone} = \frac{A_{0} + 1}{A_{0} - 1}$ (decomposability via [44]). Hence, assuming all $λ_{j}$ for penalized components are $λ_{chosen}$ ):

	${‖ 𝚫 ‖}_{1}$	$= {‖ 𝚫_{𝕊_{0}} ‖}_{1} + {‖ 𝚫_{{(𝕊_{0})}^{c}} ‖}_{1}$
		$\leq {‖ 𝚫_{𝕊_{0}} ‖}_{1} + C_{cone} {‖ 𝚫_{𝕊_{0}} ‖}_{1} = (1 + C_{cone}) {‖ 𝚫_{𝕊_{0}} ‖}_{1}$
		$\leq (1 + C_{cone}) \sqrt{s_{0}} {‖ 𝚫_{𝕊_{0}} ‖}_{2} (by Cauchy-Schwarz)$
		$\leq (1 + C_{cone}) \sqrt{s_{0}} {‖ 𝚫 ‖}_{2} .$

Substituting $C_{cone} = \frac{A_{0} + 1}{A_{0} - 1}$ :

1 + C_{cone} = 1 + \frac{A_{0} + 1}{A_{0} - 1} = \frac{A_{0} - 1 + A_{0} + 1}{A_{0} - 1} = \frac{2 ​ A_{0}}{A_{0} - 1} .

So,

{‖ 𝚫 ‖}_{1} \leq (\frac{2 ​ A_{0}}{A_{0} - 1}) ​ \sqrt{s_{0}} ​ {‖ 𝚫 ‖}_{2} .

Now substitute the bound for ${‖ 𝚫 ‖}_{2}$ :

	${‖ \hat{𝜶} - 𝜶^{*} ‖}_{1}$	$\leq (\frac{2 A_{0}}{A_{0} - 1}) \sqrt{s_{0}} (\frac{2 (A_{0} + 1) C_{g}}{κ_{L}} \sqrt{\frac{s_{0} \ln d_{α}}{N}})$
		$= \frac{4 A_{0} (A_{0} + 1) C_{g}}{(A_{0} - 1) κ_{L}} s_{0} \sqrt{\frac{\ln d_{α}}{N}} .$
		$= C_{L 1} (A_{0}, C_{g}, κ_{L}) s_{0} \sqrt{\frac{\ln d_{α}}{N}}$

where $C_{L 1} (A_{0}, C_{g}, κ_{L}) = \frac{4 A_{0} (A_{0} + 1) C_{g}}{(A_{0} - 1) κ_{L}}$ . This concludes the proof of parameter consistency in $L_{1}$ and $L_{2}$ norms. ∎

Remark 2.

The radius $δ_{R}$ governing the RSC inequality Equation 21 is critical and is typically of the same order as the target statistical error. If ${‖ 𝚫 ‖}_{2}$ exceeds $δ_{R}$ , the Lipschitz remainder $L_{I} {‖ 𝚫 ‖}_{2}^{3}$ can dominate, and/or the uniform Hessian concentration in Assumption 12 may fail on such a large neighborhood of $𝛂^{*}$ . In general M-estimation analyses (e.g., [44]), one often proves an RSC with a tolerance term,

ℓ_{N} ​ (𝜶^{*} + 𝚫) - ℓ_{N} ​ (𝜶^{*}) - ⟨ \nabla ℓ_{N} ​ (𝜶^{*}), 𝚫 ⟩ \geq \frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2} - τ_{L} ​ \frac{\ln d_{α}}{N} ​ {‖ 𝚫 ‖}_{1}^{2},

valid on a larger set. Under Assumption 12, we obtain sufficiently strong control to dispense with this tolerance and derive the cleaner quadratic curvature bound Equation 21. If Assumption 12 were weakened (e.g., only yielding a bound of the form $C_{H} \sqrt{\frac{\ln d_{α}}{N}} \frac{{‖ 𝚫 ‖}_{1}}{\sqrt{s_{0}}} {‖ 𝚫 ‖}_{2}$ on $𝚫^{⊤} (𝐇_{N} - 𝐈) 𝚫$ over the cone), then a tolerance term proportional to ${‖ 𝚫 ‖}_{1}^{2}$ would naturally appear in the RSC.

Lemma 4 (Cone Condition for $L_{1}$ -Penalized M-Estimators).

Let $\hat{𝛂}$ be any local minimizer of $Q (𝛂) = ℓ_{N} (𝛂) + P (𝛂)$ , where $ℓ_{N} (𝛂)$ is a differentiable loss function and the penalty $P (𝛂)$ is a sum of component-wise $L_{1}$ penalties:

P ​ (𝜶) = \sum_{j = 1}^{d_{α}} λ_{j} ​ | α_{j} | .

Let $𝕊_{0}$ be the true support of $𝛂^{*}$ . Let $𝚫 = \hat{𝛂} - 𝛂^{*}$ . Suppose the regularization parameters are chosen such that for some constant $A_{0} > 1$ :

λ_{j} \geq A_{0} ​ | {[\nabla ℓ_{N} ​ (𝜶^{*})]}_{j} | for all ​ j \in {(𝕊_{0})}^{c} .

(25)

and $λ_{j}$ of similar order for $j \in 𝕊_{0}$ . For simplicity, we often set $λ_{j} = λ_{chosen}$ for all penalized components, where $λ_{chosen} \geq A_{0} {‖ \nabla ℓ_{N} (𝛂^{*}) ‖}_{\infty}$ . If the RSC condition from Equation 21 holds for $𝚫$ :

ℓ_{N} ​ (𝜶^{*} + 𝚫) - ℓ_{N} ​ (𝜶^{*}) - ⟨ \nabla ℓ_{N} ​ (𝜶^{*}), 𝚫 ⟩ \geq \frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2},

then, for $A_{0} > 1$ , the error vector $𝚫$ satisfies the cone condition:

\sum_{j \in {(𝕊_{0})}^{c}} λ_{j} ​ | Δ_{j} | \leq \frac{A_{0} + 1}{A_{0} - 1} ​ \sum_{j \in 𝕊_{0}} λ_{j} ​ | Δ_{j} | .

(26)

If all $λ_{j}$ for penalized components are equal to $λ_{chosen}$ , this simplifies to:

{‖ 𝚫_{{(𝕊_{0})}^{c}} ‖}_{pen, 1} \leq \frac{A_{0} + 1}{A_{0} - 1} ​ {‖ 𝚫_{𝕊_{0}} ‖}_{pen, 1},

where $∥ \cdot ∥_{pen, 1}$ refers to the $L_{1}$ norm over the components that are actually penalized. If all components were penalized with the same $λ_{0}$ , this would be ${‖ 𝚫_{{(𝕊_{0})}^{c}} ‖}_{1} \leq \frac{A_{0} + 1}{A_{0} - 1} {‖ 𝚫_{𝕊_{0}} ‖}_{1}$ .

Remark 3.

In our case, $λ_{j} = λ$ for mean components $μ_{k d}$ , and $λ_{j} = ρ$ for off-diagonal precision components ${(Ψ_{k})}_{r s}$ , and $λ_{j} = 0$ for parameters not penalized like proportions or diagonal precision elements if they are not penalized towards a specific value. Moreover, the Cone Condition ensures that the error vector is primarily concentrated on the true support. The key is that the regularization parameter for the "noise" variables (off-support) must be sufficiently larger than the corresponding component of the score vector, allowing the penalty to effectively shrink noise components.

Proof of Lemma 4.

Let $𝒫 \subset {1, \dots, d_{α}}$ denote the index set of penalized coordinates and write ${‖ 𝒙 ‖}_{pen, 1} = \sum_{j \in 𝒫} | x_{j} |$ . In what follows, sums and $L_{1}$ -norms are taken over $𝒫$ . Assume the tuning condition holds on all penalized coordinates:

λ_{j} \geq A_{0} ​ | {[\nabla ℓ_{N} ​ (𝜶^{*})]}_{j} | \forall j \in 𝒫, A_{0} > 1,

(27)

equivalently $λ_{chosen} \geq A_{0} {‖ \nabla ℓ_{N} (𝜶^{*}) ‖}_{\infty, pen}$ when a common level is used. ²²2If some coordinates are unpenalized, they are excluded from $𝒫$ and do not enter the cone inequality.

Since $\hat{𝜶}$ is a local minimizer of $Q (𝜶)$ , it satisfies the basic optimality inequality

ℓ_{N} ​ (\hat{𝜶}) - ℓ_{N} ​ (𝜶^{*}) \leq P ​ (𝜶^{*}) - P ​ (\hat{𝜶}) .

Invoking the restricted curvature bound (RSC) Equation 21 for $𝚫 = \hat{𝜶} - 𝜶^{*}$ ,

ℓ_{N} ​ (\hat{𝜶}) - ℓ_{N} ​ (𝜶^{*}) \geq ⟨ \nabla ℓ_{N} ​ (𝜶^{*}), 𝚫 ⟩ + \frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2} .

Note that, only the nonnegativity of the quadratic remainder is needed for the cone; the explicit $κ_{L} > 0$ is used later for $L_{2}$ -rates. Therefore,

⟨ \nabla ℓ_{N} ​ (𝜶^{*}), 𝚫 ⟩ + \frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2} \leq P ​ (𝜶^{*}) - P ​ (\hat{𝜶}) .

(28)

For the penalty difference, write $P (𝜶) = \sum_{j \in 𝒫} λ_{j} | α_{j} |$ and let $𝕊_{0} \subseteq 𝒫$ denote the true support of the penalized coordinates. Using the standard $L_{1}$ support inequality with $𝒂 = 𝜶^{*}$ and $𝒃 = \hat{𝜶} = 𝜶^{*} + 𝚫$ ,

{‖ 𝒂 ‖}_{1} - {‖ 𝒃 ‖}_{1} = {‖ 𝒂_{𝕊_{0}} ‖}_{1} - {‖ 𝒃_{𝕊_{0}} ‖}_{1} - {‖ 𝒃_{{(𝕊_{0})}^{c}} ‖}_{1} \leq {‖ 𝒂_{𝕊_{0}} - 𝒃_{𝕊_{0}} ‖}_{1} - {‖ 𝒃_{{(𝕊_{0})}^{c}} ‖}_{1},

and since ${(𝜶^{*})}_{{(𝕊_{0})}^{c}} = 𝟎$ on $𝒫$ , we obtain

P ​ (𝜶^{*}) - P ​ (\hat{𝜶}) \leq \sum_{j \in 𝕊_{0}} λ_{j} ​ | Δ_{j} | - \sum_{j \in {(𝕊_{0})}^{c}} λ_{j} ​ | Δ_{j} | .

Substitute this into Equation 28 and bound the linear term by the triangle inequality over $𝒫$ :

⟨ \nabla ℓ_{N} ​ (𝜶^{*}), 𝚫 ⟩ \leq \sum_{j \in 𝒫} | {[\nabla ℓ_{N} ​ (𝜶^{*})]}_{j} | ​ | Δ_{j} | \leq \sum_{j \in 𝒫} \frac{λ_{j}}{A_{0}} ​ | Δ_{j} |,

where the last step uses Equation 27. We obtain

\frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2} \leq \sum_{j \in 𝕊_{0}} (λ_{j} + \frac{λ_{j}}{A_{0}}) ​ | Δ_{j} | - \sum_{j \in {(𝕊_{0})}^{c}} (λ_{j} - \frac{λ_{j}}{A_{0}}) ​ | Δ_{j} | .

Equivalently,

\frac{κ_{L}}{2} ​ {‖ 𝚫 ‖}_{2}^{2} \leq (1 + \frac{1}{A_{0}}) ​ \sum_{j \in 𝕊_{0}} λ_{j} ​ | Δ_{j} | - (1 - \frac{1}{A_{0}}) ​ \sum_{j \in {(𝕊_{0})}^{c}} λ_{j} ​ | Δ_{j} | .

(29)

Since the left-hand side is nonnegative, it follows that

(1 - \frac{1}{A_{0}}) ​ \sum_{j \in {(𝕊_{0})}^{c}} λ_{j} ​ | Δ_{j} | \leq (1 + \frac{1}{A_{0}}) ​ \sum_{j \in 𝕊_{0}} λ_{j} ​ | Δ_{j} |,

and for $A_{0} > 1$ this yields the cone inequality

\sum_{j \in {(𝕊_{0})}^{c}} λ_{j} ​ | Δ_{j} | \leq \frac{A_{0} + 1}{A_{0} - 1} ​ \sum_{j \in 𝕊_{0}} λ_{j} ​ | Δ_{j} | .

If all penalized coordinates share a common level $λ_{chosen}$ , this becomes ${‖ 𝚫_{{(𝕊_{0})}^{c}} ‖}_{pen, 1} \leq \frac{A_{0} + 1}{A_{0} - 1} {‖ 𝚫_{𝕊_{0}} ‖}_{pen, 1}$ , and if every coordinate is penalized equally it reduces to ${‖ 𝚫_{{(𝕊_{0})}^{c}} ‖}_{1} \leq \frac{A_{0} + 1}{A_{0} - 1} {‖ 𝚫_{𝕊_{0}} ‖}_{1}$ . ∎

With the cone condition in Lemma 4 established on the penalized coordinates, we now consolidate the standing assumptions for the ranking-consistency analysis (Lemma 5) and for Theorem 3. The goal is to avoid duplication, align the SRUW assumptions used earlier with the GMM penalized framework here, and make explicit exactly which conditions are invoked downstream.

Remark 4 (Alignment with SRUW assumptions).

Assumptions 4-6 were introduced for the SRUW model in the MNARz setting. In the present penalized GMM analysis for Theorem 3:

•

Model uniqueness (SRUW-4). We condition on a fixed mixture structure (number of components $K$ and dimension $D$ ) and require identifiability of the GMM density; this role is played by Assumption 7 below. We do not require the SRUW tuple $(K_{0}, m_{0}, r_{0}, l_{0}, 𝕍_{0})$ explicitly here because no model selection over SRUW structures is performed in Theorem 3.
•

Compactness and interiority (SRUW-5-6). These correspond directly to Assumption 8 below (compact parameter subset $𝚯_{𝕍}^{'}$ and $𝜶^{*}$ interior), together with the eigenvalue and boundedness constraints therein. Thus SRUW-5-6 are subsumed by Assumption 8.

Hence, for the purposes of Lemma 5 and Theorem 3, it suffices to work with Assumptions 7-12 below; SRUW-4-6 need not be re-stated.

To proceed rigorously towards Lemma 5 and, ultimately, Theorem 3, we now restate the full set of standing assumptions-consolidating those aligned with the $𝕊 ℝ 𝕌 𝕎$ framework and those introduced earlier for penalized likelihood analysis-into a unified assumption block.

Assumption 13 (Standing Assumptions for proving Theorem 3).

The following holds:

1.

Identifiability & Smoothness (Assumption 7): the GMM density $f_{clust} (𝒚; 𝜶)$ is identifiable and $ℓ (𝒚; 𝜶) = - \ln f_{clust} (𝒚; 𝜶)$ is three times continuously differentiable in an open ball $𝔹 (𝜶^{*}, r_{0})$ .
2.

Compactness & Bounds (Assumption 8): $𝜶^{*}$ is an interior point of a compact $𝚯_{𝕍}^{'} \subset 𝔹 (𝜶^{*}, r_{0})$ ; mixture weights, means, and covariance/precision eigenvalues satisfy the stated uniform bounds (with $π_{\min}, η, σ_{\min}^{2}, σ_{\max}^{2}, θ_{\min}, θ_{\max}$ ).
3.

Data-generating mechanism (Assumption 9): $𝒚_{1}, \dots, 𝒚_{N}$ i.i.d. from $f_{clust} (\cdot; 𝜶^{*})$ with latent $z_{n} \sim Mult (𝝅^{*})$ and $𝒚_{n} | z_{n} = k \sim 𝒩 (𝝁_{k}^{*}, 𝚺_{k}^{*})$ .
4.

Posterior responsibilities (Assumption 10): $t_{k} (𝒚; 𝜶) \in (0, 1]$ for all $𝜶 \in 𝚯_{𝕍}^{'}$ .
5.

Fisher RE on a cone (Assumption 11): restricted eigenvalue condition for $𝑰 (𝜶^{*})$ on $𝒞 (c_{0}, 𝕊_{0})$ with constant $κ_{I} > 0$ .
6.

Uniform Hessian concentration (Assumption 12): for radius $δ_{R} ≍ \sqrt{s_{0} \ln d_{α} / N}$ and $N ≳ s_{0} \ln d_{α}$ ,

$sup_{\tilde{𝜶} \in 𝔹 (𝜶^{*}, δ_{R}) \cap 𝚯_{𝕍}^{'}} sup_{𝚫 \in 𝒞 (c_{0}, 𝕊_{0}), {‖ 𝚫 ‖}_{2} = 1} | 𝚫^{⊤} (𝑯_{N} (\tilde{𝜶}) - 𝑰 (\tilde{𝜶})) 𝚫 | \leq C_{H} \sqrt{\frac{s_{0} \ln d_{α}}{N}}$

with probability at least $1 - c_{1}^{H} d_{α}^{- c_{2}^{H}}$ .
7.

Penalty on penalized coordinates. On the penalized index set $𝒫$ , choose $λ_{j}$ so that

$λ_{j} \geq A_{0} {‖ \nabla ℓ_{N} (𝜶^{*}) ‖}_{\infty, pen} for some A_{0} > 1,$

e.g. a common level $λ_{chosen} = A_{0} C_{g} \sqrt{\frac{\ln d_{α}}{N}}$ with $C_{g}$ from Lemma 2. Unpenalized coordinates are excluded from $𝒫$ and from all $∥ \cdot ∥_{pen, 1}$ norms.

Assumption 14 (Working high-probability event for ranking analysis).

Let $ℰ_{grad} = {{‖ \nabla ℓ_{N} (𝛂^{*}) ‖}_{\infty, pen} \leq C_{g} \sqrt{\ln d_{α} / N}}$ be the event in Lemma 2, and $ℰ_{RSC}$ the event on which Equation 21 holds with curvature $κ_{L} > 0$ over ${𝚫 \in 𝒞 (c_{0}, 𝕊_{0}) : {‖ 𝚫 ‖}_{2} \leq δ_{R}}$ . Under Assumption 13 and for $N$ large enough, both events hold with probability at least $1 - δ_{N}$ , where $δ_{N} = C_{1}^{grad} d_{α}^{- C_{2}^{grad}} + c_{1}^{H} d_{α}^{- c_{2}^{H}}$ . In the proof of Lemma 5, we condition on $ℰ_{grad} \cap ℰ_{RSC}$ .

Definition 4 (Quantities for Ranking Consistency).

Let $𝛂^{*}$ be the true GMM parameters, and $\hat{𝛂} (λ)$ be the estimator for a given regularization level $λ$ (with $ρ$ either tied to $λ$ or fixed).

•

Noise level.

$λ_{noise} = C_{g} \sqrt{\frac{\ln d_{α}}{N}},$

the uniform bound on the score at $𝜶^{*}$ from Lemma 2.
•

Effective curvature for means.

$H_{k j}^{eff} (𝜶^{*}) = 𝔼_{𝜶^{*}} [t_{k} (𝒚; 𝜶^{*}) {(𝚿_{k}^{*})}_{j j}] = {(𝚿_{k}^{*})}_{j j} 𝔼_{𝜶^{*}} [t_{k} (𝒚; 𝜶^{*})],$

where the expectation is with respect to

$𝒚 \sim f_{clust} (\cdot; 𝜶^{*}) .$

By Assumption 8, $𝔼_{𝜶^{*}} [t_{k} (𝒚; 𝜶^{*})] = π_{k}^{*} \geq π_{\min}$ and ${(𝚿_{k}^{*})}_{j j} \geq θ_{\min}$ (since $𝚿_{k}^{*} ≻ 0$ and $e_{j}^{⊤} 𝚿_{k}^{*} e_{j} \geq λ_{\min} (𝚿_{k}^{*})$ ), hence

$H_{k j}^{eff} (𝜶^{*}) \geq π_{\min} θ_{\min} > 0 .$
•

KKT remainder for mean coordinates.

$R_{k j} (\hat{𝜶} (λ), 𝜶^{*}, λ) = \partial_{μ_{k j}} ℓ_{N} (\hat{𝜶} (λ)) - \partial_{μ_{k j}} ℓ_{N} (𝜶^{*}) - H_{k j}^{eff} (𝜶^{*}) ({\hat{μ}}_{k j} (λ) - μ_{k j}^{*}) .$

We assume that, with high probability and for $λ$ in the regime $λ ≍ \sqrt{(\ln d_{α}) / N}$ ,

$| R_{k j} (\hat{𝜶} (λ), 𝜶^{*}, λ) | \leq E_{KKT_rem} (λ),$

where $E_{KKT_rem} (λ)$ depends on $‖ \hat{𝜶} (λ) - 𝜶^{*} ‖$ (cf. Lemma 3) and on bounds for Hessian/third derivatives (from Assumptions 7-12). In particular, when ${‖ 𝚫 ‖}_{2}$ is of order $\sqrt{s_{0} \ln d_{α} / N}$ , we take

$E_{KKT_rem} (λ) \leq C_{rem} λ_{noise}$

for some constant $C_{rem} \geq 0$ over the relevant range of $λ$ .
•

Noise-screening threshold.

$Λ_{N}^{*} = (1 + C_{rem} + ϵ_{N}) λ_{noise},$

with a small margin $ϵ_{N} > 0$ .
•

Signal-preservation threshold. For a coordinate $j$ and some $k_{0}$ attaining (or meeting) a signal condition for $| μ_{k_{0} j}^{*} |$ ,

$Λ_{S}^{*} (j; λ) = H_{k_{0} j}^{eff} (𝜶^{*}) | μ_{k_{0} j}^{*} | - (1 + ϵ_{S}) λ_{noise} - (1 + ϵ_{S}) E_{KKT_rem} (λ),$

where $ϵ_{S} > 0$ is a fixed margin. Choosing $λ$ so that $Λ_{S}^{*} (j; λ) > 0$ ensures the signal at $(k_{0}, j)$ persists (i.e., ${\hat{μ}}_{k_{0} j} (λ) \neq 0$ ) under the KKT inequalities.

Assumption 15 (True Relevant/Irrelevant Variables).

Let $𝕊_{𝛍}^{*}$ denote the set of indices $j$ such that at least one component mean has a nonzero $j$ -th entry, i.e., $j \in 𝕊_{𝛍}^{*}$ iff $\exists k \in {1, \dots, K}$ with $μ_{k j}^{*} \neq 0$ . For $j \notin 𝕊_{𝛍}^{*}$ , we have $μ_{k j}^{*} = 0$ for all $k$ .

Assumption 16 (Minimum Signal Strength).

For each $j \in 𝕊_{𝛍}^{*}$ , there exists $k_{0} \in {1, \dots, K}$ such that, for all $λ$ up to some detection level $λ_{detect _ upper}$ ,

H_{k_{0} ​ j}^{eff} ​ (𝜶^{*}) ​ | μ_{k_{0} ​ j}^{*} | \geq λ + (1 + ϵ_{S}) ​ λ_{noise} + (1 + ϵ_{S}) ​ E_{KKT ​_​ rem} ​ (λ),

(30)

with $ϵ_{S} > 0$ fixed. In particular, Equation 30 implies $Λ_{S}^{*} (j; λ) > λ$ . Moreover, we assume the separation

\min_{j \in 𝕊_{𝝁}^{*}} λ_{j}^{†} > Λ_{N}^{*}, where ​ λ_{j}^{†} := inf {λ^{'} > 0 : λ^{'} = Λ_{S}^{*} ​ (j; λ^{'})},

(31)

i.e., the smallest regularization at which the $j$ -th signal would vanish strictly exceeds the noise-screening threshold $Λ_{N}^{*}$ .

Assumption 17 (Regularization Grid).

The grid $𝒢_{λ} = {λ^{(1)}, \dots, λ^{(M_{G})}}$ covers a neighborhood of the noise threshold $Λ_{N}^{*}$ and extends up to

\min_{j \in 𝕊_{𝝁}^{*}} λ_{j}^{†} with ​ λ_{j}^{†} ​ as in Equation 31 .

Assume $ρ$ is either tied to $λ$ (e.g., $ρ ≍ λ$ ) or fixed so that its effect is absorbed by constants in the bounds.

Remark 5.

The remainder $R_{k j} (\hat{𝛂}, 𝛂^{*}, λ)$ in Definition 4 captures the deviation of the mean-coordinate KKT equation from its linearized form. It aggregates: (i) off-diagonal Fisher blocks acting on other coordinates of $𝚫$ , (ii) empirical-population curvature fluctuations $(𝐇_{N} - 𝐈)$ , and (iii) population curvature drift $𝐈 (\tilde{𝛂}) - 𝐈 (𝛂^{*})$ from evaluating at $\tilde{𝛂}$ . Under the standing assumptions (smoothness/compactness: Assumptions 7-8, Hessian concentration: Assumption 12) and the $L_{2}$ -error bound from Lemma 3, one obtains the bound

| R_{k ​ j} ​ (\hat{𝜶} ​ (λ), 𝜶^{*}, λ) | \leq E_{KKT ​_​ rem} ​ (λ) \leq C_{rem} ​ λ_{noise}

on the high-probability event $ℰ_{grad} \cap ℰ_{RSC}$ , provided the sparsity/sample-size regime ensures $C_{H} \sqrt{\frac{s_{0} \ln d_{α}}{N}} + L_{I} {‖ \hat{𝛂} (λ) - 𝛂^{*} ‖}_{2} ≲ 1$ (e.g., $s_{0} ≲ \sqrt{N / \ln d_{α}}$ ). A crude but sufficient bound for individual coordinates is ${‖ 𝚫 ‖}_{\infty} \leq {‖ 𝚫 ‖}_{2} = O (\sqrt{s_{0} \ln d_{α} / N})$ by Lemma 3. Assumption 16 then ensures the effective signal $H_{k_{0} j}^{eff} (𝛂^{*}) | μ_{k_{0} j}^{*} |$ dominates both the regularization level and the stochastic/remainder terms, yielding persistence of true signals and suppression of noise along the regularization path.

Lemma 5 (Ranking Consistency for Mean Parameters).

Under Assumptions 13, 15, 16, and 17, and on the high-probability event $ℰ_{grad} \cap ℰ_{RSC}$ from Assumption 14, let the grid $𝒢_{λ}$ contain points distributed across $[0, λ_{grid_max}]$ with

λ_{grid_max} > \min_{j \in 𝕊_{𝝁}^{*}} λ_{j}^{†}, λ_{j}^{†} := inf {λ^{'} > 0 : λ^{'} = Λ_{S}^{*} ​ (j; λ^{'})} .

Then, with probability at least

P_{rank} \geq 1 - M_{G} \cdot D \cdot δ_{N}, where ​ δ_{N} := C_{1}^{grad} ​ d_{α}^{- C_{2}^{grad}} + c_{1}^{H} ​ d_{α}^{- c_{2}^{H}},

$M_{G} = | 𝒢_{λ} |$ , and $D$ denotes the number of penalized mean coordinates per variable (e.g., $D = K$ ), the following hold:

1.

Relevant Variables. For each $j \in 𝕊_{𝝁}^{*}$ ,

$𝒪_{K} (j) \geq N_{signal} (j) := # {λ \in 𝒢_{λ} : λ < Λ_{S}^{*} (j; λ)} .$

Let $η_{R} := \min_{j \in 𝕊_{𝝁}^{*}} N_{signal} (j)$ . By the separation Equation 31,

$η_{R} > # {λ \in 𝒢_{λ} : λ \leq Λ_{N}^{*}} .$
2.

Irrelevant Variables. For each $j \notin 𝕊_{𝝁}^{*}$ ,

$𝒪_{K} (j) \leq N_{noise} := # {λ \in 𝒢_{λ} : λ \leq Λ_{N}^{*}} .$

Consequently, $η_{R} > N_{noise}$ , yielding a strict separation in ranking scores between relevant and irrelevant variables with probability at least $P_{rank}$ .

Proof of Lemma 5.

Work on the high-probability event

ℰ_{rank} := ℰ_{grad} \cap ℰ_{RSC}

(on which the uniform gradient bound $| S_{k j}^{*} | \leq λ_{noise}$ and the KKT-remainder bound $| R_{k j} (\hat{𝜶} (λ), 𝜶^{*}, λ) | \leq E_{KKT_rem} (λ)$ hold simultaneously for all penalized mean coordinates $(k, j)$ and all $λ \in 𝒢_{λ}$ ). By Assumptions 13-17 and the union bound,

ℙ ​ (ℰ_{rank}) \geq 1 - M_{G} \cdot D \cdot δ_{N}, δ_{N} := C_{1}^{grad} ​ d_{α}^{- C_{2}^{grad}} + c_{1}^{H} ​ d_{α}^{- c_{2}^{H}} .

The ranking score for variable $j$ is

𝒪_{K} (j) = \sum_{λ \in 𝒢_{λ}} ℐ (\exists k \in {1, \dots, K} : {\hat{μ}}_{k ​ j} (λ) \neq 0) .

KKT expansion. For a penalized mean coordinate $(k, j)$ , the KKT condition reads

0 = \nabla_{μ_{k ​ j}} ℓ_{N} ​ (\hat{𝜶} ​ (λ)) + λ ​ {\hat{ξ}}_{k ​ j}, {\hat{ξ}}_{k ​ j} \in {\begin{matrix} {sign ({\hat{μ}}_{k ​ j} ​ (λ))}, & {\hat{μ}}_{k ​ j} ​ (λ) \neq 0, \\ [- 1, 1], & {\hat{μ}}_{k ​ j} ​ (λ) = 0 . \end{matrix}

A first-order expansion at $𝜶^{*}$ yields

\nabla_{μ_{k ​ j}} ℓ_{N} ​ (\hat{𝜶} ​ (λ)) = \underset{=, : S_{k ​ j}^{*}}{\underset{⏟}{[\nabla_{μ_{k ​ j}} ℓ_{N} ​ (𝜶^{*})]}} + H_{k ​ j}^{eff} ​ (𝜶^{*}) ​ ({\hat{μ}}_{k ​ j} ​ (λ) - μ_{k ​ j}^{*}) + R_{k ​ j} ​ (\hat{𝜶} ​ (λ), 𝜶^{*}, λ),

hence

λ ​ {\hat{ξ}}_{k ​ j} = - S_{k ​ j}^{*} - H_{k ​ j}^{eff} ​ (𝜶^{*}) ​ ({\hat{μ}}_{k ​ j} ​ (λ) - μ_{k ​ j}^{*}) - R_{k ​ j} ​ (\hat{𝜶} ​ (λ), 𝜶^{*}, λ) .

(32)

Relevant variables ( $j \in 𝕊_{μ}^{*}$ ). Fix $j \in 𝕊_{𝝁}^{*}$ . By Assumption 16 there exists $k_{0}$ with $μ_{k_{0} j}^{*} \neq 0$ satisfying the signal condition. Assume, for contradiction, that at a given $λ \in 𝒢_{λ}$ we have ${\hat{μ}}_{k_{0} j} (λ) = 0$ . Then ${\hat{ξ}}_{k_{0} j} \in [- 1, 1]$ and ${\hat{μ}}_{k_{0} j} (λ) - μ_{k_{0} j}^{*} = - μ_{k_{0} j}^{*}$ . Applying Equation 32 and taking absolute values,

λ \geq | H_{k_{0} ​ j}^{eff} ​ (𝜶^{*}) ​ μ_{k_{0} ​ j}^{*} - S_{k_{0} ​ j}^{*} - R_{k_{0} ​ j} | \geq H_{k_{0} ​ j}^{eff} ​ (𝜶^{*}) ​ | μ_{k_{0} ​ j}^{*} | - | S_{k_{0} ​ j}^{*} | - | R_{k_{0} ​ j} | .

On $ℰ_{rank}$ ,

λ \geq H_{k_{0} ​ j}^{eff} ​ (𝜶^{*}) ​ | μ_{k_{0} ​ j}^{*} | - λ_{noise} - E_{KKT_rem} ​ (λ) .

Therefore, whenever

λ ​ < H_{k_{0} ​ j}^{eff} ​ (𝜶^{*}) | ​ μ_{k_{0} ​ j}^{*} | - λ_{noise} - E_{KKT_rem} ​ (λ),

we must have ${\hat{μ}}_{k_{0} j} (λ) \neq 0$ . Since $Λ_{S}^{*} (j; λ) = H_{k_{0} j}^{eff} (𝜶^{*}) | μ_{k_{0} j}^{*} | - (1 + ϵ_{S}) λ_{noise} - (1 + ϵ_{S}) E_{KKT_rem} (λ)$ is a stricter threshold,

λ < Λ_{S}^{*} ​ (j; λ) ⟹ {\hat{μ}}_{k_{0} ​ j} ​ (λ) \neq 0 .

Hence the count of grid points for which variable $j$ is (at least in one component) active satisfies

𝒪_{K} ​ (j) \geq N_{signal} ​ (j) := # ​ {λ \in 𝒢_{λ} : λ < Λ_{S}^{*} ​ (j; λ)} .

Taking the minimum over $j \in 𝕊_{𝝁}^{*}$ gives $η_{R} := \min_{j \in 𝕊_{𝝁}^{*}} N_{signal} (j)$ .

Irrelevant variables ( $j \notin 𝕊_{μ}^{*}$ ). Here $μ_{k j}^{*} = 0$ for all $k$ . For a given $λ$ , the zero solution ${\hat{μ}}_{k j} (λ) = 0$ is KKT-feasible iff

| \nabla_{μ_{k ​ j}} ℓ_{N} ​ (\hat{𝜶} ​ (λ)) | \leq λ .

Using the expansion with $μ_{k j}^{*} = 0$ and ${\hat{μ}}_{k j} (λ) = 0$ ,

| S_{k ​ j}^{*} + R_{k ​ j} ​ (\hat{𝜶} ​ (λ), 𝜶^{*}, λ) | \leq λ .

On $ℰ_{rank}$ , this is guaranteed whenever

λ \geq λ_{noise} + E_{KKT_rem} ​ (λ) .

By definition of the noise threshold $Λ_{N}^{*} = (1 + ϵ_{N}) λ_{noise} + (1 + ϵ_{N}) {sup}_{λ^{'} \leq Λ_{N}^{*}} E_{KKT_rem} (λ^{'})$ and the monotone/worst-case domination in its definition, any $λ > Λ_{N}^{*}$ satisfies $λ \geq λ_{noise} + E_{KKT_rem} (λ)$ . Therefore, for each $j \notin 𝕊_{𝝁}^{*}$ and all $λ > Λ_{N}^{*}$ , all coordinates ${\hat{μ}}_{k j} (λ)$ equal zero, so the indicator $ℐ (\exists k : {\hat{μ}}_{k j} (λ) \neq 0) = 0$ . Consequently,

𝒪_{K} ​ (j) \leq N_{noise} := # ​ {λ \in 𝒢_{λ} : λ \leq Λ_{N}^{*}} .

Separation and probability. By the separation Assumption Equation 31,

η_{R} > N_{noise},

yielding a strict gap between the scores of relevant and irrelevant variables on $ℰ_{rank}$ . Finally, by the union bound over at most $M_{G}$ grid points and $D$ penalized mean coordinates per variable, we obtain

ℙ ​ (the above conclusions hold for all ​ j) \geq 1 - M_{G} \cdot D \cdot δ_{N},

which is $P_{rank}$ in the statement. This completes the proof. ∎

Theorem 4 (Selection Consistency of the Two-Step SRUW Procedure).

Assume all assumptions in Lemma 3 and Lemma 5, together with Theorem 2 for the final $(K, m, r, ℓ)$ choice, hold. Let $s_{0} = | 𝕊_{0} |$ and $M_{N R} = p - s_{0}$ be the number of non- $𝕊_{0}$ variables. Moreover, suppose the following assumptions hold:

(a)

False negative for $𝕊_{0}$ . For any true relevant variable $j \in 𝕊_{0}$ and any intermediate set ${\hat{𝕊}}_{cur} \subset 𝕊_{0}$ with $j \notin {\hat{𝕊}}_{cur}$ , the probability of incorrectly rejecting $j$ from $𝕊$ by the ${BIC}_{diff}$ criterion is uniformly bounded:

$sup_{{\hat{𝕊}}_{cur} \subset 𝕊_{0}} ℙ ({BIC}_{diff} (j ∣ {\hat{𝕊}}_{cur}) \leq 0) \leq p_{S} (N),$

where $s_{0} \cdot p_{S} (N) \leq ϵ_{S, F N} (N)$ and $ϵ_{S, F N} (N) = o_{N} (1)$ .
(b)

False positive for non- $𝕊_{0}$ . For any true non-relevant variable $j \notin 𝕊_{0}$ (i.e., $j \in 𝕌_{0} \cup 𝕎_{0}$ ), given that $𝕊_{0}$ has been correctly identified (i.e., ${\hat{𝕊}}_{cur} = 𝕊_{0}$ ), the probability of ${BIC}_{diff} (j ∣ 𝕊_{0}) > 0$ is bounded:

$ℙ ({BIC}_{diff} (j ∣ 𝕊_{0}) > 0) \leq p_{N} (N),$

where $p_{N} (N) \to 0$ as $N \to \infty$ . For BIC, typically $p_{N} (N) = 𝒪 (N^{- γ_{B}})$ for some $γ_{B} > 0$ (e.g., $γ_{B} \geq Δ ν_{\min} / 2$ where $Δ ν_{\min} \geq 1$ is the minimum parameter-penalty gap).
(c)
Consistency of BIC-penalized regressions for $\hat{ℝ} [j ∣ 𝕊_{0}]$ .:
- •
  
  For $j \in 𝕎_{0}$ , $ℙ (\hat{ℝ} [j ∣ 𝕊_{0}] = \emptyset) \geq 1 - p_{reg} (N)$ .
- •
  
  For $j \in 𝕌_{0}$ , $ℙ (\hat{ℝ} [j ∣ 𝕊_{0}] = ℝ_{0} (j) \neq \emptyset) \geq 1 - p_{reg} (N)$ .
where $(w_{0} + u_{0}) p_{reg} (N) = o_{N} (1)$ .

Let the stopping-rule parameter for selecting $\hat{𝕊}$ (and $\hat{𝕎}$ ) be $c$ , understood so that once ${\hat{𝕊}}_{cur} = 𝕊_{0}$ holds, each $j \notin 𝕊_{0}$ is tested at most $c$ times by the ${BIC}_{diff} (\cdot ∣ 𝕊_{0})$ screening before termination.

For a desired tolerance $ϵ_{S, F P} > 0$ on the probability of including any false positive in $\hat{𝕊}$ , it suffices to choose $c$ so that

M_{N ​ R} ​ (1 - {(1 - p_{N} ​ (N))}^{c}) \leq ϵ_{S, F ​ P} ⟺ c \leq \frac{\ln (1 - ϵ_{S, F ​ P} / M_{N ​ R})}{\ln (1 - p_{N} ​ (N))} .

(33)

For small $p_{N} (N)$ , this is well-approximated by

c ≲ \frac{ϵ_{S, F ​ P}}{M_{N ​ R} ​ p_{N} ​ (N)} .

In particular, if $p_{N} (N)$ decays polynomially in $N$ (e.g., $N^{- γ_{B}}$ with $γ_{B} \geq 1 / 2$ ) and $M_{N R} p_{N} (N) \to 0$ , then any fixed $c_{fixed} \geq 1$ (e.g., $c_{fixed} = 1$ or $3$ ) ensures

ℙ ​ (any false positive in ​ \hat{𝕊}) \leq M_{N ​ R} ​ (1 - {(1 - p_{N} ​ (N))}^{c_{fixed}}) \leq M_{N ​ R} ​ c_{fixed} ​ p_{N} ​ (N) \to 0 .

Then, under (a)-(c) and with $c$ chosen according to Equation 33 (or with $c = c_{fixed}$ such that $M_{N R} c_{fixed} p_{N} (N) \to 0$ ), the two-step SRUW procedure recovers the true model structure $(K_{0}, m_{0}, r_{0}, ℓ_{0}, 𝕍_{0})$ with probability $ℙ (Success) \to 1$ as $N \to \infty$ .

Proof.

We show each stage succeeds with high probability and then combine them.

Conditioning on ranking accuracy. By Lemma 5, with probability at least $1 - P_{fail,rank_total}$ , the ranking event $ℰ_{rank}$ holds: all $s_{0}$ relevant variables in $𝕊_{0}$ appear before any pure-noise $𝕎_{0}$ variables, and $𝕌_{0}$ appear after $𝕊_{0}$ . We condition on $ℰ_{rank}$ for the remainder of the argument.

No false negatives in $\hat{𝕊}$ . Fix $j \in 𝕊_{0}$ and any ${\hat{𝕊}}_{cur} \subset 𝕊_{0}$ not containing $j$ . By Assumption (a),

ℙ ​ ({BIC}_{diff} ​ (j ∣ {\hat{𝕊}}_{cur}) \leq 0) \leq p_{S} ​ (N) .

By a union bound over the $s_{0}$ true variables,

ℙ (𝕊_{0} ⊈ \hat{𝕊} | ℰ_{rank}) \leq s_{0} p_{S} (N) = : ϵ_{S, F ​ N} (N) \to 0 .

Let $ℰ_{S-noFN}$ be the event $𝕊_{0} \subseteq \hat{𝕊}$ . Then $ℙ (ℰ_{S-noFN} ∣ ℰ_{rank}) \geq 1 - ϵ_{S, F N} (N)$ .

No false positives in $\hat{𝕊}$ . On $ℰ_{rank} \cap ℰ_{S-noFN}$ we have ${\hat{𝕊}}_{cur} = 𝕊_{0}$ . For any $j \notin 𝕊_{0}$ , Assumption (b) gives

ℙ ({BIC}_{diff} (j ∣ 𝕊_{0}) > 0) \leq p_{N} (N) = : q .

Let $ℰ_{S-noFP}$ be the event that no non-relevant variable is ever added to $\hat{𝕊}$ after $𝕊_{0}$ is reached.

(A) Distribution-free bound. For any fixed set $𝒥$ of candidates examined of any size,

ℙ (\exists j \in 𝒥 : {BIC}_{diff} (j ∣ 𝕊_{0}) > 0) \leq | 𝒥 | q

by a union bound. In particular,

ℙ ​ (ℰ_{S-noFP}^{c} | ℰ_{rank} \cap ℰ_{S-noFN}) \leq M_{N ​ R} ​ q,

which tends to $0$ if $M_{N R} p_{N} (N) \to 0$ . This bound requires no independence and is always valid.

(B) Sharper bound under $c$ -run termination. Assume the screening step proceeds through non-relevant candidates until it observes $c$ consecutive rejections, and that the $c$ decisions in such a run are (asymptotically) independent or, more generally, satisfy

ℙ ​ (⋂_{t = 1}^{c} {{BIC}_{diff} ​ (j_{t} ∣ 𝕊_{0}) \leq 0}) \geq {(1 - q)}^{c}

for any $c$ distinct non-relevant candidates $(j_{1}, \dots, j_{c})$ . Then the probability that a given $c$ -block fails (i.e., contains at least one FP) is

ℙ ​ (block error) \leq 1 - {(1 - q)}^{c} .

Partition the $M_{N R}$ non-relevant indices into $⌊ M_{N R} / c ⌋$ disjoint blocks of size $c$ (discarding a remainder if needed). By a union bound over blocks,

ℙ ​ (ℰ_{S-noFP}^{c} | ℰ_{rank} \cap ℰ_{S-noFN}) \leq \frac{M_{N ​ R}}{c} ​ (1 - {(1 - q)}^{c}) \leq M_{N ​ R} ​ (1 - {(1 - q)}^{c}) .

Therefore, enforcing

M_{N ​ R} ​ (1 - {(1 - p_{N} ​ (N))}^{c}) \leq ϵ_{S, F ​ P} ⟺ c \leq \frac{\ln (1 - ϵ_{S, F ​ P} / M_{N ​ R})}{\ln (1 - p_{N} ​ (N))}

yields $ℙ (ℰ_{S-noFP}^{c} ∣ ℰ_{rank} \cap ℰ_{S-noFN}) \leq ϵ_{S, F P}$ . For small $p_{N} (N)$ , $1 - {(1 - p_{N} (N))}^{c} \sim c p_{N} (N)$ , so $c ≲ ϵ_{S, F P} / (M_{N R} p_{N} (N))$ .

Combining (A)-(B): whenever the independence/decoupling condition for runs holds, we may use the sharper $(1 - {(1 - q)}^{c})$ design in the theorem statement; otherwise, the distribution-free guarantee $M_{N R} p_{N} (N)$ is valid (and implies the sharper one whenever $c$ is fixed and $M_{N R} p_{N} (N) \to 0$ ).

Put together $\hat{𝕊} = 𝕊_{0}$ . Let $ℰ_{S}^{*} := ℰ_{S-noFN} \cap ℰ_{S-noFP}$ . Then, with either (A) or (B),

ℙ ​ (\hat{𝕊} = 𝕊_{0} | ℰ_{rank}) \geq 1 - ϵ_{S, F ​ N} ​ (N) - ϵ_{S, F ​ P} ​ (N) \to 1,

for $ϵ_{S, F P} (N) = M_{N R} p_{N} (N)$ in (A), or $ϵ_{S, F P} (N) = M_{N R} (1 - {(1 - p_{N} (N))}^{c})$ in (B).

Consistency of $\hat{𝕎}, \hat{𝕌}, \hat{ℝ}$ . Given $\hat{𝕊} = 𝕊_{0}$ , the reverse scan uses BIC-penalized regressions to decide $\hat{𝕎}$ and $\hat{ℝ}$ . By Assumption (c),

ℙ ​ (\hat{𝕎} = 𝕎_{0} | ℰ_{rank} \cap ℰ_{S}^{*}) \geq 1 - (w_{0} + u_{0}) ​ p_{reg} ​ (N) - ϵ_{W, F ​ P} ​ (N) \to 1,

and

ℙ ​ (\hat{ℝ} = ℝ_{0} | ℰ_{rank} \cap ℰ_{S}^{*} \cap {\hat{𝕎} = 𝕎_{0}}) \geq 1 - u_{0} ​ p_{reg} ​ (N) \to 1 .

Final SRUW choice via BIC. By Theorem 2, the final selection of $(K, m, r, ℓ)$ is consistent, i.e., $ℙ (P_{final_choice}) \to 1$ .

Multiplying the stage-wise success probabilities,

	$ℙ (Overall Success) \geq ℙ (ℰ_{rank}) \cdot ℙ (ℰ_{S}^{*} ∣ ℰ_{rank})$	$\cdot ℙ (\hat{𝕎} = 𝕎_{0} ∣ \dots)$
		$\cdot ℙ (\hat{ℝ} = ℝ_{0} ∣ \dots) \cdot ℙ (P_{final_choice}) \to 1 .$

The slowest decaying term among $P_{fail,rank_total}, ϵ_{S, F N} (N), ϵ_{S, F P} (N), (w_{0} + u_{0}) p_{reg} (N), u_{0} p_{reg} (N)$ governs the overall rate. In particular, under the sharper design (B) with $c$ chosen as above,

ϵ_{S, F ​ P} ​ (N) = M_{N ​ R} ​ (1 - {(1 - p_{N} ​ (N))}^{c}) \leq ϵ_{S, F ​ P},

while the distribution-free design (A) yields $ϵ_{S, F P} (N) = M_{N R} p_{N} (N) \to 0$ whenever $M_{N R} p_{N} (N) \to 0$ . ∎

Theorem 5 (Equivalence of MNARz-SRUW and MAR on Augmented Data for SRUW).

Consider an observation $(𝐲_{n}, 𝐜_{n})$ where $𝐲_{n} = (𝐲_{n}^{S}, 𝐲_{n}^{U}, 𝐲_{n}^{W})$ and $𝐜_{n}$ is its missingness pattern. Let the complete-data likelihood for $𝐲_{n}$ given cluster $z_{n k} = 1$ under the SRUW model $(K, m, r, ℓ, 𝕍)$ be:

f_{SRUW} ​ (𝒚_{n} ∣ z_{n ​ k} = 1; θ_{k}) = f_{k} ​ (𝒚_{n}^{S}; 𝜶_{k}) ​ f_{reg} ​ (𝒚_{n}^{U} ∣ 𝒚_{n}^{R}; 𝜽_{reg}) ​ f_{indep} ​ (𝒚_{n}^{W}; 𝜽_{indep}) .

Assume an MNARz-SRUW mechanism for missing data, where the probability of the missingness pattern $𝐜_{n}$ depends only on the cluster membership $z_{n k}$ :

	$f (𝒄_{n} ∣ 𝒚_{n}, z_{n k} = 1; ψ_{k})$	$= f (𝒄_{n} ∣ z_{n k} = 1; ψ_{k})$		(34)
		$= \prod_{d \in S \cup U \cup W} ρ_{k d}^{c_{n d}} {(1 - ρ_{k d})}^{1 - c_{n d}},$		(34)

where $ρ_{k d} \in (0, 1)$ are components of the missingness parameter $ψ_{k}$ (thus $f (𝐜_{n} ∣ z_{n k} = 1; ψ_{k})$ does not depend on $𝐲_{n}$ ). The observed data under this MNARz-SRUW model is $(𝐲_{n}^{o}, 𝐜_{n})$ , and its likelihood is:

L_{MNARz-SRUW} ​ (𝒚_{n}^{o}, 𝒄_{n}; 𝜽, 𝝍) = \int \sum_{k = 1}^{K} π_{k} ​ f_{SRUW} ​ (𝒚_{n} ∣ z_{n ​ k} = 1; θ_{k}) ​ f ​ (𝒄_{n} ∣ z_{n ​ k} = 1; ψ_{k}) ​ d ​ 𝒚_{n}^{m} .

(35)

Now consider the augmented observed vector ${\tilde{𝐲}}_{n}^{o} = (𝐲_{n}^{o}, 𝐜_{n})$ . Assume ${\tilde{𝐲}}_{n}^{o}$ is i.i.d. from a mixture model in which the conditional law of the missing part $𝐲_{n}^{m}$ is MAR with respect to ${\tilde{𝐲}}_{n}^{o}$ (i.e., the missingness mechanism does not depend on $𝐲_{n}^{m}$ given $(𝐲_{n}^{o}, 𝐜_{n})$ ). Define the **augmented observed-data** likelihood as

{\tilde{f}}_{MAR} ({\tilde{𝒚}}_{n}^{o}; 𝜽, 𝝍) = \sum_{k = 1}^{K} π_{k} (\int f_{SRUW} (𝒚_{n}^{o}, 𝒚_{n}^{m} ∣ z_{n ​ k} = 1; θ_{k}) d 𝒚_{n}^{m}) f (𝒄_{n} ∣ z_{n ​ k} = 1; ψ_{k}),

(36)

where $f_{SRUW} (\cdot ∣ z_{n k} = 1; θ_{k})$ is the same component density as above.

Then, for fixed parameters $(𝛉, 𝛙)$ , the observed-data likelihood under MNARz-SRUW for $(𝐲_{n}^{o}, 𝐜_{n})$ is identical to the likelihood of the augmented observation ${\tilde{𝐲}}_{n}^{o} = (𝐲_{n}^{o}, 𝐜_{n})$ under the MAR interpretation:

L_{MNARz-SRUW} ​ (𝒚_{n}^{o}, 𝒄_{n}; 𝜽, 𝝍) = {\tilde{f}}_{MAR} ​ ({\tilde{𝒚}}_{n}^{o}; 𝜽, 𝝍) .

Proof of Theorem 5.

Starting from Equation 35, by the MNARz assumption $f (𝒄_{n} ∣ 𝒚_{n}, z_{n k} = 1; ψ_{k}) = f (𝒄_{n} ∣ z_{n k} = 1; ψ_{k})$ is independent of $𝒚_{n}^{m}$ , hence

	$L_{MNARz-SRUW} (𝒚_{n}^{o}, 𝒄_{n}; 𝜽, 𝝍)$	$= \int \sum_{k = 1}^{K} π_{k} f_{SRUW} (𝒚_{n}^{o}, 𝒚_{n}^{m} ∣ z_{n k} = 1; θ_{k}) f (𝒄_{n} ∣ z_{n k} = 1; ψ_{k}) d 𝒚_{n}^{m}$
		$= \sum_{k = 1}^{K} π_{k} f (𝒄_{n} ∣ z_{n k} = 1; ψ_{k}) (\int f_{SRUW} (𝒚_{n}^{o}, 𝒚_{n}^{m} ∣ z_{n k} = 1; θ_{k}) d 𝒚_{n}^{m})$
		$= \sum_{k = 1}^{K} π_{k} f_{k} (𝒚_{n}^{o} ∣ z_{n k} = 1; θ_{k}) f (𝒄_{n} ∣ z_{n k} = 1; ψ_{k}),$

where we set $f_{k} (𝒚_{n}^{o} ∣ z_{n k} = 1; θ_{k}) := \int f_{SRUW} (𝒚_{n}^{o}, 𝒚_{n}^{m} ∣ z_{n k} = 1; θ_{k}) d 𝒚_{n}^{m}$ . Comparing with Equation 36 yields

L_{MNARz-SRUW} ​ (𝒚_{n}^{o}, 𝒄_{n}; 𝜽, 𝝍) = {\tilde{f}}_{MAR} ​ ({\tilde{𝒚}}_{n}^{o}; 𝜽, 𝝍),

as claimed. ∎

By Theorem 5, the observed-data likelihood under MNARz-SRUW equals the augmented observed-data likelihood under MAR for every $(𝜽, 𝝍)$ and each observation; hence, the full-sample likelihoods coincide pointwise in $(𝜽, 𝝍)$ . The following corollary shows the resulting estimator-level equivalence.

Corollary 1 (Estimator equivalence under augmentation).

Under the conditions of Theorem 5, for any fixed $(𝛉, 𝛙)$ and for each observation,

L_{MNARz-SRUW} ​ (𝒚_{n}^{o}, 𝒄_{n}; 𝜽, 𝝍) = {\tilde{f}}_{MAR} ​ ({\tilde{𝒚}}_{n}^{o}; 𝜽, 𝝍) .

Hence the full-sample observed-data likelihoods coincide, and therefore the sets of maximum likelihood estimators for $(𝛉, 𝛙)$ under MNARz-SRUW based on $(𝐲^{o}, 𝐜)$ and under MAR on the augmented data ${\tilde{𝐲}}^{o} = (𝐲^{o}, 𝐜)$ are identical (up to label switching). Moreover, if the same priors on $(𝛉, 𝛙)$ are used in both formulations, the resulting Bayesian posteriors coincide.

Proof of Corollary 1.

By Theorem 5, for every observation $n$ and every fixed $(𝜽, 𝝍)$ ,

L_{MNARz-SRUW} ​ (𝒚_{n}^{o}, 𝒄_{n}; 𝜽, 𝝍) = {\tilde{f}}_{MAR} ​ ({\tilde{𝒚}}_{n}^{o}; 𝜽, 𝝍), {\tilde{𝒚}}_{n}^{o} = (𝒚_{n}^{o}, 𝒄_{n}) .

Let $𝒟 = {(𝒚_{n}^{o}, 𝒄_{n})}_{n = 1}^{N}$ denote the sample, assumed i.i.d. under the mixture model as in the theorem. The full-sample observed-data likelihoods are then

	$L_{MNARz-SRUW} (𝒟; 𝜽, 𝝍)$	$= \prod_{n = 1}^{N} L_{MNARz-SRUW} (𝒚_{n}^{o}, 𝒄_{n}; 𝜽, 𝝍)$
	${\tilde{L}}_{MAR} (𝒟; 𝜽, 𝝍)$	$= \prod_{n = 1}^{N} {\tilde{f}}_{MAR} ({\tilde{𝒚}}_{n}^{o}; 𝜽, 𝝍) .$

By pointwise equality for each factor, we obtain the full-sample equality

L_{MNARz-SRUW} ​ (𝒟; 𝜽, 𝝍) = {\tilde{L}}_{MAR} ​ (𝒟; 𝜽, 𝝍) for all ​ (𝜽, 𝝍) .

Consequently, the sets of maximum likelihood estimators coincide:

\arg \max_{(𝜽, 𝝍)} L_{MNARz-SRUW} ​ (𝒟; 𝜽, 𝝍) = \arg \max_{(𝜽, 𝝍)} {\tilde{L}}_{MAR} ​ (𝒟; 𝜽, 𝝍),

up to the usual permutations of mixture component labels (label switching), since both likelihoods are invariant under relabelings.

For the Bayesian statement, let $Π$ be a common prior on $(𝜽, 𝝍)$ admitting a density $π (𝜽, 𝝍)$ with respect to a common dominating measure. The posteriors are

	$π_{MNARz} (𝜽, 𝝍 ∣ 𝒟)$	$\propto π (𝜽, 𝝍) L_{MNARz-SRUW} (𝒟; 𝜽, 𝝍)$
	$π_{MAR} (𝜽, 𝝍 ∣ 𝒟)$	$\propto π (𝜽, 𝝍) {\tilde{L}}_{MAR} (𝒟; 𝜽, 𝝍) .$

Since the likelihoods coincide pointwise in $(𝜽, 𝝍)$ , the unnormalized posteriors agree, hence their normalizing constants (integrals over the same parameter space) are also equal. Therefore

π_{MNARz} ​ (𝜽, 𝝍 ∣ 𝒟) = π_{MAR} ​ (𝜽, 𝝍 ∣ 𝒟),

again, modulo label switching. ∎

Remark 6.

In the MAR interpretation of the augmented data ${\tilde{𝐲}}_{n}^{o}$ , the components $𝐲_{n}^{o}$ have density $f_{k} (𝐲_{n}^{o} ∣ z_{n k} = 1; θ_{k})$ in cluster $k$ , and the components $𝐜_{n}$ (which are fully observed within ${\tilde{𝐲}}_{n}^{o}$ ) have density $f (𝐜_{n} ∣ z_{n k} = 1; ψ_{k})$ in cluster $k$ . The MAR assumption applies to $𝐲_{n}^{m}$ , meaning its missingness mechanism, given $𝐲_{n}^{o}$ and $𝐜_{n}$ , does not depend on $𝐲_{n}^{m}$ itself. The likelihood of ${\tilde{𝐲}}_{n}^{o}$ is formed by integrating out $𝐲_{n}^{m}$ from $f (𝐲_{n}, 𝐜_{n} ∣ z_{n k} = 1)$ , which leads to the product $f_{k} (𝐲_{n}^{o} ∣ z_{n k} = 1; θ_{k}) f (𝐜_{n} ∣ z_{n k} = 1; ψ_{k})$ for each cluster $k$ .

Appendix E Detailed EM Algorithms for the Two-Stage Procedure

This section gives complete EM derivations for both stages. In Stage A (ranking), we run a penalized GMM on the imputed and standardized data $\bar{𝒀} = std (\tilde{𝒀})$ , where $\tilde{𝒀}$ is the single-imputed matrix defined in Algorithm 1 which is used for ranking only. In Stage B (role assignment), we fit the unpenalized SRUW model under MNARz by exploiting the equivalence to MAR on the augmented data $(𝒀, 𝑪)$ . Throughout, $𝚿_{k} = 𝚺_{k}^{- 1}$ , the SRUW partition is $V = (𝕊, ℝ, 𝕌, 𝕎)$ , and $t_{n k}$ denotes EM responsibilities. We write $ℓ (\cdot)$ for (penalized) log-likelihoods and $Q (\cdot; \cdot)$ for EM $Q$ -functions.

Stage A (ranking): Penalized EM for adaptive-regularized GMM

Penalized objective.

Given ${\bar{𝒚}}_{1}, \dots, {\bar{𝒚}}_{N} \in ℝ^{D}$ , we maximize the penalized observed-data log-likelihood

ℓ_{pen} ​ (𝜶) = \sum_{n = 1}^{N} \log [\sum_{k = 1}^{K} π_{k} ​ ϕ ​ ({\bar{𝒚}}_{n} ∣ 𝝁_{k}, 𝚺_{k})] - λ ​ \sum_{k = 1}^{K} {‖ 𝝁_{k} ‖}_{1} - ρ ​ \sum_{k = 1}^{K} \sum_{i \neq j} 𝑷_{k, i ​ j} ​ | 𝚿_{k, i ​ j} | .

(37)

Assumption: For each EM run at a fixed $(λ, ρ)$ , the weights $𝑷_{k}$ are computed from the warm start and then held fixed.³³3If $𝑷_{k}$ is updated during EM, one must add a majorization step to preserve monotonicity.

At iteration $t$ , with responsibilities $t_{n k}^{(t)}$ , the penalized $Q$ -function is

	$Q_{pen} (𝜶; 𝜶^{(t - 1)})$	$= \sum_{n = 1}^{N} \sum_{k = 1}^{K} t_{n k}^{(t)} {\log π_{k} - \frac{1}{2} \log \| 𝚺_{k} \| - \frac{1}{2} {({\bar{𝒚}}_{n} - 𝝁_{k})}^{⊤} 𝚿_{k} ({\bar{𝒚}}_{n} - 𝝁_{k})}$
		$- λ \sum_{k = 1}^{K} {‖ 𝝁_{k} ‖}_{1} - ρ \sum_{k = 1}^{K} \sum_{i \neq j} 𝑷_{k, i j} \| 𝚿_{k, i j} \| .$		(38)

E-step.

t_{n ​ k}^{(t)} = \frac{π_{k}^{(t - 1)} ​ ϕ ​ ({\bar{𝒚}}_{n} ∣ 𝝁_{k}^{(t - 1)}, 𝚺_{k}^{(t - 1)})}{\sum_{ℓ = 1}^{K} π_{ℓ}^{(t - 1)} ​ ϕ ​ ({\bar{𝒚}}_{n} ∣ 𝝁_{ℓ}^{(t - 1)}, 𝚺_{ℓ}^{(t - 1)})}, n_{k}^{(t)} = \sum_{n = 1}^{N} t_{n ​ k}^{(t)} .

(39)

M-step: mixing weights.

$π_{k}^{(t)} = n_{k}^{(t)} / N .$

M-step: means $𝝁_{k}$ with $ℓ_{1}$ penalty.

Given $𝚿_{k}^{(t - 1)}$ , the subproblem in $𝝁_{k}$ is convex. Let

{\bar{𝒎}}_{k}^{(t)} := \frac{1}{n_{k}^{(t)}} ​ \sum_{n = 1}^{N} t_{n ​ k}^{(t)} ​ {\bar{𝒚}}_{n} .

Then the gradient of the smooth part is

\nabla_{𝝁_{k}} [\frac{1}{2} ​ \sum_{n = 1}^{N} t_{n ​ k}^{(t)} ​ {({\bar{𝒚}}_{n} - 𝝁_{k})}^{⊤} ​ 𝚿_{k}^{(t - 1)} ​ ({\bar{𝒚}}_{n} - 𝝁_{k})] = n_{k}^{(t)} ​ 𝚿_{k}^{(t - 1)} ​ (𝝁_{k} - {\bar{𝒎}}_{k}^{(t)}) .

The KKT optimality for coordinate $j$ is

n_{k}^{(t)} ​ {(𝚿_{k}^{(t - 1)})}_{j ​ j} ​ μ_{k ​ j} + n_{k}^{(t)} ​ \sum_{v \neq j} {(𝚿_{k}^{(t - 1)})}_{j ​ v} ​ μ_{k ​ v} - n_{k}^{(t)} ​ {(𝚿_{k}^{(t - 1)} ​ {\bar{𝒎}}_{k}^{(t)})}_{j} \in λ ​ \partial | μ_{k ​ j} | .

A coordinate-descent update is

μ_{k ​ j} \leftarrow \frac{1}{n_{k}^{(t)} ​ {(𝚿_{k}^{(t - 1)})}_{j ​ j}} ​ 𝒮_{λ} ​ (n_{k}^{(t)} ​ {(𝚿_{k}^{(t - 1)} ​ {\bar{𝒎}}_{k}^{(t)})}_{j} - n_{k}^{(t)} ​ \sum_{v \neq j} {(𝚿_{k}^{(t - 1)})}_{j ​ v} ​ μ_{k ​ v}),

where $𝒮_{λ} (u) = sign (u) \max {| u | - λ, 0}$ .

M-step: precisions $𝚿_{k}$ via weighted graphical lasso.

With $𝝁_{k}^{(t)}$ fixed, define the responsibility-weighted covariance

𝑺_{k}^{(t)} = \frac{1}{n_{k}^{(t)}} ​ \sum_{n = 1}^{N} t_{n ​ k}^{(t)} ​ ({\bar{𝒚}}_{n} - 𝝁_{k}^{(t)}) ​ {({\bar{𝒚}}_{n} - 𝝁_{k}^{(t)})}^{⊤} .

Then

𝚿_{k}^{(t)} \in \arg \min_{𝚿 ≻ 0} {- \log ​ det 𝚿 + tr ​ (𝑺_{k}^{(t)} ​ 𝚿) + \frac{2 ​ ρ}{n_{k}^{(t)}} ​ \sum_{i \neq j} 𝑷_{k, i ​ j} ​ | 𝚿_{i ​ j} |},

with diagonals unpenalized; standard glasso solvers apply.

Stage B (role assignment): EM for SRUW under MNARz

Observed likelihood via augmentation.

Let $𝒟_{MAR} \dot{\cup} 𝒟_{MNAR} = [D]$ . Under MNARz,

ℓ ​ (Θ; 𝒀, 𝑪) = \sum_{n = 1}^{N} \log [\sum_{k = 1}^{K} π_{k} ​ f_{k, MAR}^{o} ​ (𝒚_{n}^{o}; 𝜶_{k}, 𝝃) ​ f_{c}^{MNARz} ​ (𝒄_{n, MNAR}; 𝝍_{k})],

is a standard mixture on the augmented observation $(𝒚_{n}^{o}, 𝒄_{n, MNAR})$ .

Complete-data log-likelihood and $Q$ -function.

With $Θ = (𝝅, {𝜶_{k}}, 𝝃, {𝝍_{k}})$ and latent ${z_{n k}}$ ,

Q ​ (Θ; Θ^{(t - 1)}) = \sum_{n = 1}^{N} \sum_{k = 1}^{K} t_{n ​ k}^{(t)} ​ {\log π_{k} + 𝔼 ​ [\log f_{k} ​ (𝒀_{n}; 𝜶_{k}, 𝝃) ∣ 𝒚_{n}^{o}, z_{n ​ k} = 1] + \log f_{c}^{MNARz} ​ (𝒄_{n, MNAR}; 𝝍_{k})} .

E-step.

t_{n ​ k}^{(t)} = \frac{π_{k}^{(t - 1)} ​ f_{k, MAR}^{o} ​ (𝒚_{n}^{o}; 𝜶_{k}^{(t - 1)}, 𝝃^{(t - 1)}) ​ f_{c}^{MNARz} ​ (𝒄_{n, MNAR}; 𝝍_{k}^{(t - 1)})}{\sum_{ℓ = 1}^{K} π_{ℓ}^{(t - 1)} ​ f_{ℓ, MAR}^{o} ​ (𝒚_{n}^{o}; 𝜶_{ℓ}^{(t - 1)}, 𝝃^{(t - 1)}) ​ f_{c}^{MNARz} ​ (𝒄_{n, MNAR}; 𝝍_{ℓ}^{(t - 1)})} .

M-step: $π_{k}$ and MNARz parameters.

π_{k}^{(t)} = \frac{1}{N} ​ \sum_{n = 1}^{N} t_{n ​ k}^{(t)}, {\hat{ρ}}_{k}^{(t)} = \frac{\sum_{n = 1}^{N} t_{n ​ k}^{(t)} ​ \sum_{d \in 𝒟_{MNAR}} c_{n ​ d}}{\sum_{n = 1}^{N} t_{n ​ k}^{(t)} ​ | 𝒟_{MNAR} |}, 𝝍_{k}^{(t)} = \log \frac{{\hat{ρ}}_{k}^{(t)}}{1 - {\hat{ρ}}_{k}^{(t)}} .

M-step: SRUW data model updates.

Write

f_{k} ​ (𝒚; 𝜶_{k}, 𝝃) = f_{clust} ​ (𝒚^{𝕊}; 𝝁_{k}^{𝕊}, 𝚺_{k}^{𝕊}) ​ f_{reg} ​ (𝒚^{𝕌} ∣ 𝒚^{ℝ}; 𝒂, 𝜷, 𝛀) ​ f_{indep} ​ (𝒚^{𝕎}; 𝜸, 𝚪) .

Let $𝔼_{k} [\cdot ∣ 𝒚_{n}^{o}]$ denote Gaussian conditional expectations under component $k$ . Then the sufficient statistics are the mixture-weighted moments

𝔼 [\cdot ∣ 𝒚_{n}^{o}] = \sum_{k = 1}^{K} t_{n ​ k}^{(t)} 𝔼_{k} [\cdot ∣ 𝒚_{n}^{o}, z_{n ​ k} = 1] .

Cluster block $𝕊$ : for each $k$ , compute

{\bar{𝒚}}_{k}^{𝕊, (t)} = \frac{1}{n_{k}^{(t)}} ​ \sum_{n = 1}^{N} t_{n ​ k}^{(t)} ​ 𝔼_{k} ​ [𝒚_{n}^{𝕊} ∣ 𝒚_{n}^{o}], 𝑺_{k}^{𝕊, (t)} = \frac{1}{n_{k}^{(t)}} ​ \sum_{n = 1}^{N} t_{n ​ k}^{(t)} ​ 𝔼_{k} ​ [𝒚_{n}^{𝕊} ​ {(𝒚_{n}^{𝕊})}^{⊤} ∣ 𝒚_{n}^{o}],

and set $𝝁_{k}^{𝕊, (t)} = {\bar{𝒚}}_{k}^{𝕊, (t)}$ and $𝚺_{k}^{𝕊, (t)} = 𝑺_{k}^{𝕊, (t)} - {\bar{𝒚}}_{k}^{𝕊, (t)} {({\bar{𝒚}}_{k}^{𝕊, (t)})}^{⊤}$ .

Regression block $𝕌 ∣ ℝ$ : let $𝑿_{n} = [1 {(𝒚_{n}^{ℝ})}^{⊤}]$ . Form the mixture-weighted global moments

𝑨 = \sum_{n = 1}^{N} \sum_{k = 1}^{K} t_{n ​ k}^{(t)} ​ 𝔼_{k} ​ [𝑿_{n}^{⊤} ​ 𝑿_{n} ∣ 𝒚_{n}^{o}], 𝑩 = \sum_{n = 1}^{N} \sum_{k = 1}^{K} t_{n ​ k}^{(t)} ​ 𝔼_{k} ​ [𝑿_{n}^{⊤} ​ 𝒚_{n}^{𝕌} ∣ 𝒚_{n}^{o}],

then $[\begin{matrix} 𝒂^{(t)} \\ 𝜷^{(t)} \end{matrix}] = 𝑨^{- 1} 𝑩,$ and

𝛀^{(t)} = \frac{1}{N} ​ \sum_{n = 1}^{N} \sum_{k = 1}^{K} t_{n ​ k}^{(t)} ​ 𝔼_{k} ​ [(𝒚_{n}^{𝕌} - 𝒂^{(t)} - 𝒚_{n}^{ℝ} ​ 𝜷^{(t)}) ​ {(𝒚_{n}^{𝕌} - 𝒂^{(t)} - 𝒚_{n}^{ℝ} ​ 𝜷^{(t)})}^{⊤} | 𝒚_{n}^{o}] .

Independent block $𝕎$ :

𝜸^{(t)} = \frac{1}{N} ​ \sum_{n = 1}^{N} \sum_{k = 1}^{K} t_{n ​ k}^{(t)} ​ 𝔼_{k} ​ [𝒚_{n}^{𝕎} ∣ 𝒚_{n}^{o}], 𝚪^{(t)} = \frac{1}{N} ​ \sum_{n = 1}^{N} \sum_{k = 1}^{K} t_{n ​ k}^{(t)} ​ 𝔼_{k} ​ [𝒚_{n}^{𝕎} ​ {(𝒚_{n}^{𝕎})}^{⊤} ∣ 𝒚_{n}^{o}] - 𝜸^{(t)} ​ {(𝜸^{(t)})}^{⊤} .

Numerical stability.

In Stage A, we use warm starts and pathwise $(λ, ρ)$ ; diagonals unpenalized and enforce $𝚿_{k} ≻ 0$ . In Stage B, if some $n_{k}^{(t)}$ is tiny, we add a small ridge to $𝚺_{k}^{𝕊}$ or merge/discard components per BIC.

Appendix F Additional Experiments

F.1 Metric Details

We outline some metrics that we use to evaluate the clustering accuracy and imputation error, which are summarized below:

Table 3: Summary of evaluation metrics for clustering and imputation quality

Imputation error
Name	Formula	Range
NRMSE	$\frac{\sqrt{\frac{1}{N} \sum_{i = 1}^{N} {(y_{i} - {\hat{y}}_{i})}^{2}}}{σ}$	$[0, 1]$
WNRMSE	$\frac{\sum_{c = 1}^{C} ({NRMSE}_{c} \times w_{c})}{\sum_{c = 1}^{C} w_{c}}$	$[0, 1]$
Clustering similarity
ARI	$\frac{RI - 𝔼 [RI]}{\max (RI) - 𝔼 [RI]}$	$[0, 1]$
Composite
CIIE	$α \times (1 - NRMSE) + 𝜷 \times Similarity Score$	$[0, 1]$

F.2 Assessing Integrated MNARz Handling

We conduct an additional experiment to showcase the performance of handling MNAR pattern within the variable selection framework of which the setups are similar to [62]. Datasets were simulated with $n = 100$ observations, $K = 3$ true clusters with proportions $𝝅 = (0.5, 0.25, 0.25)$ ), and $D = 6, 9$ variables. True cluster memberships $𝐙$ were drawn according to $𝝅$ . The complete data $𝒀$ was generated as $Y_{n d} = \sum_{k^{'} = 1}^{K} Z_{n k^{'}} δ_{k^{'} d} + ϵ_{n d}$ , where $ϵ_{n d} \sim 𝒩 (0, 1)$ , and $𝜹$ defined cluster-specific mean shifts with signal strength $τ = 2.31$ with $δ_{11} = δ_{14} = δ_{22} = δ_{25} = δ_{33} = δ_{36} = τ$ , others zero). For a specific MNAR scenario, the class-specific intercept component $ψ_{k}^{z}$ was $0$ , and variable-specific slopes $ψ_{j}^{y}$ were $(1.45, 0.2, - 3, 1.45, 0.2, - 3)$ , resulting in $P (M_{i j} = 1 | Y_{i j}, Z_{i k} = 1) = Φ (ψ_{j}^{y} Y_{i j})$ .

In Figure 4 and 5, we plot the boxplots of ARI and NRMSE over 20 replications. We observe that MNAR-based approaches attain better results over methods not designed for MNAR patterns. These methods deliver competitive performance even when the true missingness mechanism is more complex (MNARy, MNARyz); thereby, supporting the conclusion in [62]. Furthermore, our framework demonstrates even slightly higher ARI and lower NRMSE in some cases compared to standard MNARz.

F.3 Sensitivity on the choice of c

As discussed in [13] and further investigated through Theorem 4, the choice of the hyperparameter $c$ plays a crucial role in the stepwise construction of the relevant variable set $\hat{𝕊}$ , balancing the risk of stopping the selection process too early with the risk of incorrectly including irrelevant variables. Our theoretical work, particularly Theorem 4 and the formulation for selecting $c$ in Equation 33, suggests that $c$ should ideally be determined by considering the probability of a single incorrect inclusion ( $p_{N} (N)$ ), the number of non-relevant variables ( $M_{N R}$ ), and a desired tolerance for overall false positives in $\hat{𝕊}$ ( $ϵ_{S, F P}$ ). The experimental results presented here provide practical insights into this interplay and generally support our theoretical conclusions.

As illustrated in Figures 6 and 7, the two datasets (Dataset 1: $D = 7, s_{0} = 3$ ; Dataset 2: $D = 14, s_{0} = 2$ ) show different optimal ranges for $c$ . This aligns with the theoretical expectation that $c$ is not a universal constant but rather interacts with dataset-specific factors, which are encapsulated by $p_{N} (N)$ and $M_{N R}$ . For Dataset 1, the excellent performance with $c = 2$ (achieving 100% correct selection of relevant variables) points to a very low $p_{N} (N)$ . This suggests that after the three true relevant variables are found, the remaining four non- $𝕊_{0}$ variables consistently produce ${BIC}_{diff} \leq 0$ , making a small stopping value like $c = 2$ both effective and safe. The continued strong performance at $c = 7$ (90% correct) further indicates that $p_{N} (N)$ is small enough that even scanning all $M_{N R} = 4$ non-relevant variables rarely leads to a false inclusion. This situation is consistent with a theoretical scenario where $p_{N} (N) ≪ 1 / M_{N R}$ , minimizing the risk of false positives regardless of $c$ within a reasonable range.

In contrast, Dataset 2, which has more non- $𝕊_{0}$ variables ( $M_{N R} = 12$ ), demonstrates a clearer trade-off. Good performance is observed for $c = 2$ (80%) and $c = 3$ (90%), implying that $p_{N} (N)$ is still relatively small. However, the notable decline in performance when $c = 7$ (only 5% correct $𝕊_{0}$ selection) empirically validates a key theoretical concern: if $c$ is set too high and $p_{N} (N)$ is not sufficiently close to zero, the algorithm examines more non- $𝕊_{0}$ variables while awaiting $c$ consecutive negative ${BIC}_{diff}$ values. Each additional variable examined increases the cumulative chance of a Type I error (a non- $𝕊_{0}$ variable having ${BIC}_{diff} > 0$ by chance). If the likelihood of obtaining $c$ correct rejections in a row, ${(1 - p_{N} (N))}^{c}$ , diminishes significantly as more variables are processed, false inclusions become more probable. The poor result for $c = 7$ in Dataset 2 suggests its $p_{N} (N)$ value makes it unlikely to achieve seven consecutive correct rejections before a false positive occurs among the $M_{N R} = 12$ candidates. This aligns with the theoretical relationship $c \approx \ln (M_{N R} / ϵ_{S, F P}) / p_{N} (N)$ : for a given $ϵ_{S, F P}$ , a higher $p_{N} (N)$ or a larger $M_{N R}$ would generally favor a smaller $c$ to maintain that error tolerance, or a large fixed $c$ might lead to a poorer effective $ϵ_{S, F P}$ . The finding that $c = 3$ is effective for Dataset 2 is consistent with the heuristic used in prior work [13], suggesting it offers a practical compromise when $p_{N} (N)$ is small but non-negligible, and $M_{N R}$ is moderate. Furthermore, the relative stability of cluster number selection across different values of $c$ suggests that $c$ ’s main influence is on the selection of variables for $𝕊$ , as predicted by theory, with more indirect effects on the overall model choice, primarily if $\hat{𝕊}$ is significantly misidentified.

F.4 Computational Times and Scalability

Complexity Analysis

We count arithmetic operations up to absolute constants (big-Oh). Throughout this part we denote: $N \in ℕ$ (samples), $D \in ℕ$ (variables), $K \in ℕ$ (mixture components). Let $M_{EM}$ be a uniform upper bound on the per-fit number of EM iterations until convergence (Assumption A2 below). In Stage A (ranking), the M-step contains a graphical-lasso solve per component with at most $M_{glasso}$ outer iterations. Covariance inverses and log-determinants are computed by Cholesky factorizations.

Assumptions. We work under the following explicit conditions.

•

A1. The ground-truth SRUW partition has $D_{eff} := | 𝕊_{true} | + | 𝕌_{true} | ≪ D$ .
•

A2. Each EM fit terminates in at most $M_{EM}$ iterations. For population/regularized EM, this is justified by established convergence rates: either sublinear convergence $M_{EM} = 𝒪_{p} (1 / \sqrt{N})$ [31] or geometric convergence $M_{EM} = 𝒪 (\log (1 / ϵ))$ to achieve $ϵ$ -accuracy [76]. Our analysis uses the more conservative bounded iteration assumption for clarity.
•

A3. With probability $1 - o (1)$ , the Stage-A ranking lists all $D_{eff}$ informative variables before any purely irrelevant variables (proved in Theorem 3 of the paper).
•

A4. Let $C_{glasso} (d)$ denote the arithmetic cost of one graphical-lasso solve of size $d \times d$ . In general, $C_{glasso} (d) = Θ (M_{glasso} d^{3})$ . Under connected-component decomposition with largest block size $s_{\max}$ (as in [73, 41]), $C_{glasso} (d) = Θ (M_{glasso} \sum_{c} p_{c}^{3}) \leq Θ (M_{glasso} d s_{\max}^{2})$ .

Per-iteration costs for a $d$ -variate GMM. One EM iteration for a $K$ -component Gaussian mixture in dimension $d$ has:

•

E-step: Responsibilities $t_{n k}$ require evaluating Gaussian log-densities for all $(n, k)$ . With per-component $𝚺_{k}^{- 1}$ and $\log det 𝚺_{k}$ fixed within the iteration, each evaluation uses a matrix-vector product and a quadratic form, costing $Θ (d^{2})$ . Hence $Θ (N K d^{2})$ .
•

M-step (means, weights): Weighted sums are $Θ (N K d)$ .
•

M-step (covariances): Weighted second moments yield $Θ (N K d^{2})$ . The per-component matrix factorization/inversion is $Θ (d^{3})$ , hence $Θ (K d^{3})$ .

Therefore one EM iteration in dimension $d$ costs

Θ ​ (N ​ K ​ d^{2}) + Θ ​ (K ​ d^{3}),

and one EM fit (up to convergence) costs

𝒞_{EM} ​ (d) = Θ ​ (M_{EM} ​ (N ​ K ​ d^{2} + K ​ d^{3})) .

(40)

The classical SRUW selection starts from all $D$ variables and eliminates one at a time. At step $j$ ( $j = D, D - 1, \dots, 2$ ), to remove one variable it evaluates $j$ candidates; each evaluation requires an EM fit in dimension $j - 1$ . Using Equation 40, the total cost is

\sum_{j = 2}^{D} j \cdot 𝒞_{EM} ​ (j - 1) = Θ ​ (M_{EM} ​ \sum_{j = 2}^{D} j ​ (N ​ K ​ {(j - 1)}^{2} + K ​ {(j - 1)}^{3})) .

Using the polynomial sums:

\sum_{j = 1}^{D} j^{3} = \frac{D^{2} ​ {(D + 1)}^{2}}{4} = Θ ​ (D^{4})

and

\sum_{j = 1}^{D} j^{4} = \frac{D ​ (D + 1) ​ (2 ​ D + 1) ​ (3 ​ D^{2} + 3 ​ D - 1)}{30} = Θ ​ (D^{5})

, we obtain the tight bound

𝒞_{stepwise} = Θ ​ (M_{EM} ​ (N ​ K ​ D^{4} + K ​ D^{5})) .

(41)

The $D^{5}$ term renders this approach impractical beyond moderate $D$ .

Stage A (Ranking). For each $(λ, ρ)$ on a grid of size $M_{grid}$ , we run a penalized EM. Per iteration: the E-step remains $Θ (N K D^{2})$ ; the M-step adds $K$ graphical-lasso solves. By Assumption A4,

per iter cost = Θ ​ (N ​ K ​ D^{2}) + Θ ​ (K ​ C_{glasso} ​ (D)) .

Therefore the total Stage-A cost is

𝒞_{rank} = Θ ​ (M_{grid} ​ M_{EM} ​ (N ​ K ​ D^{2} + K ​ C_{glasso} ​ (D))) .

(42)

Two explicit regimes follow immediately from A4:

(Dense/general)	$C_{glasso} (D) = Θ (M_{glasso} D^{3})$
	$\Rightarrow 𝒞_{rank} = Θ (M_{grid} M_{EM} (N K D^{2} + K M_{glasso} D^{3})) .$	(43)
$(Connected components of size s_{\max} )$	$C_{glasso} (D) = Θ (M_{glasso} D s_{\max}^{2})$
	$\Rightarrow 𝒞_{rank} = Θ (M_{grid} M_{EM} (N K D^{2} + K M_{glasso} D s_{\max}^{2})) .$	(44)

Stage B (Role assignment). A single forward/backward pass evaluates a constant number of SRUW-MNARz fits per newly considered variable. Under A3 the pass stops after $D_{eff}$ additions; hence the Stage-B complexity is

𝒞_{role} = Θ ​ (M_{EM} ​ (N ​ K ​ D_{eff}^{2} + K ​ D_{eff}^{3})) .

(45)

Since $D_{eff} ≪ D$ (A1), $𝒞_{role}$ is strictly lower order than $𝒞_{rank}$ and Stage A dominates. Combining Equation 43-Equation 44 and Equation 45, the two-stage complexity is

𝒞_{two-stage} = 𝒞_{rank} + 𝒞_{role} = Θ ​ (M_{grid} ​ M_{EM} ​ (N ​ K ​ D^{2} + K ​ C_{glasso} ​ (D))) + o ​ (𝒞_{rank}) .

(46)

From Equation 41 and Equation 46, the speedup factor satisfies

\frac{𝒞_{stepwise}}{𝒞_{two-stage}} = Ω ​ (\frac{N ​ K ​ D^{4} + K ​ D^{5}}{M_{grid} ​ (N ​ K ​ D^{2} + K ​ C_{glasso} ​ (D))}) .

Two explicit lower bounds follow.

•

If $N \geq M_{glasso} D$ (E-step dominates Stage A): using Equation 43,

$\frac{𝒞_{stepwise}}{𝒞_{two-stage}} = Ω (\frac{D^{2}}{M_{grid}}) .$
•

If $N < M_{glasso} D$ (glasso dominates Stage A): using Equation 43,

$\frac{𝒞_{stepwise}}{𝒞_{two-stage}} = Ω (\frac{D^{2}}{M_{grid} M_{glasso}}) .$

Under connected-component sparsity with block size $s_{\max}$ Equation 44 the second case strengthens to

\frac{𝒞_{stepwise}}{𝒞_{two-stage}} = Ω ​ (\frac{D^{2}}{M_{grid}} \cdot \min {1, \frac{N}{M_{glasso} ​ s_{\max}^{2}}}) .

In all regimes the speedup is $Ω (D^{2} / M_{grid})$ , and strictly larger when sparsity (small $s_{\max}$ ) is present.

Empirical Validation

We report wall-clock times (seconds) for representative scenarios, confirming the predicted polynomial speedups:

Scenario	$N$	$D$	$K$	SelvarMNARz (s)	Clustvarsel (s)	Speedup
Varying $D$	750	15	4	12.2	190	$\sim$ 15 $\times$
Varying $D$	750	21	4	14.4	640	$\sim$ 44 $\times$
Varying $D$	750	27	4	15.5	2054	$\sim$ 132 $\times$
Varying $N$	1000	20	4	19.0	838	$\sim$ 44 $\times$
Varying $K$	750	20	12	9.77	1242	$\sim$ 127 $\times$

These measurements are consistent with the theory: the two-stage method scales like $D^{2}$ in the dense case (and better under sparsity), whereas backward stepwise scales as $D^{4}$ - $D^{5}$ .

Dependency of Computational Times under Degree of Missingness

We analyze the computational dependency of our framework on the proportion of missing data, revealing an advantageous property: runtime decreases with increasing missing rates due to the efficient handling of incomplete data patterns in the EM algorithm.

The key insight stems from the E-step computational complexity in Stage B, which operates directly on the observed data patterns. Let $X_{n}$ denote the number of observed entries in observation $𝒚_{n}$ . The E-step cost for Gaussian mixture models scales as:

𝒞_{E-step} = Θ ​ (\sum_{n = 1}^{N} \sum_{k = 1}^{K} X_{n}^{2}) = Θ ​ (K ​ N ​ 𝔼 ​ [X^{2}])

For different missingness mechanisms:

•

MCAR at rate $r$ : $X \sim Binom (D, 1 - r)$ , yielding

$𝔼 [X^{2}] = Var (X) + 𝔼 {[X]}^{2} = D (1 - r) r + D^{2} {(1 - r)}^{2}$

This produces a quadratic decrease in E-step work as $r ↑ 1$ .
•

MAR/MNARz: The same complexity bound applies, with $X_{n}$ representing the random count of observed coordinates per observation. Each per-record Gaussian density evaluation scales quadratically with observed entries, maintaining the $Θ (K N 𝔼 [X^{2}])$ complexity.

Stage A employs efficient single imputation and remains largely insensitive to missing rates, while Stage B inherits the beneficial $𝔼 [X^{2}]$ scaling. Consequently, total runtime decreases monotonically with increasing missingness rates.

We empirically verified this property by fixing $(N, D, K) = (1000, 14, 4)$ and systematically increasing the missing data rate under two mixed missingness scenarios. The results, summarized in Table 4 and Figure 8, confirm the theoretical predictions.

Table 4: Runtime (seconds) with increasing missing rate under mixed mechanisms.

Missing Mechanism	20%	30%	50%	80%
Mixed (MAR+MNAR)	54.1	46.2	32.9	23.9
Mixed (MCAR+MAR+MNAR)	34.9	23.8	25.5	22.8

Three key observations emerge:

1.

Runtime consistently decreases with higher missing rates, as predicted by the $𝔼 [X^{2}]$ scaling. The MNARz block processes smaller observed patterns during E-step calculations, reducing computational burden for conditional expectations and complete-data sufficient statistics.
2.

The slight variation at 30%-50% missingness in the three-mechanism scenario likely stems from random allocation of missing positions. When missing data concentrates in clustering variables ( $𝕊$ ), the number of EM iterations for convergence may vary, but the overall inverse relationship persists.
3.

This property provides significant practical benefits. Users can expect faster processing on datasets with higher missing rates-a valuable characteristic for real-world applications where extensive missing data is common.

The combination of mathematical analysis and empirical results demonstrates that our framework not only handles high missing rates effectively but also becomes more computationally efficient as missing data increases, making it particularly suitable for challenging real-world datasets with substantial missingness.

F.5 More on Simulated Dataset

Performance under Model Misspecification. Previous simulations focused on “pure” MAR and MNAR scenarios to clarify the benefits of our model when combined with the MNARz mechanism. However, real-world missingness mechanisms are often mixed. To assess robustness, we conducted additional simulations with a dataset of $D = 14$ variables, where the true clustering variable set is $𝕊^{⋆} = {1, 2}$ (remaining variables play roles of $ℝ, 𝕌, 𝕎$ according to scenario 8 in section 5. We generated missing data using a mixture of mechanisms:

•

MCAR: Some variables have missing values completely at random
•

MAR: Missingness depends only on observed components $y_{n}^{o}$
•

MNARy: Missingness depends directly on unobserved values $𝒚^{m}$ (actively violating MNARz assumption)

The MNARy mechanism (see [62] for precise formulation) specifically tests our model’s robustness to misspecification, as it depends on the unobserved values rather than cluster assignments. We compare our method against several baselines, including a novel two-step approach designed to isolate the benefits of joint modeling:

1.

MNARz + SelvarMix: Estimate GMM-MNARz model and impute missing data, then run SelvarMix [13] on the imputed dataset
2.

Multiple Imputation variants: Using both missRanger and gcimputeR (cite here) with Mclust
3.

VarSelLCM: A competing joint modeling approach

Table 5: Experiment with mixed MAR+MNARy (True set

{1, 2}

)

Method	ARI	NRMSE	Relevant Variables Selected
SelvarMNARz (ours)	0.808	0.176	1, 2 (correct)
VarSelLCM	0.779	–	1-11 (extra variables)
missRanger + Mclust	0.799	0.190	–
gcimputeR + Mclust	0.774	0.413	–
MNARz + SelvarMix	0.328	0.070	1 (missing variable)

Table 6: Experiment with mixed MCAR+MAR+MNARy (True set

{1, 2}

)

Method	ARI	NRMSE	Relevant Variables Selected
SelvarMNARz (ours)	0.808	0.181	1, 2 (correct)
VarSelLCM	0.772	–	1-11 (extra variables)
missRanger + Mclust	0.783	0.198	–
gcimputeR + Mclust	0.755	0.401	–
MNARz + SelvarMix	0.328	0.087	1 (missing variable)

The poor performance of the MNARz + SelvarMix baseline prompted a deeper investigation. To test whether properly handling imputation uncertainty could rescue the decoupled strategy, we also ran Multiple Imputation (MI) variants (missRanger-MI and MNARz-MI with random initializations) and pooled the results. Using the default Rmixmod backend for SelvarMix yielded the results below.

Table 7: Performance on Mixed (MAR+MNARy) Data (Rmixmod backend)

Method	ARI	NRMSE	Relevant Variables
SelvarMNARz (Ours)	0.778	0.162	1, 2
Decoupled (SI MNARz)	0.328	0.263	1
Decoupled (MI missRanger)	0.336	0.257	1
Decoupled (MI MNARz)	0.328	0.331	1

Table 8: Performance on Mixed (MCAR+MAR+MNARy) Data (Rmixmod backend)

Method	ARI	NRMSE	Relevant Variables
SelvarMNARz (Ours)	0.773	0.164	1, 2
Decoupled (SI MNARz)	0.328	0.387	1
Decoupled (MI MNARz)	0.336	0.322	1
Decoupled (MI missRanger)	0.328	0.248	1

From Table 7 and Table 8, we deduce two key takeaways: (i) Our joint model retains high ARI and correct selection under mixed mechanisms, including MNARy misspecification; (ii) MI does not repair the decoupled pipeline in this setting, with results mirroring single-imputation. This suggests two contributing factors:

1.

Uncertainty Propagation. Decoupled pipelines treat completed data as observed, discarding posterior uncertainty in $𝒚^{m}$ . Our EM-based framework propagates this uncertainty through all parameter and role updates.
2.

Downstream Stability. The variable-selection backend (SelvarMix with its Rmixmod engine) shows instability (e.g., sensitivity to local maxima); MI then averages multiple weak fits. The joint estimation procedure is empirically more stable in this context.

Finally, under these mixed mechanisms, the class-level MNARz parameters in our model often converge to similar values across clusters for variables whose missingness is effectively MAR or MCAR, while remaining discriminative where missingness is truly class-linked. This helps explain the model’s robustness to misspecification.

Effect of Initialization on EM Stability and Accuracy. We compared a hierarchical clustering (HC) based initialization (Ward’s linkage on Euclidean distances with cluster centers extracted from the dendrogram cut at $K$ ) against multiple random initializations (MIs). HC places initial centers in high–density regions, yielding more stable responsibilities at the first E–step and fewer poor local optima than purely random starts. As shown in Table 9, HC attains higher ARI on 7 of 8 scenarios, and trails slightly once, confirming its overall robustness and improved convergence behavior for the EM algorithm.

Table 9: Comparison of EM initialization methods under 8 data scenarios in Section 5 (higher ARI is better). MIs: Multiple random initializations.

Scenario	MIs (ARI)	HC (ARI)
1	0.283	0.317
2	0.479	0.533
3	0.551	0.536
4	0.441	0.654
5	0.760	0.771
6	0.711	0.778
7	0.716	0.780
8	0.785	0.786

HC initialization offers a more sensible and stable warm start than multiple random restarts, typically improving both EM convergence and final clustering accuracy (ARI), with negligible overhead relative to the overall EM cost.

F.6 More on Transcriptome Dataset

Background and Prior Analyses

Dataset. We analyze the Arabidopsis thaliana transcriptome comprising $1267$ genes measured across $27$ experimental conditions aggregated from seven projects P1-P7. Genes were preselected for differential expression at least once in the hypocotyl growth switch time course (Project 6), making P6 biologically central. Following [37, 39], we retain all $1267$ genes: $1149$ are fully observed; $118$ contain missing entries (107 with one, 10 with two, 1 with three). Overall, $9.3 %$ of genes have any missingness and the global missing rate is $0.38 %$ .

Prior findings. SelvarClust [37] (complete cases) found that including irrelevant variables degrades homogeneity; variable selection produced more coherent clusters and recovered known co-expression groups (e.g., a cluster of 15 genes co-clustered with 4 well-studied markers). SelvarClustMV [39] (MAR setting) expanded the gene set by reprocessing previously excluded genes and treating missing data within an EM framework, concluding that P6 (hypocotyl switch) and P7 (isoxaben treatment) are clustering‐relevant, together with P1-P4, whereas P5 (nematode infection) is not primarily grouping. Both studies support P2 (iron signaling) as a core axis for defining co-expression groups. Note that the 2012 analysis assumes MAR for the missing entries. In contrast, our framework explicitly models class-dependent missingness (MNARz), while retaining MAR/MCAR as limiting cases through parameterization.

Additional Interpretation of Our Results on the Transcriptome

Global outcome. Fitting SelvarMNARz for $K \in 2, \dots, 20$ with $c = 5$ , spectral distance weights $𝑷_{k}$ , and $p_{k} L C$ structure, the selected model yields $18$ clusters and a global role assignment with P1-P4 in $𝕊$ and P5-P7 in $𝕌$ . This agrees with prior work on P5 (not clustering-relevant) but differs by reclassifying P6-P7 from $𝕊$ (in [37, 39]) to $𝕌$ .

Cluster-level diagnostics clarify the difference. Let $R_{k}^{2}$ denote the coefficient of determination from regressing the $𝕌$ -block (here, P5-P7) on the $𝕊$ -block (P1-P4) within cluster $k$ . Table 2 shows pronounced heterogeneity:

•

Several clusters (e.g., 6, 7, 8, 10, 12, 18) have $R_{k}^{2} > 0.60$ , indicating that P5-P7 are largely explained by P1-P4 within those clusters. For these groups, assigning P5-P7 to $𝕌$ is appropriate.
•

A large aggregate (Cluster 1) exhibits low $R^{2}$ alongside flat $𝕊$ -profiles and detectable $𝕌$ -activity, suggesting local decoupling of P5-P7 from P1-P4. This mirrors the biological intuition behind earlier inclusion of P6-P7 in $𝕊$ .

Hence, both the earlier global $𝕊$ -assignment (P6-P7 in $𝕊$ ) and our global $𝕌$ -assignment (P6-P7 in $𝕌$ ) are locally valid—but on different clusters. The discrepancy is explained by heterogeneity: the relationship between early axes (P1-P4) and late/stress projects (P5-P7) is cluster-specific.

Why does our global BIC prefer P5-P7 in $𝕌$ ? Two factors are at play:

1.

Global model selection. BIC aggregates fit-complexity tradeoffs across all clusters. Because many clusters exhibit high $R_{k}^{2}$ (P5-P7 explained by P1-P4), the global criterion prefers a parsimonious $𝕊$ (P1-P4) and assigns P5-P7 to $𝕌$ .
2.

MNARz identifiability and shrinkage. Under MNARz, class-specific missingness parameters $ρ_{k d}$ absorb class-linked absence patterns. When a project effectively behaves as MAR/MCAR for many clusters, the fitted $ρ_{k d}$ across $k$ becomes nearly homogeneous and the conditional dependence of P5-P7 on P1-P4 becomes tighter, further favoring $𝕌$ globally.

Biological reading consistent with both views. P2 (iron signaling) is reaffirmed as a core driver. P5-P7 behave as late/stress outputs strongly coupled to P1-P4 in many clusters (high $R_{k}^{2}$ ), but retain independent variation in a sizable subset (e.g., Cluster 1). This reconciles the prior decision to place P6-P7 in $𝕊$ (emphasizing hypocotyl-centric signals) with our global $𝕌$ assignment (emphasizing aggregate parsimony across all clusters).

Practical implication. Our diagnostics suggest a natural extension: cluster-adaptive role assignment or a hierarchical prior tying per-cluster roles, which would allow P6-P7 to enter $𝕊$ only where local evidence (low residual error) warrants it while retaining parsimony elsewhere. This aligns with the heterogeneity revealed by $R_{k}^{2}$ and preserves the strengths of both global viewpoints.

Our unified $𝕊$ / $𝕌$ decision is globally efficient and statistically supported by MNARz-aware likelihood, while cluster-level diagnostics uncover biologically meaningful deviations. Together, they provide a coherent picture: P1-P4 are the principal axes; P5-P7 are predominantly redundant but locally informative in specific clusters, explaining the divergence from prior MAR-based analyses.

Appendix G Additional Details on Related Work in the Literature

G.1 Model-based Clustering

Model-based clustering conceptualizes clustering as a statistical inference problem, where the data are assumed to be generated from a finite mixture of probability distributions, each corresponding to a latent cluster. Unlike heuristic-based methods, this paradigm enables principled inference, allowing for parameter estimation via likelihood-based techniques and objective model selection to determine the number of clusters.

A prototypical instance is the GMM, wherein each cluster is characterized by a multivariate Gaussian distribution. Parameter estimation is typically conducted using the EM algorithm [16], yielding soft assignments in which each observation is associated with posterior probabilities across clusters. GMMs offer more flexibility than simpler methods like $k$ -means, as they can model clusters with varying shapes and overlapping regions via covariance structures.

Bayesian mixture models (MMs) extend this framework by treating both the parameters and the number of components as random variables with prior distributions. This fully Bayesian approach enables comprehensive uncertainty quantification over both model parameters and cluster allocations. For a fixed number of components, inference is often performed using Markov Chain Monte Carlo (MCMC) or variational methods. However, Bayesian MMs face the label switching problem due to the symmetry of the likelihood with respect to component labels, which renders the posterior non-identifiable without additional constraints. Strategies such as relabeling algorithms and identifiability constraints have been proposed to address this issue [65]. A notable advantage of the Bayesian approach is the incorporation of priors on model complexity, e.g., via Reversible Jump MCMC or birth-death processes, to infer the number of clusters. Despite their flexibility and robustness, fully Bayesian clustering methods can be computationally demanding, particularly when MCMC chains converge slowly or require extensive post-processing to resolve label ambiguity [29].

G.2 Variable Selection for Model-based Clustering

Historically, variable selection was performed using best-subset or stepwise selection approaches, typically guided by information criteria such as AIC [1] or BIC [58]. While effective for moderate-dimensional settings, these methods become computationally prohibitive as the number of features $D$ increases.

The advent of penalized likelihood methods improved scalability and enabled sparse modeling. The LASSO [70], a seminal technique using an $L_{1}$ penalty, enables both variable selection and coefficient shrinkage. Predecessors include the nonnegative garrote [9] and ridge regression [28], the latter using an $L_{2}$ penalty, which does not induce sparsity. Efficient algorithms such as LARS [18] and coordinate descent have facilitated high-dimensional applications of these methods. However, the time complexity of LARS limits its stability when $D$ is very large.

Law et al. [32] proposed a wrapper approach that jointly performs clustering and variable selection through greedy subset evaluation, introducing the notion of feature saliency to assess variable importance in cluster discrimination. Andrews and McNicholas [3] developed the VSCC algorithm, combining filter and wrapper strategies: variables are ranked based on within-cluster variance from an initial clustering, then incrementally added subject to correlation thresholds to enhance separability. Their R package implementation, vscc, offers efficient noise filtering prior to model-based refinement.

Raftery and Dean [54] introduced a model selection framework for variable selection within GMMs, categorizing variables as clustering, candidate, or noise. Each candidate is evaluated using BIC to compare models where the variable does or does not influence clustering. Their framework accommodates statistical dependence between irrelevant and clustering variables via regression, avoiding overly simplistic independence assumptions. Scrucca and Raftery [59] enhanced this method, introducing computational heuristics in the clustvarsel R package to streamline EM evaluations.

Expanding this idea, Maugis et al. [38] proposed a three-role framework, relevant, irrelevant, and redundant variables, with redundancies modeled via linear regression on relevant variables. This design improves accuracy in scenarios with correlated predictors. Nevertheless, the stepwise search remains computationally demanding as dimensionality increases.

To address scalability, Celeux et al. [13] proposed a two-stage approach: first, variables are ranked by penalized likelihood using the method of Zhou et al. [75], which penalizes component means and precisions. Then, a linear scan through this ranking assigns roles. This heuristic drastically reduces computation, preserves model interpretability, and achieves strong empirical performance, though theoretical guarantees for recovering the correct $𝕊 ℝ 𝕌 𝕎$ partition remain open.

Extensions to categorical data include adaptations of the Raftery-Dean method to Latent Class Analysis (LCA) by Dean and Raftery [15], with further refinement by Fop et al. [20], resulting in improved class separation in clinical datasets. Bontemps and Toussile [8] considered mixtures of multinomial distributions, using slope-heuristic-adjusted penalization to improve model selection under small-sample conditions.

G.3 Handling Missing Data

In practical datasets, missing values often occur and are categorized as MCAR, MAR, or MNAR. Under the MCAR assumption, missingness is unrelated to any data values; MAR allows dependence on observed data; MNAR, the most complex case, involves dependence on unobserved or latent variables.

Under MAR, Multiple Imputation (MI) [57] has emerged as a robust strategy to address uncertainty. Instead of single imputation, MI generates multiple completed datasets via draws from predictive distributions, followed by Rubin’s combination rules to aggregate inference. MICE [71], a flexible implementation, fits univariate models conditionally and iteratively imputes missing values, supporting mixed data types and nonlinear relationships.

High-dimensional settings pose new challenges, where full joint modeling becomes unstable. To address this, Zhao and Long [74] incorporated Lasso and Bayesian Lasso into the MICE framework to enhance prediction accuracy through regularized imputation, effectively reducing overfitting and variable selection bias in high- $D$ regimes.

Machine learning techniques have also advanced imputation. MissForest [63] uses random forests to iteratively impute variables based on others, capturing nonlinearities in a nonparametric manner. Although not explicitly designed for clustering, the ensemble mechanism implicitly reflects local data structures akin to clustering. Deep learning methods, such as MIWAE [36], adopt autoencoder architectures to generate multiple imputations from learned latent spaces, enabling further extensions like Fed-MIWAE [4] for privacy-sensitive contexts. These models assume a latent manifold structure and are best suited for large sample sizes, though interpretability remains a challenge.

Addressing MNAR data in clustering models remains difficult due to identifiability issues and the need for strong assumptions or auxiliary information. Two general strategies exist: (1) Selection models, specifying a joint model for data and missingness mechanisms, and (2) Pattern mixture models, defining distributions conditioned on missingness patterns and modeling their influence on cluster membership. In [62], Sportisse et al. investigated identifiability conditions in MNAR selection models and proposed methods to augment clustering models with missingness-informed constraints to improve identifiability.

[bib.bib1] [1] H. Akaike. A new look at the statistical model identification. IEEE transactions on automatic control, 19(6):716–723, 2003. ↩ 1 G.2

[bib.bib2] [2] E. S. Allman, C. Matias, and J. A. Rhodes. Identifiability of parameters in latent structure models with many observed variables. Annals of Statistics, 37(6A):3099–3132, 2009. ↩ D.1

[bib.bib3] [3] J. L. Andrews and P. D. McNicholas. Variable selection for clustering and classification. Journal of Classification, 31(2):136–153, 2014. ↩ 1 G.2

[bib.bib4] [4] I. Balelli, A. Sportisse, F. Cremonesi, P.-A. Mattei, and M. Lorenzi. Fed-miwae: Federated imputation of incomplete data via deep generative models. arXiv preprint arXiv:2304.08054, 2023. ↩ 1 G.3

[bib.bib5] [5] E. Barut, J. Fan, and A. Verhasselt. Conditional sure independence screening. Journal of the American Statistical Association, 111(515):1266–1277, 2016. ↩ 1

[bib.bib6] [6] J.-P. Baudry, C. Maugis, and B. Michel. Slope heuristics: overview and implementation. Statistics and Computing, 22(2):455–470, 2012. ↩ 2

[bib.bib7] [7] C. Biernacki, G. Celeux, and G. Govaert. Assessing a mixture model for clustering with the integrated completed likelihood. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7):719–725, 2000. ↩ 2

[bib.bib8] [8] D. Bontemps and W. Toussile. Clustering and variable selection for categorical multivariate data. Electronic Journal of Statistics, 7:2344–2371, Jan. 2013. ↩ 6 G.2

[bib.bib9] [9] L. Breiman. Better subset regression using the nonnegative garrote. Technometrics, 37(4):373–384, 1995. ↩ 1 G.2

[bib.bib10] [10] L. Breiman. Random forests. Machine Learning, 45:5–32, 2001. ↩ 5

[bib.bib11] [11] A. Casa, A. Cappozzo, and M. Fop. Group-wise shrinkage estimation in penalized model-based clustering. Journal of Classification, 39(3):648–674, 2022. ↩ 2 A

[bib.bib12] [12] G. Celeux and G. Govaert. Gaussian parsimonious clustering models. Pattern Recognition, 28(5):781–793, 1995. Publisher: Elsevier. ↩ 2

[bib.bib13] [13] G. Celeux, C. Maugis-Rabusseau, and M. Sedki. Variable selection in model-based clustering and discriminant analysis with a regularization approach. Advances in Data Analysis and Classification, 13:259–278, 2019. ↩ 1 2 3 4 5 A F.3 F.5 G.2

[bib.bib14] [14] M. C. Chong, H. D. Nguyen, and TrungTin Nguyen. Risk Bounds for Mixture Density Estimation on Compact Domains via the h-Lifted Kullback–Leibler Divergence. Transactions on Machine Learning Research, 2024. ↩ 2

[bib.bib15] [15] N. Dean and A. E. Raftery. Latent class analysis variable selection. Annals of the Institute of Statistical Mathematics, 62:11–35, 2010. ↩ 6 G.2

[bib.bib16] [16] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–38, 1977. ↩ 1 G.1

[bib.bib17] [17] D. Do, L. Do, S. A. McKinley, J. Terhorst, and X. Nguyen. Dendrogram of mixing measures: Learning latent hierarchy and model selection for finite mixture models. arXiv preprint arXiv:2403.01684, 2024. ↩ 2

[bib.bib18] [18] B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of Statistics, pages 407–451, 2004. ↩ 1 G.2

[bib.bib19] [19] J. Fan and J. Lv. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society Series B: Statistical Methodology, 70(5):849–911, 2008. ↩ 1

[bib.bib20] [20] M. Fop, K. M. Smart, and T. B. Murphy. Variable selection for latent class analysis with application to low back pain diagnosis. The Annals of Applied Statistics, pages 2080–2110, 2017. ↩ 6 G.2

[bib.bib21] [21] F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel. Mixture of expert posterior surrogates for approximate Bayesian computation. In JDS 2022 - 53èmes Journées de Statistique de la Société Française de Statistique (SFdS), Lyon, France, June 2022. ↩ 2

[bib.bib22] [22] F. Forbes, H. D. Nguyen, T. Nguyen, and J. Arbel. Summary statistics and discrepancy measures for approximate Bayesian computation via surrogate posteriors. Statistics and Computing, 32(5):85, Oct. 2022. ↩ 2

[bib.bib23] [23] R. Foygel and M. Drton. Extended Bayesian Information Criteria for Gaussian Graphical Models. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010. ↩ 2

[bib.bib24] [24] S. Frühwirth-Schnatter, C. Pamminger, A. Weber, and R. Winter-Ebmer. Labor market entry and earnings dynamics: Bayesian inference using mixtures-of-experts Markov chain clustering. Journal of Applied Econometrics, 27(7):1116–1137, 2012. ↩ 2

[bib.bib25] [25] C. R. Genovese and L. Wasserman. Rates of convergence for the Gaussian mixture sieve. The Annals of Statistics, 28(4):1105 – 1127, 2000. ↩ 2

[bib.bib26] [26] N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures. The Annals of Statistics, 44(6):2726 – 2755, 2016. ↩ 2

[bib.bib27] [27] N. Ho and X. Nguyen. On strong identifiability and convergence rates of parameter estimation in finite mixtures. Electronic Journal of Statistics, 10(1):271–307, 2016. ↩ 2

[bib.bib28] [28] A. E. Hoerl and R. W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55–67, 1970. ↩ 1 G.2

[bib.bib29] [29] A. Jasra, C. C. Holmes, and D. A. Stephens. Markov Chain Monte Carlo Methods and the Label Switching Problem in Bayesian Mixture Modeling. Statistical Science, 20(1):50 – 67, 2005. ↩ 1 G.1

[bib.bib30] [30] S. Kim, M. G. Tadesse, and M. Vannucci. Variable selection in clustering via dirichlet process mixture models. Biometrika, 93(4):877–893, 2006. ↩ 1

[bib.bib31] [31] J. Kwon and C. Caramanis. The em algorithm gives sample-optimality for learning mixtures of well-separated gaussians. In Conference on Learning Theory, pages 2425–2487. PMLR, 2020. ↩ F.4

[bib.bib32] [32] M. H. Law, M. A. Figueiredo, and A. K. Jain. Simultaneous feature selection and clustering using mixture models. IEEE transactions on pattern analysis and machine intelligence, 26(9):1154–1166, 2004. ↩ 1 G.2

[bib.bib33] [33] R. J. Little and D. B. Rubin. Statistical analysis with missing data. John Wiley & Sons, 2019. ↩ A

[bib.bib34] [34] P.-L. Loh and M. J. Wainwright. Regularized M-estimators with Nonconvexity: Statistical and Algorithmic Theory for Local Optima. Journal of Machine Learning Research, 16(19):559–616, 2015. ↩ A

[bib.bib35] [35] M. Marbac and M. Sedki. Variable selection for mixed data clustering: A model-based approach. arXiv preprint arXiv:1703.02293, 2017. ↩ 5 A

[bib.bib36] [36] P.-A. Mattei and J. Frellsen. MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4413–4423. PMLR, June 2019. ↩ 1 G.3

[bib.bib37] [37] C. Maugis, G. Celeux, and M.-L. Martin-Magniette. Variable selection for clustering with gaussian mixture models. Biometrics, 65(3):701–709, 2009. ↩ 2 5 ↑ B D.1 F.6

[bib.bib38] [38] C. Maugis, G. Celeux, and M.-L. Martin-Magniette. Variable selection in model-based clustering: A general variable role modeling. Computational Statistics & Data Analysis, 53(11):3872–3882, 2009. ↩ 1 2 3 4 A B D.1 D.2 G.2

[bib.bib39] [39] C. Maugis-Rabusseau, M.-L. Martin-Magniette, and S. Pelletier. Selvarclustmv: Variable selection approach in model-based clustering allowing for missing values. Journal de la Société Française de Statistique, 153(2):21–36, 2012. ↩ 1 2 3 5 ↑ F.6

[bib.bib40] [40] M. Mayer. missRanger: Fast Imputation of Missing Values, 2025. R package version 2.6.2, https://mayer79.github.io/missRanger/. ↩ 5

[bib.bib41] [41] R. Mazumder and T. Hastie. Exact covariance thresholding into connected components for large-scale graphical lasso. Journal of Machine Learning Research, 13(27):781–794, 2012. ↩ F.4

[bib.bib42] [42] G. J. McLachlan and K. E. Basford. Mixture Models: Inference and Applications to Clustering, volume 38. Marcel Dekker, 1988. ↩ 1

[bib.bib43] [43] G. J. McLachlan and D. Peel. Finite mixture models. John Wiley & Sons, 2004. ↩ 1

[bib.bib44] [44] S. Negahban and M. J. Wainwright. Restricted strong convexity and weighted matrix completion: Optimal bounds with noise. Journal of Machine Learning Research, 13(53):1665–1697, 2012. ↩ A D.3

[bib.bib45] [45] D. N. Nguyen and Z. Li. Joint learning of Gaussian graphical models in heterogeneous dependencies of high-dimensional transcriptomic data. In The 16th Asian Conference on Machine Learning (Conference Track), 2024. ↩ 2

[bib.bib46] [46] H. Nguyen, T. Nguyen, and N. Ho. Demystifying Softmax Gating in Gaussian Mixture of Experts. In Advances in Neural Information Processing Systems, Dec. 2023. ↩ 2

[bib.bib47] [47] H. Nguyen, T. Nguyen, K. Nguyen, and N. Ho. Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts. In S. Dasgupta, S. Mandt, and Y. Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 2683–2691. PMLR, May 2024. ↩ 2

[bib.bib48] [48] H. D. Nguyen, T. Nguyen, and F. Forbes. Bayesian Likelihood Free Inference using Mixtures of Experts. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2024. ↩ 2

[bib.bib49] [49] T. Nguyen, F. Chamroukhi, H. D. Nguyen, and G. J. McLachlan. Approximation of probability density functions via location-scale finite mixtures in Lebesgue spaces. Communications in Statistics - Theory and Methods, 52(14):5048–5059, 2023. ↩ 2

[bib.bib50] [50] T. Nguyen, D. N. Nguyen, H. D. Nguyen, and F. Chamroukhi. A non-asymptotic theory for model selection in high-dimensional mixture of experts via joint rank and variable selection. In AJCAI Australasian Joint Conference on Artificial Intelligence 2023, Dec. 2023. ↩ 2

[bib.bib51] [51] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and F. Forbes. A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models. Electronic Journal of Statistics, 16(2):4742 – 4822, 2022. ↩ 2

[bib.bib52] [52] T. Nguyen, H. D. Nguyen, F. Chamroukhi, and G. J. McLachlan. Approximation by finite mixtures of continuous density functions that vanish at infinity. Cogent Mathematics & Statistics, 7(1):1750861, Jan. 2020. ↩ 2

[bib.bib53] [53] X. Nguyen. Convergence of latent mixing measures in finite and infinite mixture models. The Annals of Statistics, 41(1):370–400, 2013. ↩ 2

[bib.bib54] [54] A. E. Raftery and N. Dean. Variable selection for model-based clustering. Journal of the American Statistical Association, 101(473):168–178, 2006. ↩ 1 G.2

[bib.bib55] [55] A. Rakhlin, D. Panchenko, and S. Mukherjee. Risk bounds for mixture density estimation. ESAIM: PS, 9:220–229, 2005. ↩ 2

[bib.bib56] [56] D. B. Rubin. Inference and missing data. Biometrika, 63(3):581–592, 1976. ↩ A

[bib.bib57] [57] D. B. Rubin. An overview of multiple imputation. In Proceedings of the survey research methods section of the American statistical association, volume 79, page 84. Citeseer, 1988. ↩ 1 G.3

[bib.bib58] [58] G. Schwarz. Estimating the dimension of a model. The annals of statistics, pages 461–464, 1978. ↩ 1 2 G.2

[bib.bib59] [59] L. Scrucca and A. E. Raftery. clustvarsel: A package implementing variable selection for gaussian model-based clustering in r. Journal of Statistical Software, 84:1–28, 2018. ↩ 1 5 G.2

[bib.bib60] [60] W. Shen, S. T. Tokdar, and S. Ghosal. Adaptive Bayesian multivariate density estimation with Dirichlet mixtures. Biometrika, 100(3):623–640, 2013. ↩ 2

[bib.bib61] [61] C.-Y. Sin and H. White. Information criteria for selecting possibly misspecified parametric models. Journal of Econometrics, 71(1):207–225, 1996. ↩ 2

[bib.bib62] [62] A. Sportisse, M. Marbac, F. Laporte, G. Celeux, C. Boyer, J. Josse, and C. Biernacki. Model-based clustering with missing not at random data. Statistics and Computing, 34(4):135, 2024. ↩ 1 3 A F.2 F.5 G.3

[bib.bib63] [63] D. J. Stekhoven and P. Bühlmann. Missforest—non-parametric missing value imputation for mixed-type data. Bioinformatics, 28(1):112–118, 2012. ↩ 1 5 G.3

[bib.bib64] [64] M. Stephens. Bayesian analysis of mixture models with an unknown number of components—an alternative to reversible jump methods. The Annals of Statistics, 28(1):40 – 74, 2000. ↩ 1

[bib.bib65] [65] M. Stephens. Dealing with label switching in mixture models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 62(4):795–809, 2000. ↩ G.1

[bib.bib66] [66] N. Städler, P. Bühlmann, and S. van de Geer. $l_{1}$ -penalization for mixture regression models. TEST, 19(2):209–256, Aug. 2010. ↩ A

[bib.bib67] [67] M. G. Tadesse, N. Sha, and M. Vannucci. Bayesian variable selection in clustering high-dimensional data. Journal of the American Statistical Association, 100(470):602–617, 2005. ↩ 1

[bib.bib68] [68] H. Teicher. Identifiability of finite mixtures. The Annals of Mathematical Statistics, 34(4):1265–1269, December 1963. ↩ D.1

[bib.bib69] [69] T. Thai, T. Nguyen, D. Do, N. Ho, and C. Drovandi. Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures. arXiv preprint arXiv:2505.13052, 2025. ↩ 2

[bib.bib70] [70] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267–288, 1996. ↩ 1 A G.2

[bib.bib71] [71] S. Van Buuren and K. Groothuis-Oudshoorn. mice: Multivariate imputation by chained equations in r. Journal of statistical software, 45:1–67, 2011. ↩ 1 G.3

[bib.bib72] [72] J. Westerhout, T. Nguyen, X. Guo, and H. D. Nguyen. On the Asymptotic Distribution of the Minimum Empirical Risk. In Forty-first International Conference on Machine Learning, 2024. ↩ 2

[bib.bib73] [73] D. M. Witten, J. H. Friedman, and N. Simon. New insights and faster computations for the graphical lasso. Journal of Computational and Graphical Statistics, 20(4):892–900, 2011. ↩ F.4

[bib.bib74] [74] Y. Zhao and Q. Long. Multiple imputation in the presence of high-dimensional data. Statistical Methods in Medical Research, 25(5):2021–2035, 2016. ↩ 1 G.3

[bib.bib75] [75] H. Zhou, W. Pan, and X. Shen. Penalized model-based clustering with unconstrained covariance matrices. Electronic Journal of Statistics, 3(none):1473 – 1496, 2009. Publisher: Institute of Mathematical Statistics and Bernoulli Society. ↩ 1 2 3 A G.2

[bib.bib76] [76] M. Zhou et al. Global convergence of gradient em for over-parameterized gaussian mixtures. arXiv preprint arXiv:2506.06584, 2025. ↩ F.4

[bib.bib77] [77] H. Zou. The adaptive lasso and its oracle properties. Journal of the American Statistical Association, 101(476):1418–1429, 2006. ↩ 2

A Unified Framework for Variable Selection in Model-Based Clustering with Missing Not at Random

Abstract

1 Introduction

2 Background

3 Our Proposal: Variable Selection in Model-Based Clustering with MNAR

4 Theoretical Properties

Theorem 1 (Informal: Identifiability of the 𝕊​ℝ​𝕌​𝕎 Model).

Theorem 2 (Informal: BIC consistency for the 𝕊​ℝ​𝕌​𝕎 model).

Theorem 3 (Informal: Selection consistency of the two-stage procedure).

5 Experiments

6 Conclusion, Limitations and Perspectives

Acknowledgments

References

Appendix A More Details on Main Challenges and Contributions

Appendix B Preliminary of 𝕊​ℝ and 𝕊​ℝ​𝕌​𝕎 Models

Definition 1 (𝕊​ℝ Model for Complete Data).

Definition 2 (𝕊​ℝ​𝕌​𝕎 Model for Complete Data).

Proposition 1.

Proof of Proposition 1.

Fact 1 (Theorem 1 in [38]).

Appendix C Regular Assumptions for Main Theoretical Results

Assumption 1 (𝕊​ℝ).

Assumption 2.

Assumption 3.

Assumption 4 (𝕊​ℝ​𝕌​𝕎).

Assumption 5.

Assumption 6.

Appendix D Proof of Main Results

D.1 Proof of Theorem 1

Proof.

D.2 Proof of Theorem 2

Proposition 2.

Proof.

Proposition 3.

Proof.

Proposition 4.

Proof.

Proof of Theorem 2.

D.3 Proof of Theorem 3

Lemma 1 (Score Function Components).

Proof of Lemma 1.

Assumption 7 (Identifiability & Smoothness).

Assumption 8 (Compact Parameter Space & True Parameter Properties).

Assumption 9 (Data Distribution).

Assumption 10 (Bounded Posteriors).

Definition 3 (True Support of Penalized Parameters 𝕊0).

Assumption 11 (Restricted Eigenvalue Condition).

Remark 1.

Assumption 12 (Uniform Hessian Concentration).

Lemma 2 (Gradient Bound).

Proof of Lemma 2.

Lemma 3 (Parameter Consistency for Penalized GMM Estimator).

Proof of Lemma 3.

Remark 2.

Lemma 4 (Cone Condition for L1-Penalized M-Estimators).

Remark 3.

Proof of Lemma 4.

Remark 4 (Alignment with SRUW assumptions).

Assumption 13 (Standing Assumptions for proving Theorem 3).

Assumption 14 (Working high-probability event for ranking analysis).

Definition 4 (Quantities for Ranking Consistency).

Assumption 15 (True Relevant/Irrelevant Variables).

Assumption 16 (Minimum Signal Strength).

Assumption 17 (Regularization Grid).

Remark 5.

Lemma 5 (Ranking Consistency for Mean Parameters).

Proof of Lemma 5.

Theorem 4 (Selection Consistency of the Two-Step SRUW Procedure).

Proof.

Theorem 5 (Equivalence of MNARz-SRUW and MAR on Augmented Data for SRUW).

Proof of Theorem 5.

Corollary 1 (Estimator equivalence under augmentation).

Proof of Corollary 1.

Remark 6.

Appendix E Detailed EM Algorithms for the Two-Stage Procedure

Stage A (ranking): Penalized EM for adaptive-regularized GMM

Penalized objective.

E-step.

M-step: mixing weights.

M-step: means 𝝁k with ℓ1 penalty.

Theorem 1 (Informal: Identifiability of the $𝕊 ℝ 𝕌 𝕎$ Model).

Theorem 2 (Informal: BIC consistency for the $𝕊 ℝ 𝕌 𝕎$ model).

Appendix B Preliminary of $𝕊 ℝ$ and $𝕊 ℝ 𝕌 𝕎$ Models

Definition 1 ( $𝕊 ℝ$ Model for Complete Data).

Definition 2 ( $𝕊 ℝ 𝕌 𝕎$ Model for Complete Data).

Assumption 1 ( $𝕊 ℝ$ ).

Assumption 4 ( $𝕊 ℝ 𝕌 𝕎$ ).

Definition 3 (True Support of Penalized Parameters $𝕊_{0}$ ).

Lemma 4 (Cone Condition for $L_{1}$ -Penalized M-Estimators).

M-step: means $𝝁_{k}$ with $ℓ_{1}$ penalty.

M-step: precisions $𝚿_{k}$ via weighted graphical lasso.

Complete-data log-likelihood and $Q$ -function.

M-step: $π_{k}$ and MNARz parameters.