Software · Mixture of experts

API reference

The estimator classes, in the style of scikit-learn's generated class pages (e.g. BayesianGaussianMixture). The package also re-exports scikit-learn's GaussianMixture and BayesianGaussianMixture unchanged.

GaussianMixtureOfExperts

GaussianMixtureOfExperts(n_components=2, gate='softmax', *, covariance_type='full',
    tol=1e-4, reg_covar=1e-6, max_iter=200, n_init=1, init_params='kmeans',
    random_state=None, warm_start=False, verbose=0, gate_max_iter=100)

Frequentist maximum-likelihood EM for a softmax- or Gaussian-gated mixture of linear-Gaussian experts (scalar target). Faithful port of the CRPE-SGaME (NeurIPS 2023) and CRPE-GMoE / GLLiM (AISTATS 2024) estimators.

Parameters

name	meaning
`n_components`	int, default=2. Number of experts K.
`gate`	{'softmax','gaussian'}, default='softmax'. Gating family.
`covariance_type`	{'full','tied','diag','spherical'}, default='full'. Structure of the expert covariance Sigma_k in output space; the Gaussian gate always uses a full covariance.
`tol`	float, default=1e-4. EM convergence threshold on the mean lower bound.
`reg_covar`	float, default=1e-6. Added to variances/covariances for stability.
`max_iter`	int, default=200. Maximum EM iterations.
`n_init`	int, default=1. EM restarts; the highest-lower-bound fit is kept.
`init_params`	{'kmeans','random'}, default='kmeans'. Responsibility initialisation.
`gate_max_iter`	int, default=100. Inner iterations of the softmax gate MM update.
`random_state, warm_start, verbose`	Standard scikit-learn controls.

Attributes

name	meaning
`coef_`	(n_components, n_features + 1) expert coefficients; column 0 is the intercept.
`variances_`	(n_components,) per-expert residual variances sigma_k^2.
`gate_coef_`	(n_features + 1, n_components) softmax gate weights (last column is 0). Softmax gate only.
`gate_weights_, gate_means_, gate_covariances_`	Gaussian gate parameters pi_k, c_k, Gamma_k. Gaussian gate only.
`converged_, n_iter_, lower_bound_`	Whether EM converged, iterations run, and best mean log-likelihood.
`n_features_in_`	Number of input features seen during fit.

Methods

name	meaning
`fit(X, y)`	Estimate parameters by EM (best of n_init runs). Returns self.
`fit_predict(X, y)`	Fit and return the hard expert assignment (argmax responsibility).
`predict(X)`	Conditional mean E[y\|x] = sum_k g_k(x) a_k . x_tilde.
`predict_proba(X)`	Gating weights g_k(x); shape (n_samples, n_components); rows sum to 1.
`predict_expert(X)`	Index of the highest-weight expert for each input.
`score_samples(X, y)`	Per-sample conditional log-likelihood log p(y_i \| x_i).
`score(X, y)`	Mean conditional log-likelihood (higher is better).
`bic(X, y) / aic(X, y)`	Information criteria on the conditional model (lower is better).

BayesianGaussianMixtureOfExperts

BayesianGaussianMixtureOfExperts(n_components=2, gate='gaussian', *,
    covariance_type='diag', weight_prior_type='dirichlet',
    weight_concentration_prior=None, n_importance_samples=2000, expert_reg=1e-3,
    mean_precision_prior=1.0, tol=1e-4, reg_covar=1e-6, max_iter=300, n_init=1,
    init_params='kmeans', random_state=None, warm_start=False, verbose=0)

Full mean-field variational Bayes: approximate posterior distributions over all parameters, an evidence lower bound (ELBO), and automatic expert pruning. The Gaussian gate is fully conjugate with a diagonal (Normal-Gamma) or full-covariance (Normal-Inverse-Wishart gate + Matrix-Normal-Wishart experts) posterior, and a Dirichlet, Dirichlet-process or Pitman-Yor weight prior; the softmax gate uses a Bohning variational-EM (full VB experts, point gate). Scalar or multivariate y. The method effective_n_components() reports the number of surviving experts.

Parameters

name	meaning
`n_components`	int, default=2. Maximum number of experts; redundant ones are pruned to weight ~0.
`gate`	{'gaussian','softmax'}, default='gaussian'. 'gaussian' is fully conjugate VB; 'softmax' is a Bohning variational-EM.
`covariance_type`	{'diag','full'}, default='diag'. 'full' gives a Normal-Inverse-Wishart Gaussian gate and matching Matrix-Normal-Wishart (full-covariance) experts (BNP-GLLiM); 'diag' is Normal-Gamma with diagonal experts.
`weight_prior_type`	{'dirichlet','dirichlet_process','pitman_yor'}, default='dirichlet'. 'dirichlet_process' is a truncated stick-breaking (BNP) prior; 'pitman_yor' is the full two-parameter process (discount sigma>0, estimated by importance sampling). Both select the number of experts automatically.
`weight_concentration_prior`	float or None, default=None (= 1/n_components). Dirichlet / DP concentration; small values encourage pruning.
`n_importance_samples`	int, default=2000. Monte-Carlo samples for the Pitman-Yor (alpha, sigma) importance-sampling update; ignored for the other weight priors.
`expert_reg`	float, default=1e-3. Precision of the Gaussian (ridge) prior on the expert coefficients.
`mean_precision_prior`	float, default=1.0. Prior precision on the Gaussian-gate means.
`tol, reg_covar, max_iter, n_init, init_params, random_state, warm_start, verbose`	Standard controls (see the frequentist estimator).

Attributes

name	meaning
`coef_`	(n_components, n_features + 1) expert coefficients; column 0 is the intercept.
`variances_`	(n_components,) per-expert residual variances sigma_k^2.
`gate_coef_`	(n_features + 1, n_components) softmax gate weights (last column is 0). Softmax gate only.
`gate_weights_, gate_means_, gate_covariances_`	Gaussian gate parameters pi_k, c_k, Gamma_k. Gaussian gate only.
`converged_, n_iter_, lower_bound_`	Whether EM converged, iterations run, and best mean log-likelihood.
`n_features_in_`	Number of input features seen during fit.

Methods

name	meaning
`fit(X, y)`	Estimate parameters by EM (best of n_init runs). Returns self.
`fit_predict(X, y)`	Fit and return the hard expert assignment (argmax responsibility).
`predict(X)`	Conditional mean E[y\|x] = sum_k g_k(x) a_k . x_tilde.
`predict_proba(X)`	Gating weights g_k(x); shape (n_samples, n_components); rows sum to 1.
`predict_expert(X)`	Index of the highest-weight expert for each input.
`score_samples(X, y)`	Per-sample conditional log-likelihood log p(y_i \| x_i).
`score(X, y)`	Mean conditional log-likelihood (higher is better).
`bic(X, y) / aic(X, y)`	Information criteria on the conditional model (lower is better).

Model-selection functions

function	meaning
`MixingMeasureDendrogram(est)`	Dendrogram of the fitted mixing measure (Hai et al., AISTATS 2026): rate-weighted dissimilarity + softmax-weighted merge; n_experts(), dsc(X, y), labels(K). No model sweep.
`merge_truncate_merge(est, radius=0.5)`	Merge-Truncate-Merge (Nguyen et al., 2024, Alg. 1): merge within a radius, truncate small weights, merge again; returns the surviving count.
`slope_heuristic(dims, neg_loglik)`	Dimension-jump penalty calibration (Nguyen et al., 2022): returns (kappa_hat, selected index) at the prescribed 2 kappa_hat.
`compare_selectors(cls, X, y, ...)`	Fit the sweep and the over-specified model; report the number of experts chosen by AIC, BIC, ICL, the slope heuristic, the dendrogram, the DSC, MTM and the ELBO count.

Sampling-based inference

The sampling side of the variational-versus-MCMC comparison (conjugate Gaussian-gated model, univariate x, scalar y). Each returns a dict of parameter chains plus diagnostics.

function	meaning
`gibbs_sampler(x, y, K, ...)`	Conjugate Gibbs sampling: categorical z, Dirichlet weights, Normal-Inverse-Gamma gate and regression updates. Asymptotically exact; a single chain mixes poorly across modes.
`nuts_sampler(x, y, K, ...)`	The No-U-Turn Sampler (Hoffman-Gelman) on the collapsed posterior; hand-coded, finite-difference-checked gradients, dual-averaging step size. Exact but typically explores one mode.
`parallel_tempering(x, y, K, n_chains=8, beta_min=0.02, ...)`	Parallel tempering with a NUTS kernel per inverse temperature and standard exchange swaps; the cold chain hops between well-separated modes.
`sparse_mixture_sampler(x, y, K=8, e0=0.01, ...)`	Sparse finite mixture MCMC (Malsiner-Walli et al., 2016): overfitted K with a sparse Dirichlet prior; returns the posterior of the number of ACTIVE experts K+ and relabeled chains.
`posterior_bootstrap(x, y, K, n_boot=200, ...)`	Nonparametric posterior bootstrap (Fong et al., 2019): embarrassingly parallel Dirichlet-weighted EM replicates with random restarts; covers separated modes without a Markov chain.
`vb_warm_start(x, y, vi_class, sampler='pt'\|'sfm', ...)`	The recommended hybrid pipeline: a variational fit with Dirichlet-process pruning supplies the mode, the effective K and the warm start (hot chains start at label permutations of it), then parallel tempering or the sparse-mixture sampler adds mode-complete uncertainty with a much shorter warmup.
`compare_inference(x, y, K, ...)`	Run everything (plus the variational estimator) on one dataset and report the trade-off table empirically: slope posteriors, VI-vs-Gibbs sd ratio, mode hops, K+ posterior, bootstrap mode coverage, timings.
`count_mode_hops(trace, modes)`	Transitions of a scalar chain between the nearest of the given mode locations.

API reference

GaussianMixtureOfExperts

Parameters

Attributes

Methods

BayesianGaussianMixtureOfExperts

Parameters

Attributes

Methods

Model-selection functions

Sampling-based inference

See also