Multinomial mixture of experts: MM training and expert selection
Two live experiments on mixture-of-experts classifiers: a minorization-maximization trainer whose likelihood provably cannot decrease, and a contaminated setting where choosing the number of experts from a single fit beats a full sweep over candidate orders. Everything runs client-side.
Multinomial mixture of experts: MM training and expert selection
Two interactive pieces. The first is the softmax-gated multinomial-logistic model and its batch minorization-maximization training. The second is a contaminated binary-expert setting where a dendrogram criterion is compared against the three standard likelihood criteria.
1 · Batch MM training ascends the likelihood at every step
Softmax gate over K experts, each a multinomial-logistic classifier, fitted by the quadratic Bohning-type MM surrogate with closed-form gate and expert updates. Every iteration is guaranteed not to decrease the observed-data log-likelihood, so the curve below can never go down.
2 · A contaminant that fools the likelihood criteria
Two clean binary-regression experts on x in [0,1], plus a fraction ε of contaminated points drawn as a sharp spike near the right boundary and labelled y=1. The likelihood criteria pay for the spike with an extra expert; the dendrogram criterion can merge it back. Drag ε to 0 and all four agree.
What is verified. At ε=0.10 the three likelihood criteria recover the true K₀=2 in 0 of 12 replications at every sample size tested, at mean selected order 3.00 to 3.08, while on clean data all three return K=2. The dendrogram criterion is consistent: its recovery frequency rises with n, measured over 12 replications per cell as 0.50 at n=1000, 0.67 at n=3000, 0.92 at n=10000 and 0.92 at n=30000, with the mean selected order falling from 2.67 to 2.17. Selection is a large-sample statement, so slide n up and the gap widens.
Reading the small-n end. Below about n=3000 the dendrogram criterion misses K₀ a third to a half of the time. That is the criterion behaving as its theory says rather than a defect: the height term carries the structural signal, but it competes against the merged log-likelihood multiplied by log n, and that likelihood term is only flat once n is large. Its spread along the merge path falls from about 2.1 nats at n=1000 to 0.57 at n=10000, which is exactly when the heights take over and recovery approaches 1.