Standard KL risk blows up wherever the fitted mixture q touches zero; the h-lifted divergence floors the ratio so the loss stays finite and still controls estimation at an O(1/(k+2) + 1/sqrt(n)) rate. Use the ▸ buttons beside each control (or the Run the experiments launchers) to auto-play; everything runs client-side.

Risk Bounds via the h-Lifted KL Divergence

To estimate a density f on a compact domain by a finite mixture q, the natural loss is the Kullback-Leibler divergence. But KL blows up to infinity wherever the fitted mixture touches zero, so classical risk bounds must assume every density stays bounded away from 0, an assumption real mixtures and real targets violate. This paper's fix is the h-lifted KL divergence, KL_h(f‖q) = ∫(f+h) log[(f+h)/(q+h)], which floors the ratio by a positive lift h and stays finite for all continuous densities, while still controlling estimation: the maximum h-lifted likelihood estimator obeys an O(1/(k+2) + 1/√n) excess-risk bound with no strict-positivity assumption. Sharpen or shift the mixture components to open a valley where q→0 and watch the two losses diverge.

Mixture density estimationh-lifted KL divergenceOracle risk boundsBregman divergence

Read the paper Full text Code (R / Python)

Loss geometry: where KL diverges, KL_h stays bounded

top: target f (teal) vs mixture q (amber) with lift floor h · bottom: the two loss integrands on the same x

excess h-lifted risk vs sample size n

Sharpness 204 Shift 0.00 Lift h (γ) 0.30 Components k 4 Sample n 150

target f mixture q KL integrand KL_h integrand

A 1D toy on the compact domain 𝒳 = [0, 1], following the paper's Section 4.2 setup. The target f is one of the paper's two not-strictly-positive densities: f₂ is a tent that touches 0 at x = 1/2, and f₁ is two plateaus with a zero gap on (2/5, 3/5). The mixture q = ∑_j π_j Beta(x; a_j, b_j) is built from the paper's beta base class (k components, weights following f). As you raise Sharpness, the beta spikes narrow and deep valleys open where q→0: a red star marks the deepest valley inside f's support. In the bottom strip the standard KL integrand f log(f/q) (pink) spikes toward +∞ there, so ∫f log(f/q) is unbounded, exactly why classical KL risk bounds must forbid q from touching 0. The h-lifted integrand (f+h) log[(f+h)/(q+h)] (blue) stays bounded because the lift h (the constant floor set by the γ slider) keeps the denominator positive. The right panel shows the h-lifted excess risk E{KL_h(f‖f_k,n)} − KL_h(f‖𝒞) of the maximum h-lifted likelihood estimator decaying at the paper's O(1/(k+2) + 1/√n) rate as n and k grow, while the standard-KL risk has no finite bound once q has a zero. (Faithful surrogates for a first-draft build: h is the uniform-density lift scaled by γ (a tunable positive floor, sub-probability when γ < 1), q's weights follow f rather than a full MM refit, and the right-panel curve is the theoretical rate with sampling scatter (New data reseeds), not a rerun of the MM / nonlinear-least-squares fit. The divergence definitions, the tail blow-up, and the bounded h-lift are computed exactly on the grid.)

Run the experiments

Every animation runs live in your browser. Click a button to run that experiment on the demo (it scrolls up and starts); drag any control to take over. Nothing is downloaded, it is generated on the fly.

Sharpen q until KL blows up

Narrow the beta components until deep valleys open where q approaches 0; the pink standard-KL integrand spikes toward infinity while the blue h-lifted integrand stays bounded.

Move the q-valley

Slide the whole comb of component centres so the near-zero tail (the red star) sweeps across the support of the target f, changing where the KL loss diverges.

Raise the lift floor h

Increase the positive lift h added to numerator and denominator; watch KLh shrink and stay finite, the mechanism that lets the risk bound drop the strict-positivity assumption.

Add mixture components

Grow k and see the h-lifted excess-risk curve drop by the c1/(k+2) approximation term from Corollary 6.

Grow the sample size n

Sweep n from 8 to 5000 (log scale) and watch the h-MLLE excess risk decay along the 1/sqrt(n) rate, while the standard-KL risk has no finite bound.

Both not-strictly-positive targets

Tour the paper's two Section 4.2 targets: the tent f2 that touches 0 at x = 1/2 and the notch f1 with a zero gap, each a density that classical KL bounds cannot handle.

Resample the risk scatter

Redraw fresh sampling noise around the theoretical excess-risk rate, the scatter of measured h-lifted likelihood values the paper fits by nonlinear least squares.

The idea in three steps

1 · Diagnose

KL blows up at zeros

Where the fitted mixture q touches 0 but the target f is positive, the integrand f log(f/q) diverges. Classical mixture risk bounds dodge this only by assuming every density is bounded away from 0, which finite mixtures and compactly supported targets violate.

2 · Lift

Floor the ratio with h

KL_h(f‖q) = ∫(f+h) log[(f+h)/(q+h)] adds a positive lift h to numerator and denominator. It is a genuine Bregman divergence, is bounded for all continuous densities, and satisfies a Pinsker-like L₁/L₂ sandwich, so it still controls estimation error.

3 · Bound

An O(1/(k+2)+1/√n) rate

The maximum h-lifted likelihood estimator, computed by a minorization-maximization algorithm, obeys E{KL_h(f‖f_k,n)} − KL_h(f‖𝒞) ≤ c₁/(k+2) + c₂/√n, with no strict-positivity assumption on f or the components.

For the Bregman-divergence construction, the boundedness and Pinsker-type properties, Theorem 5 and Corollary 6, the MM algorithm, and the beta-mixture experiments, see Risk Bounds for Mixture Density Estimation on Compact Domains via the h-Lifted Kullback-Leibler Divergence (Chiu Chong, H. D. Nguyen & TrungTin Nguyen, Transactions on Machine Learning Research 2024).