Interactive demo · Foundations of Deep Learning · Chapter 16

Independent component analysis: unmixing two sources

Two independent sources z are mixed linearly, x = Az, and only x is observed. PCA whitening (16.24) removes all second-order structure but leaves an unknown rotation: every rotation of white data is still white. ICA fixes the rotation by maximum likelihood with a non-Gaussian prior on the sources, here the heavy-tailed p(z) = 1/(π cosh z) of (16.56). The lower-left panel plots the log likelihood along all rotations: its peaks are the separating angles. Run the natural-gradient ascent, change the sources, and see when the method works and when it cannot.

Sources (500 samples)

Prior used by ICA

Mixing directions a₁, a₂ (degrees)

Unmixing rotation θ (degrees)

Run

Try these

Whitening: y = L−1/2UT(x − x̄). Unmixing: ẑ = W y, average log likelihood ℓ(W) = ln|det W| + (1/N)Σn,j ln p(ẑnj). Natural gradient: W ← W + η(I − (1/N)Σn φ(ẑn)ẑnT)W, η = 0.1, φ = −(ln p)′ (tanh for the sech prior, z3 for the light-tailed prior p ∝ e−z4/4). The slider θ starts the run from W = R(θ); after a run, θ shows the rotation part of W. Gold ticks: the separating angles.

In the lecture notes