Interactive demo · Statistical Learning: Theory and Methods · Chapter 14

Non-negative matrix factorisation of digits: parts against principal components

The 300 × 256 matrix X of grey levels of handwritten 2s (all entries ≥ 0) is approximated by X ≈ WH with W (300 × r) and H (r × 256) non-negative, fitted by maximising the Poisson log-likelihood L = ∑ [xij log(WH)ij − (WH)ij] with Lee and Seung's multiplicative updates. Each image is a sum of the r basis images (rows of H) with non-negative weights, so nothing can be cancelled: the basis images become strokes, parts of a 2. Principal components, fitted to the same data, use signed global patterns that cancel each other. Change the rank and the number of sweeps, and pick a digit to reconstruct.

Rank r

Multiplicative sweeps

Digit to reconstruct

Try these

Data: the first 300 training 2s of the zip-code data, grey levels rescaled to [0, 1]. NMF starts from fixed random values in [0.5, 1.5]; each sweep updates W and then H by (14.74). PCA is fitted to the centred data (mean image plus r components). NMF basis images are each scaled to their own maximum (black = large); PCA images are red for positive and blue for negative loadings.

In the lecture notes