Interactive demo · Foundations of Deep Learning · Chapter 19

A denoising autoencoder learns the score

Data x lie near a curve; each is corrupted to x̃ = x + σε, and a small network y(x̃) is trained to undo the corruption by minimising ‖y(x̃) − x‖² (19.3), here in your browser. The best possible denoiser is the posterior mean E[x|x̃], and Tweedie's formula says E[x|x̃] − x̃ = σ²∇ ln pσ(x̃), with pσ the density of the corrupted data. Because the data here are a mixture of small Gaussians along the curve, that field is known exactly: compare the learned field with the optimal one, and watch both turn from "back to the curve" (small σ) to "towards the middle" (large σ).

Data

Noise level σ

Hidden units per layer H

Adam steps (batch 64)

Learning rate

Show

Run

Try these

Network: y(x̃) = x̃ + W3 tanh(W2 tanh(W1x̃ + b1) + b2) + b3, trained with Adam on fresh samples every step. Data: an equal mixture of N(ck, τ²I) with centres along the curve (τ = 0.03 or 0.04). Arrows are drawn at half their length. Error = E‖f − f*‖²/E‖f*‖² over 1000 corrupted points, with f = y − x̃ and f* the optimal residual.

In the lecture notes