Interactive demo · Foundations of Deep Learning · Appendix C

Lagrange multipliers and the KKT conditions

Maximise f(x) = c0 − (x − m)TS(x − m) in the plane, subject to a constraint g(x) = 0 or g(x) ≥ 0, where g is a line or a circle. At a constrained maximum the gradient of f has no component along the constraint curve, so ∇f + λ∇g = 0 (C.3). Move the probe along the curve and watch the tangential part of ∇f (pink) vanish exactly at the stationary points; switch to an inequality and watch λ become zero whenever the unconstrained maximum is feasible.

Constraint

Line normal angle φ, n = √2(cos φ, sin φ)

Offset b in g = nTx − b

Radius r in g = r² − |x|²

Centre of f m = (m₁, m₂)

Curvatures and rotation of f q₁, q₂, θ

Probe position on the curve

Try these

The line constraint is g = nTx − b with |n| = √2, so φ = 45°, b = 1 is the book's x₁ + x₂ − 1. Rescaling g rescales λ (2g has multiplier λ/2): only the product λ∇g is meaningful. Equality: every stationary point of f along the curve is a stationary point of the Lagrangian (hollow dots); the solution is the one with the largest f. Inequality (g ≥ 0 is the shaded side): if the unconstrained maximum m is feasible it is the answer with λ = 0; otherwise the answer is on the boundary with λ > 0.

In the lecture notes