Interactive demo · Foundations of Deep Learning · Chapter 8

Forward mode and reverse mode on an evaluation trace

Code that evaluates a function is a graph of elementary operations vi (8.50)–(8.56). Forward mode carries a tangent v̇i = ∂vi/∂xj along with each value, summing over the parents (8.57): one pass gives one column of the Jacobian. Reverse mode first records the values, then sends adjoints v̄i = ∂f/∂vi backwards, summing over the children (8.69): one pass gives one row, that is the whole gradient of one output. The numbers on the edges are the local derivatives ∂vi/∂vj.

Function

Mode

Seed

Evaluation step

Inputs

Try these

Highlighted node: the one just evaluated. In reverse mode the first half of the steps is the forward sweep that stores every vi (the memory cost of reverse mode); the second half is the backward sweep. The soft ReLU is ζ(a) = ln(1 + ea) (8.46), whose derivative is the logistic sigmoid.

In the lecture notes