Proposition 1I.6.P0122 of 100 in the corpus
Backpropagation is the chain rule with the intermediate results kept.
There is no separate learning algorithm inside a network. There is a graph, one traversal forwards to compute values, and one backwards to accumulate derivatives.
Demonstration
Take a small network as a chain of operations:
u = Wx h = σ(u) ŷ = Vh L = ℓ(ŷ, y)The forward pass evaluates these left to right, keeping each intermediate. Backpropagation evaluates the derivative of L with respect to each parameter, right to left, by the chain rule:
∂L/∂V = (∂L/∂ŷ)(∂ŷ/∂V)
∂L/∂W = (∂L/∂ŷ)(∂ŷ/∂h)(∂h/∂u)(∂u/∂W)Every factor in those products is local — it depends only on one operation and the values that operation saw. So the algorithm is: walk backwards, and at each node multiply the derivative arriving from above by the node’s own local derivative. Nothing more.
Two consequences are worth stating because they explain most of what training costs.
The backward pass costs about what the forward pass costs. Each node does a comparable amount of arithmetic in each direction. Training is roughly three times inference, not a thousand times.
The intermediates must be stored. ∂h/∂u needs u. That is why memory
during training scales with depth times batch times width, why activation
checkpointing exists — recompute intermediates rather than store them — and why
a model that runs comfortably at inference may not train on the same hardware.
The name automatic differentiation is the accurate one. Reverse-mode AD is a general technique for any composition of differentiable operations, and neural networks are simply one place it is applied. There is no learning in it.
Corollary
If a gradient is zero where it should not be, the fault is at one node, and the chain rule says which: the product is zero if any factor is. A saturated sigmoid, a rectifier stuck negative, a detached tensor or an in-place operation that destroyed a stored intermediate — each shows up as one factor going to zero, and the backward pass is short enough to check by hand.
Sources
Depends on
Used by
- I.6.P02 — The graph is the program, and the forward pass must keep what the backward pass will read.
- I.7.P01 — The gradient supplies the direction and never the distance.
- I.8.P02 — The normalisation formula is incomplete until its axes are named.
- I.9.P02 — Backpropagation through time accumulates shared gradients through a product of temporal Jacobians.
- I.12.P01 — An autoencoder is supervised learning whose target is its own input.
- I.13.P03 — The reparameterisation trick moves the randomness off the path of the gradient.
- II.6.P02 — Layers write into a shared stream rather than replacing it.
Problems using this
- I.6.B01 — One full backward pass through a 2-2-1 networknumeric▲△△
- I.6.B02 — Deriving the layer recurrence from the graphsymbolic▲▲△
- I.6.B03 — A shape audit of the backward pass, with a batchshape▲▲△
- I.6.B04 — A depth at which the first layer receives nothingcounterexample▲▲▲
- I.6.B05 — What the backward pass costs, in arithmetic and in memorycomplexity▲▲△
- I.6.X01 — One extra local Jacobian rescales every gradient below itnumeric▲△△
- I.6.X02 — The pairing that collapses a dense Jacobian to a subtractiongradient▲▲△
- I.6.X03 — The vector–Jacobian product, without forming the Jacobiansymbolic▲▲△
- I.6.X04 — A tied weight that receives half its gradientcounterexample▲▲△
- I.6.X05 — Where the batch dimension goes, and when you want it keptshape▲△△
- I.6.X06 — The one point where ReLU has no derivativelimit▲▲△
- I.6.X07 — An in-place operation that corrupts a stored intermediatecounterexample▲▲△
- I.6.X08 — Choosing a mode is arithmetic, not preferencecomplexity▲▲△
- I.6.X09 — What it costs when the sink is not a scalarcomplexity▲▲△
- I.6.X10 — Why a skip connection cannot lose the gradientproof▲▲▲
- I.BOOK.B01 — XOR, from the perceptron to one step of trainingnumeric▲▲△
- I.BOOK.B07 — Cross-entropy through a graph convolutiongradient▲▲△