Md. Asif Uddin
    I.6.X08

    Choosing a mode is arithmetic, not preference

    complexity▲▲△

    Reverse mode is not simply better than forward mode. For f:Rn→Rmf: \R^n \to \R^m, price both, decide which wins for four different tasks, and say what reverse mode pays for its advantage.

    Hint

    A forward sweep propagates a derivative with respect to one input; a reverse sweep propagates the derivative of one output. Count how many of each you need to fill a Jacobian.

    Solution

    The rule. Forward mode carries ∂⋅/∂xi\partial \cdot / \partial x_i for one input ii through the whole graph, so it fills one column of the Jacobian per sweep: nn sweeps. Reverse mode carries ∂yj/∂⋅\partial y_j / \partial \cdot for one output, filling one row per sweep: mm sweeps. That is I.6.T2, and everything else follows from it.

    tasknnmmforwardreversecheaper
    training a network10810^8110810^81reverse
    a scalar of few inputs3131reverse
    sensitivity of an ODE410610^6410610^6forward
    Jacobian of a 102421024^2 layer1024102410241024tie
    one directional derivative10810^810610^6110610^6forward

    Training. nn in the hundred millions, m=1m = 1. Reverse mode is cheaper by a factor of 10810^8 — the entire parameter count. This is the case the field is built around, and it is why “autodiff” and “backpropagation” are used interchangeably even though the first is much more general than the second.

    The last row is the interesting one. Both nn and mm are enormous, yet forward mode wins outright, because only one direction is wanted rather than the whole Jacobian. A single forward sweep seeded with that direction returns Jv\mat{J}\vec{v} — a Jacobian–vector product — at the price of one evaluation. Reverse mode has no way to produce it in fewer than mm sweeps.

    What reverse mode pays. Memory. Forward mode carries one extra value alongside each node and never looks back, so nothing is stored: 32 76832\,768 values live at any moment for the model of I.6.B05. Reverse mode must keep the whole tape to have the local Jacobians on the way back: 1 572 8641\,572\,864 values, 48×48\times more, exactly the depth of the network.

    That is the trade in one line — reverse mode buys a factor of nn in arithmetic with a factor of LL in memory, and the exchange is worth it precisely because n≫Ln \gg L.

    Where neither is cheap. The middle row: a full 1024×10241024 \times 1024 Jacobian costs 1024 sweeps either way. There is no mode that produces a dense Jacobian cheaply, and a method that needs one — some second-order and some sensitivity-analysis methods — is expensive for a reason no implementation detail will fix.

    Draws on