Same prediction, different backward graph

STATEMENT

Compare full backpropagation with a one-step truncation that detaches the state after the second input.

GIVEN

Use I.9.B01’s linear scalar model: $(u,w,b,v,c)=(1,1/2,0,2,0)$, inputs $(1,2,0)$, initial state zero, and final target one. The full forward states are $(1,5/2,5/4)$. There is only a final output loss.

FIND

The prediction and loss in both computations, and their derivatives with respect to $w$ and $u$. Explain the correct finite-difference check for each.

STRATEGY

After detaching $h_2$, replace its computation by the constant $5/2$. Differentiate the remaining one-step function and compare it with I.9.B02.

SOLUTION

The full graph has $h_3=u(w^2+2w)$ at $b=0$. At the stated parameters its gradients are $\partial_w\mathcal L=9$ and $\partial_u\mathcal L=15/4$.

The detached graph instead evaluates

At the original parameters this gives the same $h_3=5/4$, $z_3=5/2$ and loss $9/8$. Its last-state derivative is still $\delta_3=3$, but there is no earlier parameter path:

The derivative with respect to the boundary value itself would be $w\delta_3=3/2$. Detachment stops it from propagating into the calculation that produced that value.

To finite-difference the full graph, perturb $w$ or $u$ and recompute every state. To finite-difference the truncated graph, first compute the boundary state at the original parameters, freeze it at $5/2$, then perturb the parameters only in the final chunk.

Recomputing the boundary during the truncated check reattaches the dependency you intended to remove. Its result would be the full gradient, not a failure of the truncated backward implementation.

Check — sanity

Resetting instead of detaching would set the boundary to zero. Because $x_3=b=c=0$, that would give prediction zero and loss $1/2$, a different forward computation. Detach preserves the value; reset replaces it.