Md. Asif Uddin
    I.6.X10

    Why a skip connection cannot lose the gradient

    proof▲▲▲

    For a residual block y=x+F(x)\vec{y} = \vec{x} + F(\vec{x}), derive the Jacobian and then the adjoint recurrence for a stack of LL such blocks. Show that the product cannot vanish for want of a route, and state precisely what the identity term does and does not guarantee.

    Hint

    Differentiate the sum term by term. One of the two derivatives does not depend on FF at all.

    Solution

    The Jacobian. Differentiating y=x+F(x)\vec{y} = \vec{x} + F(\vec{x}) coordinate by coordinate:

    ∂yj∂xi=δij+∂Fj∂xi,soJ=I+JF\frac{\partial y_j}{\partial x_i} = \delta_{ij} + \frac{\partial F_j}{\partial x_i}, \qquad \text{so} \qquad \mat{J} = \mat{I} + \mat{J}_F

    which is (I.6.5).

    The recurrence. By (I.6.2) the adjoint entering block ℓ\ell from block ℓ+1\ell + 1 is

    xˉ(ℓ)=xˉ(ℓ+1)(I+JFℓ)T=xˉ(ℓ+1)+xˉ(ℓ+1)JFℓT\bar{\vec{x}}^{(\ell)} = \bar{\vec{x}}^{(\ell+1)}\left(\mat{I} + \mat{J}_{F_\ell}\right)^{\mathsf{T}} = \bar{\vec{x}}^{(\ell+1)} + \bar{\vec{x}}^{(\ell+1)}\mat{J}_{F_\ell}^{\mathsf{T}}

    Across the stack. Expanding the product over LL blocks:

    ∂L∂x(0)=xˉ(L)∏ℓ=1L(I+JFℓ)T\frac{\partial L}{\partial \vec{x}^{(0)}} = \bar{\vec{x}}^{(L)}\prod_{\ell=1}^{L}\left(\mat{I} + \mat{J}_{F_\ell}\right)^{\mathsf{T}}

    Multiplying out gives 2L2^L terms, one for each subset of blocks whose JFJ_F is taken. Exactly one of those terms takes the identity from every block:

    xˉ(L)IL=xˉ(L)\bar{\vec{x}}^{(L)}\mat{I}^L = \bar{\vec{x}}^{(L)}

    The adjoint that left the loss arrives at the first block undiminished, whatever the FℓF_\ell are and however many there are. In the plain stack of I.6.B04 every term carried a factor below 1/41/4, so the whole product decayed; here one term carries no factor at all.

    What this does not promise. Three things.

    It is not a guarantee that the gradient is useful. The identity path transmits the adjoint of the loss with respect to the block’s input, not with respect to its parameters — the parameter gradients still flow through JFℓ\mat{J}_{F_\ell}, and a block whose FF is saturated still learns nothing. The signal reaches it; whether it can act on it is separate.

    It does not prevent explosion. I+JF\mat{I} + \mat{J}_F has eigenvalues 1+λi1 + \lambda_i, and if JF\mat{J}_F has eigenvalues near +1+1 the product grows like 2L2^L. Residual stacks are normally paired with normalisation for exactly this reason, and the pairing is not decoration.

    It says nothing about optimisation beyond the gradient’s magnitude. A route for the adjoint is a necessary condition for training a deep stack, not a sufficient one, and the empirical fact that residual networks train at depths plain stacks cannot is a result rather than a corollary of this algebra.

    Draws on