I.11.X09

An identity path can still be cancelled

counterexample▲▲△

For $y=x+F(x)$ take $F(x)=-x$ . Compute the output and Jacobian. Now consider a residual branch changing shape from $(N,16,32,32)$ to $(N,32,16,16)$ . Can it be added directly to the input?

Hint

Differentiate the sum, and check every axis before adding tensors.

Solution

The output is zero and the Jacobian is $I-I=0$ . Thus the identity term in (I.11.8) does not give a positive lower bound on the total gradient. It creates a route that can help training, but another route can cancel it.

The second addition is invalid: channel and spatial dimensions differ. One possible shortcut is a $1$ by $1$ projection from 16 to 32 channels with stride two and no padding. It produces $(N,32,16,16)$ . Its derivative is the projection’s Jacobian rather than an identity. The alignment of the two branches still needs checking.

Draws on

I.11.P03 — Depth grows a receptive field, but does not guarantee that the model uses it.