Md. Asif Uddin
    Problem I.6.B04

    A depth at which the first layer receives nothing

    counterexample▲▲▲

    Six significant figures; fp32's smallest normal is 1.17549435e-38.

    STATEMENT

    Backpropagation is exact. Show that exactness is not sufficiency: exhibit a network on which the gradient reaching the first layer is correct to the last bit and numerically zero, and establish the depth at which that happens. Then show that the obvious remedy — larger weights — makes it worse.

    GIVEN

    A chain of scalar sigmoid units, the simplest object with the property under study:

    a(0)=0.5,z(ℓ)=w a(ℓ−1),a(ℓ)=σ(z(ℓ))a^{(0)} = 0.5, \qquad z^{(\ell)} = w\,a^{(\ell-1)}, \qquad a^{(\ell)} = \sigma(z^{(\ell)})

    for ℓ=1…L\ell = 1 \dots L, with one shared weight ww and no biases. The adjoint leaving the loss is normalised to 11.

    FIND

    A bound on ∣∂L/∂a(0)∣|\partial L/\partial a^{(0)}| in terms of LL alone; the measured ratio at L=10L = 10 and L=40L = 40 for w=1w = 1; the smallest LL at which the ratio falls below fp32’s smallest normal number; and the effect of raising ww.

    STRATEGY

    Apply the recurrence (I.6.3) down the chain and collect factors. Each layer contributes two: the weight, and the slope of the nonlinearity at the stored pre-activation. The second factor has a ceiling that does not depend on anything trainable, and a product of numbers below a ceiling is bounded by the ceiling raised to the depth.

    SOLUTION

    Step 1 — the bound. For the logistic function σ′=σ(1−σ)\sigma' = \sigma(1-\sigma), maximised where σ=1/2\sigma = 1/2:

    max⁡zσ′(z)=12(1−12)=14\max_z \sigma'(z) = \tfrac{1}{2}\left(1 - \tfrac{1}{2}\right) = \tfrac{1}{4}

    The run confirms σ′(0)=0.250000\sigma'(0) = 0.250000. Now walk the chain back. Each layer multiplies the travelling adjoint by ww and by σ′(z(ℓ))\sigma'(z^{(\ell)}), so

    ∂L∂a(0)=∏ℓ=1Lw σ′(z(ℓ)),∣∂L∂a(0)∣≤(∣w∣4)L\frac{\partial L}{\partial a^{(0)}} = \prod_{\ell=1}^{L} w\,\sigma'(z^{(\ell)}), \qquad \left|\frac{\partial L}{\partial a^{(0)}}\right| \le \left(\frac{|w|}{4}\right)^{L}

    With w=1w = 1 this is 4−L4^{-L}, and nothing about the data, the loss or the training procedure can raise it. It is a property of the architecture.

    Step 2 — the measurement. Propagating the adjoint explicitly:

    depthmeasured ratiobound 4−L4^{-L}
    12.350037×10−12.350037 \times 10^{-1}2.5×10−12.5 \times 10^{-1}
    25.340838×10−25.340838 \times 10^{-2}6.25×10−26.25 \times 10^{-2}
    56.079863×10−46.079863 \times 10^{-4}9.765625×10−49.765625 \times 10^{-4}
    103.483116×10−73.483116 \times 10^{-7}9.536743×10−79.536743 \times 10^{-7}
    201.143096×10−131.143096 \times 10^{-13}9.094947×10−139.094947 \times 10^{-13}
    401.231153×10−261.231153 \times 10^{-26}8.271806×10−258.271806 \times 10^{-25}

    The measured values sit below the bound at every depth, as they must, and the gap widens because the units are not all sitting at their maximum slope.

    Step 3 — the depth where it stops being a small number. fp32’s smallest normal value is 1.17549435×10−381.17549435 \times 10^{-38}. The first depth at which the ratio falls below it is

    L=59,ratio 5.901063×10−39L = 59, \qquad \text{ratio } 5.901063 \times 10^{-39}

    At that depth the gradient of the first layer is not small. It is unrepresentable: it enters the subnormal range, where fp32 has almost no precision left, and one more layer flushes it to exactly zero. The parameter receives an update of zero and stays at its initialisation for the whole of training, while the last layers train perfectly well. The loss falls, then stalls, and nothing about the curve says which layers stopped.

    Step 4 — why more weight makes it worse. The bound (∣w∣/4)L(|w|/4)^L suggests ∣w∣>4|w| > 4 should grow the adjoint rather than shrink it. Measured:

    wwdepth 10depth 40
    1.03.483116×10−73.483116 \times 10^{-7}1.231153×10−261.231153 \times 10^{-26}
    2.02.954702×10−62.954702 \times 10^{-6}1.227533×10−231.227533 \times 10^{-23}
    4.05.486035×10−115.486035 \times 10^{-11}1.536491×10−441.536491 \times 10^{-44}
    4.51.449225×10−121.449225 \times 10^{-12}3.182489×10−513.182489 \times 10^{-51}

    Doubling ww to 2 helps a little. Raising it to 4 makes the gradient at depth 40 eighteen orders of magnitude smaller than at w=1w = 1. The bound is not wrong; it is not tight, and the reason is that the two factors are not independent. A large ww drives zz away from zero, and a sigmoid away from zero is saturated, so σ′(z)\sigma'(z) collapses. The weight enters the product linearly and takes the slope down exponentially with it.

    This is what makes the failure a counterexample rather than a difficulty. There is no setting of the free parameter that rescues it: small ww loses to the factor ww, large ww loses to saturation, and the best case is a shallow optimum near w=2w = 2 that still vanishes, only more slowly.

    Answer

    ∣∂L/∂a(0)∣≤(∣w∣/4)L|\partial L/\partial a^{(0)}| \le (|w|/4)^{L}, so with w=1w = 1 the adjoint is bounded by 4−L4^{-L} regardless of training.

    Measured at w=1w = 1: 3.483116×10−73.483116 \times 10^{-7} at depth 10, and 1.231153×10−261.231153 \times 10^{-26} at depth 40. The ratio falls below fp32’s smallest normal at L=59L = 59, where it is 5.901063×10−395.901063 \times 10^{-39}.

    Raising ww does not help: at depth 40, w=4w = 4 gives 1.536491×10−441.536491 \times 10^{-44}, worse than w=1w = 1 by eighteen orders of magnitude, because a large weight saturates the sigmoid and its slope falls faster than the weight rises.

    Check — sanity

    Every measured ratio is below its bound. 4−L4^{-L} was derived from σ′≤1/4\sigma' \le 1/4 with no reference to the values, so a measurement above it would mean the propagation is wrong rather than that the bound is loose.

    Depth 1 nearly attains the bound. 0.2350040.235004 against 0.250.25: the first unit sees z=0.5z = 0.5, close enough to zero that its slope is close to maximal. The later units see larger zz, saturate more, and the product falls away from the bound — which is why the gap at depth 40 is two orders of magnitude.

    The arithmetic of the exponent checks out. 4−59=8.6×10−364^{-59} = 8.6 \times 10^{-36}, and the measured 5.9×10−395.9 \times 10^{-39} is about 1500 times smaller — consistent with a mean slope near 0.230.23 rather than 0.250.25, since (0.23/0.25)59≈0.0075(0.23/0.25)^{59} \approx 0.0075.

    Where this breaks

    The chain here is one unit wide, and width changes the constant without changing the mechanism. In a real layer the adjoint is multiplied by (W(ℓ))T(\mat{W}^{(\ell)})^{\mathsf{T}} rather than by a scalar, so the relevant quantity is that matrix’s largest singular value and the product is over matrices rather than numbers. A width-512 layer can have a spectral norm well above one while every individual weight is small, which is why the same architecture can vanish in one initialisation and explode in another.

    The demonstration also uses the logistic function, whose 1/41/4 ceiling is what makes the bound so sharp. ReLU has slope exactly 11 wherever it is active, so this particular argument does not apply to it at all — and the failure returns in a different form, as dead units whose slope is exactly 00 rather than merely small. The general lesson is about products of Jacobians, and the specific number 1/41/4 belongs to one activation function.

    Nothing here is an argument against depth. It is an argument that depth needs a route the product cannot close, which is exactly (I.6.5): a residual block contributes I+JF\mat{I} + \mat{J}_F, and the identity term keeps a path of gain one open however many blocks are stacked.

    Variation

    Replace σ\sigma with tanh⁡\tanh, whose derivative is 1−tanh⁡21 - \tanh^2 and whose ceiling is 11 rather than 1/41/4. Predict the new bound, then find the depth at which the ratio underflows fp32 and compare it with 59. Does a four-times better ceiling buy four times the depth?

    Draws on