Where squared loss stops sending a gradient

STATEMENT

Construct a classification case in which squared loss produces a vanishing gradient and cross-entropy does not. Derive both gradients with respect to the logit, evaluate them at five logits, and identify precisely which factor is responsible.

GIVEN

A binary classifier emitting a logit $z$, with $p = \sigma(z)$ and target $y = 1$. Two candidate losses:

Recall $\sigma'(z) = \sigma(z)\big(1-\sigma(z)\big) = p(1-p)$ from I.3.B02.

FIND

$\mathrm{d}L/\mathrm{d}z$ for each loss in closed form; both evaluated at $z \in \{-1, -3, -5, -10, -20\}$; and their ratio.

STRATEGY

Differentiate each loss with respect to $p$, then apply the chain rule through $\sigma$. The whole result turns on whether the $\sigma'(z)$ factor survives or cancels, so keep it visible rather than simplifying early.

SOLUTION

Step 1 — MSE’s gradient. Differentiating with respect to $p$ first:

$\frac{\partial L_{\text{MSE}}}{\partial p} = \frac{\partial}{\partial p}\,\tfrac12 (p-y)^2 = (p - y)$

Then the chain rule through the sigmoid (The chain rule · 0.MC.03):

The factor $\sigma'(z)$ is present, and it is the problem. Chapter I.3 established that $\sigma' \le 1/4$ always, and that it collapses toward zero as $|z|$ grows.

Step 2 — BCE’s gradient. Differentiating with respect to $p$:

To see the last equality, put the bracket over a common denominator:

and negating gives $(p-y)/[p(1-p)]$.

Now the chain rule:

The $\sigma'$ factor cancelled exactly. The logarithm in the loss produced a $1/[p(1-p)]$ that met the $p(1-p)$ from the sigmoid. This is the same cancellation as I.4.B03’s Step 7, in the binary case.

Step 3 — compare the two closed forms.

They differ by exactly one factor, $\sigma'(z) \in (0, 1/4]$. So

Step 4 — evaluate. With $y = 1$:

Step 5 — read the two columns. As $z$ becomes more negative the model becomes more wrong: $p \to 0$ while $y = 1$.

Cross-entropy’s gradient grows toward its maximum magnitude of $1$. At $z = -20$ it is $-1.000000$: the loss is shouting.

MSE’s gradient shrinks toward zero. At $z = -20$ it is $-2.06\times10^{-9}$. In float32, where the smallest normal value is about $1.18\times10^{-38}$, this is still representable — but multiplied through a few more layers of I.3’s saturation factors it is not, and in fp16 it underflowed long before.

The model is as wrong as it is possible to be, and MSE reports almost nothing. That is the counterexample.

Step 6 — which factor is responsible. Not the squaring. Substituting $y = 1$ into (a):

$\frac{\mathrm{d}L_{\text{MSE}}}{\mathrm{d}z} = (p-1)\,p(1-p) = -p(1-p)^2$

As $p \to 0$ this behaves as $-p$, and $p = \sigma(z) \to 0$ exponentially in $z$. The culprit is the $\sigma'$ factor that (b) cancels and (a) keeps — exactly the saturation of Chapter I.3, reaching the loss instead of a hidden layer.

Step 7 — the other end, which is the honest caveat. At a point the model already gets right:

| $z$ | MSE $|\mathrm{d}L/\mathrm{d}z|$ | BCE $|\mathrm{d}L/\mathrm{d}z|$ | |---|---|---| | $+5$ | $4.449445\mathrm{e}{-5}$ | $6.692851\mathrm{e}{-3}$ | | $+10$ | $2.060873\mathrm{e}{-9}$ | $4.539787\mathrm{e}{-5}$ |

Both shrink, and MSE shrinks faster. So MSE is not uniformly worse — it is quieter everywhere. The asymmetry that matters is that cross-entropy stays loud where the model is wrong and goes quiet where it is right, while MSE goes quiet in both directions.

Check — sanity

BCE’s gradient is bounded by 1. Every entry satisfies $|p - y| \le 1$ since $p \in (0,1)$ and $y \in \{0,1\}$. The table’s largest is $-1.000000$, approached but not exceeded ✓

The ratio equals $1/[p(1-p)]$ exactly. At $z = -5$: $1/(6.692851\times10^{-3} \times 0.993307) = 150.4$, matching the table’s ratio column — computed from the closed form rather than by dividing the two columns, so it is an independent check.

The two gradients agree where $\sigma'$ is largest. At $z = 0$, $\sigma' = 1/4$, so the ratio would be $4$ — the smallest it can ever be. The table’s smallest ratio, $5.1$ at $z = -1$, is consistent with approaching $4$ as $z \to 0$.

Signs are correct throughout. With $y = 1$ and $p < 1$, both gradients are negative, so descent raises $z$ — which is what should happen when the model under-predicts the positive class.