Problems

STATEMENT

Five residuals are given, one of them an outlier. Compute MSE, MAE and Huber with $\delta = 1$ on all five. Then determine, for each loss, what share of the total the outlier alone contributes, and what derivative each loss sends back for it.

GIVEN

The residuals $r_i = \hat{y}_i - y_i$:

The three losses, as per-residual functions before any averaging:

with $\delta = 1$.

FIND

The per-residual contribution under each loss; the three totals and means; the outlier’s percentage share of each total; and $\mathrm{d}\ell/\mathrm{d}r$ evaluated at $r = 4.0$ for each.

STRATEGY

Build one table column at a time rather than one row at a time. Each column is a single function applied five times, so a slip is visible as a break in the column’s pattern — whereas working row by row hides it.

SOLUTION

Step 0 — which branch of Huber each residual takes. The branch is decided by $|r|$ against $\delta = 1$, so check all five before computing anything:

Four take the quadratic branch; the fifth alone takes the linear one. Settling this first means the rest is arithmetic with no case analysis mixed in.

Step 1 — the squared column. Squaring each residual:

Note the sign has already vanished — squaring is why MSE never needs an absolute value. Summing:

$\sum r_i^2 = 0.25 + 0.64 + 0.04 + 0.09 + 16.00 = 17.02$

$\text{MSE} = \frac{17.02}{5} = 3.4040$

Step 2 — the absolute column. Taking magnitudes:

$0.5,\quad 0.8,\quad 0.2,\quad 0.3,\quad 4.0$

$\sum |r_i| = 0.5 + 0.8 + 0.2 + 0.3 + 4.0 = 5.8$

$\text{MAE} = \frac{5.8}{5} = 1.1600$

Step 3 — the Huber column. For the four quadratic-branch residuals, $\tfrac12 r^2$, which is just half of Step 1’s numbers:

For the outlier, the linear branch with $\delta = 1$:

$\mathrm{Huber}_1(4.0) = (1)\!\left(4.0 - \tfrac12(1)\right) = 4.0 - 0.5 = 3.5000$

Summing:

$\sum \mathrm{Huber} = 0.1250 + 0.3200 + 0.0200 + 0.0450 + 3.5000 = 4.0100$

$\overline{\mathrm{Huber}} = \frac{4.0100}{5} = 0.8020$

The completed table.

| $r$ | $r^{2}$ | $|r|$ | $\mathrm{Huber}_1$ | |---|---|---|---| | $0.5$ | $0.2500$ | $0.5000$ | $0.1250$ | | $-0.8$ | $0.6400$ | $0.8000$ | $0.3200$ | | $0.2$ | $0.0400$ | $0.2000$ | $0.0200$ | | $-0.3$ | $0.0900$ | $0.3000$ | $0.0450$ | | $4.0$ | $16.0000$ | $4.0000$ | $3.5000$ | | sum | $17.0200$ | $5.8000$ | $4.0100$ | | mean | $3.4040$ | $1.1600$ | $0.8020$ |

Step 4 — the outlier’s share. Divide the outlier’s own contribution by each total:

Step 5 — the derivatives, which are the operative quantity. The loss value is a diagnostic; the derivative is what training actually uses. Differentiating each per-residual loss at $r = 4.0$:

For Huber on the linear branch, $\mathrm{d}/\mathrm{d}r\,[\delta(r - \tfrac12\delta)] = \delta$:

$\mathrm{Huber}_1'(4.0) = \delta = 1.0000$

So the outlier pulls eight times harder under MSE than under either of the other two.

Step 6 — what the two measurements say together. Compare the share of the value with the share of the pull:

Huber’s value share, $87.3\%$, is close to MSE’s — because the four inliers are small and halving them makes the outlier look even more dominant. But its derivative is MAE’s. That combination is the entire design: Huber reports a large loss when there is a large error, while refusing to let that error dominate the step. A loss’s value and a loss’s gradient are different measurements and can disagree, and only the second one moves the model.

Step 7 — the four inliers alone. Removing the outlier and averaging over $n = 4$:

MSE fell by a factor of $3.4040/0.2550 = 13.35$; MAE by $1.1600/0.4500 = 2.58$. One point in five moved MSE more than thirteenfold.

Check — sanity

Huber sits between half-MSE and MAE, as its definition forces. Per residual, $\mathrm{Huber}_1(r) \le \tfrac12 r^2$ always (equality on the quadratic branch, strictly below on the linear one) and $\mathrm{Huber}_1(r) \le |r|$ always. Check the totals: $4.0100 \le \tfrac12(17.02) = 8.51$ ✓ and $4.0100 \le 5.80$ ✓.

The two Huber branches meet. At $|r| = \delta = 1$ the quadratic branch gives $\tfrac12(1)^2 = 0.5$ and the linear branch gives $(1)(1 - 0.5) = 0.5$. Equal, so the function is continuous. Their derivatives also meet: $r = 1$ against $\delta = 1$. Continuity of the derivative is what makes Huber usable by a gradient method, and it is not automatic — it is what the $-\tfrac12\delta$ term in the linear branch is for.

MAE’s ordering is preserved. The residual magnitudes ordered $0.2 < 0.3 < 0.5 < 0.8 < 4.0$, and the MAE column reproduces that order exactly, since $|\cdot|$ is monotone in magnitude. A column out of order would signal a transcription error.

Units check on the derivative. $\mathrm{d}(r^2)/\mathrm{d}r$ has the units of $r$; $\mathrm{d}|r|/\mathrm{d}r$ is dimensionless. That is why MSE’s pull grows with the error and MAE’s cannot — a dimensional argument reaching the same conclusion as the arithmetic.

STATEMENT

Derive squared error as the negative log-likelihood of a Gaussian observation model with fixed variance, and cross-entropy as the negative log-likelihood of a categorical one. Track every constant that appears, and state exactly which ones may be discarded and why.

GIVEN

Gaussian model. Observations are generated as $y_i = f(x_i;\theta) + \varepsilon_i$ with $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$ independent, $\sigma^2$ fixed and not learned. The density is

Categorical model. A prediction is a distribution $\vec{q} = (q_1, \dots, q_K)$ on the simplex, the observation is a class $c \in \{1,\dots,K\}$, and

$p(c \mid x, \theta) = \prod_{k=1}^{K} q_k^{\,y_k}, \qquad y_k = \mathbb{1}[k = c]$

FIND

$-\log p(\mathcal{D} \mid \theta)$ for each model, reduced to a loss, with every discarded term named.

STRATEGY

Write the likelihood of the whole dataset, take a logarithm to turn the product into a sum, negate, then separate the terms containing $\theta$ from those that do not. Only the first group can affect the minimiser, and the second group is where the constants go.

SOLUTION

Part 1 — the Gaussian case

Step 1 — the dataset likelihood. Independence turns a joint density into a product:

writing $r_i = y_i - f(x_i;\theta)$ for the $i$-th residual.

Step 2 — take the logarithm. A product becomes a sum, and $\log(ab) = \log a + \log b$ splits each factor:

The exponential and the logarithm are inverses, so the second term simplifies outright:

Step 3 — separate. The first term does not depend on $i$, so summing it $n$ times gives a single constant:

Step 4 — negate.

Step 5 — account for $A$ and $B$ precisely.

Term $A$ is additive and $\theta$-free. Adding a constant to a function shifts its graph vertically and moves no stationary point: $\nabla_\theta (g(\theta) + A) = \nabla_\theta g(\theta)$. So $A$ may be dropped without changing the minimiser or any gradient.

Term $B$ is a positive multiplicative constant. Since $\sigma^2 > 0$ we have $B > 0$, and $\arg\min_\theta\, Bg(\theta) = \arg\min_\theta g(\theta)$ for any $B > 0$. So $B$ may be dropped from the objective. It may not be dropped from the gradient if the learning rate is fixed, because $\nabla(Bg) = B\nabla g$ — dropping $B$ rescales every step by $1/B$, which is a change of effective learning rate and nothing more.

Step 6 — conclude.

$\blacksquare$ Minimising squared error is maximum likelihood under a fixed-variance Gaussian.

Part 2 — the categorical case

Step 7 — one example. The indicator exponent means all but one factor is raised to the power zero:

$p(c\mid x,\theta) = \prod_{k=1}^{K} q_k^{\,y_k} = q_c$

Step 8 — take the logarithm and negate.

using $\log(a^b) = b\log a$ on each factor. The right-hand side is exactly the cross-entropy $H(\vec{y}, \vec{q})$ of Definition 8.

Step 9 — the dataset. By independence again,

$-\log p(\mathcal{D}\mid\theta) = \sum_{i=1}^{n} H(\vec{y}_i, \vec{q}_i)$

$\blacksquare$

Note the asymmetry with Part 1: there is no constant to discard. The categorical density has no normalising factor outside the probabilities themselves, because $\sum_k q_k = 1$ is built into the parameterisation. Every term of the cross-entropy depends on $\theta$.

What the correspondence costs

Reading Steps 1–6 backwards is the uncomfortable direction. If minimising MSE is maximum likelihood under a Gaussian, then choosing MSE asserts a Gaussian, whether or not anyone intended to assert anything. Three properties come with that assertion:

Symmetry. $p(\varepsilon) = p(-\varepsilon)$, so over-prediction and under-prediction cost the same. False whenever the two have different consequences.

Constant variance. One $\sigma^2$ for every $x$. False whenever the noise scales with the signal, which is the ordinary situation for counts, prices and concentrations.

Unbounded support. $\varepsilon$ may take any real value, so $y$ may too. False for anything bounded — a probability, a proportion, a nonnegative count.

Chapter VII.2 will meet count data where all three fail at once, and will need a negative binomial likelihood instead. The route there is exactly this derivation run forwards with a different density.

Check — sanity

The Gaussian result reproduces the known optimum. For a constant model $f = \mu$, minimising $\sum(y_i - \mu)^2$ gives $\mu = \bar{y}$, the sample mean — which is the maximum-likelihood estimate of a Gaussian mean. Two routes, one answer.

Dropping $B$ is exactly a learning-rate change. With $\sigma^2 = 1$, $B = \tfrac12$. Training on $\sum r_i^2$ at rate $\eta$ and on $\tfrac12\sum r_i^2$ at rate $2\eta$ produce identical parameter sequences. This is checkable in three lines of code, and it is why the factor of $\tfrac12$ in front of squared losses is a convention rather than a claim.

The categorical result reduces to the binary case. At $K = 2$ with $q_2 = 1 - q_1$ and $\vec{y}$ one-hot, Step 8 becomes $-[y_1\log q_1 + (1-y_1)\log(1-q_1)]$, which is binary cross-entropy — the identity established in I.1.B04, recovered here as a special case rather than assumed.

The units are consistent. A log-likelihood is dimensionless (a log of a probability), and so is cross-entropy. But $\sum r_i^2$ carries the square of $y$‘s units — the mismatch is absorbed by $1/(2\sigma^2)$, whose units are the inverse square of $y$‘s. Discarding $B$ therefore discards the dimensional bookkeeping too, which is a small reason MSE values are hard to interpret across problems.

STATEMENT

Derive $\partial L/\partial \vec{z} = \vec{p} - \vec{y}$ for softmax followed by cross-entropy. Do not quote the result: obtain the softmax Jacobian from the quotient rule, apply the chain rule through it in full, and show explicitly which terms cancel and why. Then instantiate the whole calculation on three logits.

GIVEN

Logits $\vec{z} \in \R^{K}$, predictions and target

For the numeric part, $\vec{z} = (2, 1, 0)$ and $c = 1$ (the first entry).

FIND

$\partial p_i / \partial z_j$ for all $i, j$; then $\partial L/\partial z_j$; then both evaluated at the given logits.

STRATEGY

Differentiate softmax by the quotient rule, splitting into the $i = j$ and $i \neq j$ cases because the numerator depends on $z_j$ only in the first. Then push the loss gradient through the Jacobian and use $\sum_k y_k = 1$ — that one identity is what collapses a $K \times K$ matrix product to a subtraction.

SOLUTION

Part 1 — the softmax Jacobian

Step 1 — name the denominator. Let

$S = \sum_{j=1}^{K} e^{z_j}, \qquad\text{so}\qquad p_i = \frac{e^{z_i}}{S}$

The key observation before any differentiation: $S$ depends on every logit. So $p_i$ depends on $z_j$ even when $i \neq j$ — through the denominator alone. That is the whole source of the off-diagonal terms.

Differentiating $S$:

$\frac{\partial S}{\partial z_j} = \frac{\partial}{\partial z_j}\sum_{m} e^{z_m} = e^{z_j}$

since every term but the $j$-th is constant in $z_j$.

Step 2 — the diagonal case, $i = j$. Apply the quotient rule $\left(\frac{u}{v}\right)' = \frac{u'v - uv'}{v^2}$ with $u = e^{z_i}$ and $v = S$. Here $u' = e^{z_i}$ and $v' = e^{z_i}$:

Split the fraction into two, so each piece becomes a $p$:

Step 3 — the off-diagonal case, $i \neq j$. Now $u = e^{z_i}$ does not depend on $z_j$, so $u' = 0$ and only the denominator contributes:

Step 4 — combine. Using the Kronecker delta $\delta_{ij} = 1$ if $i=j$ and $0$ otherwise, the two cases are one formula:

This is the softmax Jacobian of The softmax Jacobian · 0.MC.06, derived here rather than cited, because the two cancellations that follow depend on knowing where each factor came from.

Check it reproduces both branches: at $i=j$, $p_i(1 - p_i)$ ✓; at $i \ne j$, $p_i(0 - p_j) = -p_ip_j$ ✓.

Part 2 — the chain rule through it

Step 5 — differentiate the loss with respect to the probabilities.

$\frac{\partial L}{\partial p_i} = \frac{\partial}{\partial p_i}\left(-\sum_k y_k \log p_k\right) = -\frac{y_i}{p_i}$

only the $k = i$ term surviving.

Step 6 — assemble. The chain rule for a vector-to-vector map sums over the intermediate index (The chain rule · 0.MC.03):

Step 7 — the first cancellation. The factor $p_i$ from the Jacobian meets the $1/p_i$ from the loss and they cancel exactly:

This is the step that makes the whole thing work, and it is why softmax and cross-entropy are paired rather than chosen independently. Had the loss been anything but a logarithm, the $1/p_i$ would not have appeared and nothing would cancel — which is exactly what problem I.4.B06 shows happening with MSE.

Step 8 — expand the bracket and use $\sum_i y_i = 1$.

The first sum has exactly one non-zero term, at $i = j$, giving $y_j$. In the second, $p_j$ does not depend on $i$, so it factors out:

$\blacksquare$

Where each hypothesis was used. Step 7 needed the loss to be logarithmic. Step 8 needed $\vec{y}$ to sum to one — note it did not need $\vec{y}$ to be one-hot, so the result holds for soft targets and therefore for label smoothing unchanged. That is worth recording, because it is often stated as requiring one-hot labels and does not.

Part 3 — the numbers

Step 9 — softmax at $\vec{z} = (2,1,0)$. Subtract the maximum first, which changes nothing and prevents overflow (Log-sum-exp · 0.NU.02):

$S = 1.000000 + 0.367879 + 0.135335 = 1.503215$

Sum: $0.6652 + 0.2447 + 0.0900 = 0.9999 \approx 1$ ✓ (the last digit is rounding).

Step 10 — the loss. $\vec{y} = (1,0,0)$, so

$L = -\log p_1 = -\log 0.6652 = 0.4076 \text{ nats}$

Step 11 — the full Jacobian, entry by entry. Using $p_i(\delta_{ij} - p_j)$:

and so on, giving

Step 12 — the long route, to confirm the short one. $\partial L/\partial \vec{p} = (-1/0.6652,\ 0,\ 0) = (-1.5032,\ 0,\ 0)$. Multiplying by the Jacobian, only the first row contributes:

Step 13 — the short route.

Identical to Step 12, entry for entry.

Check — sanity

The gradient sums to zero. $-0.3348 + 0.2447 + 0.0900 = -0.0001 \approx 0$. This must hold: softmax is invariant to adding a constant $\lambda$ to every logit, so the directional derivative along $(1,1,\dots,1)$ is zero, which is exactly $\sum_j \partial L/\partial z_j = 0$. A gradient not summing to zero means an arithmetic slip.

The signs are right. The true class has a negative gradient, so gradient descent raises its logit; every other class has a positive gradient, so their logits are lowered. That is the behaviour the loss should produce, read directly off the sign pattern.

The magnitudes are bounded. Every entry of $\vec{p} - \vec{y}$ lies in $[-1, 1]$, since $p_j \in (0,1)$ and $y_j \in \{0,1\}$. The gradient can never explode, whatever the logits. Contrast with the MSE-through-sigmoid gradient of I.4.B06, which is bounded too — but by a number that shrinks to zero exactly when it is needed.

The Jacobian is symmetric with zero row sums. $J_{12} = J_{21} = -0.1628$, and $0.2227 - 0.1628 - 0.0599 = 0.0000$. Both properties follow from the formula: $p_i(\delta_{ij} - p_j)$ is symmetric because $p_ip_j$ is, and rows sum to $p_i(1 - \sum_j p_j) = p_i(1-1) = 0$.

STATEMENT

Evaluate binary cross-entropy at four logits by two routes: the naive one that forms a probability first, and the stable identity that never does. Derive the identity, show the naive route failing on two ordinary inputs, and state the logit at which each failure begins.

GIVEN

A logit $z \in \R$, a label $y \in \{0,1\}$, and $p = \sigma(z) = 1/(1+e^{-z})$. The naive computation is

$\mathrm{BCE}_{\text{naive}} = -\big[\,y\log p + (1-y)\log(1-p)\,\big]$

Test at $(z, y) \in \{(2,1),\ (0,1),\ (-5,1),\ (-50,1),\ (-800,1),\ (800,0)\}$. IEEE double precision: $e^{x}$ overflows for $x > 709.78$, and $\sigma(z)$ rounds to exactly $1.0$ once $e^{-z} < 2^{-53} \approx 1.11\times10^{-16}$.

FIND

The stable identity; both routes’ values at each test point; and the two thresholds at which the naive route breaks.

STRATEGY

Derive the identity by substituting $\sigma$ into the definition and simplifying until no probability appears — only $z$ and a logarithm of something safely near one. Then evaluate both routes and watch where they part.

SOLUTION

Part 1 — deriving the stable form

Step 1 — substitute for $y = 1$. With $p = 1/(1+e^{-z})$:

using $-\log(1/a) = \log a$.

Step 2 — substitute for $y = 0$. First simplify $1 - p$:

so

Step 3 — one formula for both. Combining Steps 1 and 2:

$\mathrm{BCE}(z,y) = -zy + z(1-y)\cdot 0 + \dots$

More carefully — write the two cases and look for the pattern:

Step 4 — the remaining danger. This is exact, but $e^{-z}$ still overflows for $z < -709.78$. Fix it by pulling out the larger of the two terms inside the logarithm. For $z < 0$ write $1 + e^{-z} = e^{-z}(1 + e^{z})$, so

$\log(1+e^{-z}) = -z + \log(1+e^{z})$

Substituting into Step 3’s formula for $z < 0$ and combining with the $z \ge 0$ case gives the symmetric form

Why this one is safe. The exponential’s argument is $-|z| \le 0$, so $e^{-|z|} \in (0, 1]$ and can never overflow. It can underflow to $0$, but then $\log(1+0) = 0$ exactly, which is the correct limit rather than an error. Every other term is elementary arithmetic on $z$ itself.

Part 2 — the two routes side by side

Step 5 — verifying agreement where both work. At the three safe points the two routes agree to ten decimal places, with $|{\text{diff}}| \le 2.78\times10^{-17}$ — one unit in the last place of a double. So (I.4.5) is not an approximation; it is the same number computed differently.

Step 6 — the first failure, $z = -800$, $y = 1$. The naive route needs $\sigma(-800) = 1/(1 + e^{800})$. But $e^{800}$ exceeds the largest double ($1.798\times10^{308}$) and overflows, raising OverflowError before any logarithm is reached.

The stable route computes

which is correct: a logit of $-800$ with target $1$ is wrong by $800$ nats.

Step 7 — the second failure, $z = 800$, $y = 0$. Here $e^{-800}$ underflows to $0$, so $\sigma(800)$ evaluates to exactly $1.0$. The naive route then needs $\log(1 - 1.0) = \log(0) = -\infty$, and Python raises ValueError. In a framework that returns $-\infty$ silently instead, the loss becomes inf, every gradient becomes nan, and the entire model is destroyed in one step with no message.

The stable route gives $\max(800,0) - (800)(0) + \log(1+e^{-800}) = 800 + 0 = 800.000000$.

Step 8 — the thresholds.

Overflow. $e^{-z}$ overflows when $-z > 709.78$, so the naive route fails for $z < -709.78$.

Rounding to one. $\sigma(z)$ becomes exactly $1.0$ once $e^{-z}$ falls below the spacing of doubles near $1$, that is $e^{-z} < 2^{-53}$, giving $z > 53\ln 2 = 36.74$. This is the more dangerous of the two, because $z = 37$ is an entirely ordinary logit — it appears whenever a model becomes confident — and the failure is a silent $-\infty$ rather than a raised exception.

In float32 the corresponding threshold is $z > 24\ln 2 = 16.6$, which is reached routinely within the first epoch of ordinary training.

Check — sanity

The identity gives the right answer at $z = 0$. There $p = 0.5$ and the loss should be $-\log 0.5 = \log 2 = 0.693147$. The formula gives $\max(0,0) - 0 + \log(1+e^{0}) = \log 2$ ✓.

Large-$|z|$ behaviour is linear, as it must be. For $z \to -\infty$ with $y=1$, $-\log\sigma(z) = \log(1+e^{-z}) \approx -z$. The table confirms it: $z = -50 \Rightarrow 50.000000$ and $z = -800 \Rightarrow 800.000000$, both equal to $|z|$ to six decimals. A loss growing linearly rather than exponentially in the logit is exactly why cross-entropy’s gradient stays bounded.

The two failures are the two ends of the same problem. Underflow of $e^{-|z|}$ is harmless — $\log(1+0) = 0$ is correct. Overflow of $e^{+|z|}$ is fatal. The identity’s whole content is arranging that only the harmless one can occur.

Symmetry check. $\mathrm{BCE}(z, 1) = \mathrm{BCE}(-z, 0)$: predicting logit $z$ for a positive should cost the same as predicting $-z$ for a negative. From the formula at $z = 2$: $\mathrm{BCE}(2,1) = 2 - 2 + \log(1+e^{-2}) = 0.126928$ and $\mathrm{BCE}(-2,0) = 0 - 0 + \log(1+e^{-2}) = 0.126928$ ✓.

STATEMENT

A binary dataset holds $950$ negatives and $50$ positives. Derive the class weights that make the two classes contribute equal gradient mass, compute them, verify the loss scale is unchanged, and compare with the effective-number weighting of Cui et al. (2019).

GIVEN

$n_- = 950$, $n_+ = 50$, so $N = 1000$ and $K = 2$. The weighted empirical risk is

where $c(i)$ is example $i$‘s class. Assume each example’s loss has comparable magnitude, so that a class’s contribution is proportional to its count times its weight.

FIND

Weights $w_-$ and $w_+$ equalising the two contributions; their ratio; a check that the mean weight is $1$; and the same quantities under effective-number weighting at $\beta = 0.999$.

STRATEGY

Write the condition “the two classes contribute equally” as one equation, add the normalisation “the average weight is one” as a second, and solve the pair. Two conditions, two unknowns — the weights are then determined, not chosen.

SOLUTION

Step 1 — the unweighted imbalance. Without weights, class $c$ contributes $n_c$ of the $N$ terms, so its share of the gradient is

The negatives outvote the positives nineteen to one. A model that predicts “negative” always achieves $95\%$ accuracy and a low loss, and the gradient pushing it away from that solution is one-nineteenth of the gradient holding it there.

Step 2 — the equalisation condition. Class $c$‘s weighted contribution is $n_c w_c$. Requiring the two to be equal:

n_- w_- = n_+ w_+ \tag{i}

Step 3 — the normalisation condition. Requiring the mean weight over the dataset to be $1$, so the weighted loss has the same scale as the unweighted one and the learning rate need not be retuned:

\frac{1}{N}\sum_{i=1}^{N} w_{c(i)} = \frac{n_- w_- + n_+ w_+}{N} = 1 \tag{ii}

Step 4 — solve. Substitute (i) into (ii). Writing $M$ for the common value $n_-w_- = n_+w_+$:

$\frac{M + M}{N} = 1 \quad\Longrightarrow\quad 2M = N \quad\Longrightarrow\quad M = \frac{N}{2}$

Then from $n_c w_c = M$ for each class,