Md. Asif Uddin

    Chapter 4 I.4

    Loss Functions

    Every loss is a negative log-likelihood under some noise model, and choosing a loss is choosing that model whether or not anyone says so.

    A loss is the sentence 'here is what wrong means', written so a derivative exists.

    How this chapter is built

    M3Load-bearing

    The content is mathematics. Understanding is demonstrated by computation, not recall.

    basics11/11what the words mean
    concept2/2what to picture
    theory4/4why it works, and when it does not
    mathematics15/15derive it, then compute it
    practice9/9build it, break it, read the papers

    Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

    Before you start

    The problem

    Training needs one scalar to descend. Which scalar is not a detail: it fixes the implied noise model, it fixes what the optimal prediction even is, and — as this chapter's counterexample shows — it decides whether any gradient at all reaches a model at the point where it is most confidently wrong.

    A loss surface, drawn as contoursNested contour rings around a deep minimum on the left and a shallower one on the right, with a saddle between them. A dashed line runs from a starting point downhill into the nearer basin, not necessarily the better one.loss surface over two parametersgloballocalstartThe loss is the only statement of the objective the optimiser can read. What it omits is not optimised.
    Fig. 4 — A loss surface drawn as contours, with a deep basin and a shallower one. Descent finds the nearer minimum, which is not always the better one.

    What this chapter covers

    • Loss and risk
    • Mean squared error
    • Mean absolute error
    • Huber loss
    • Binary cross-entropy
    • Categorical cross-entropy
    • Logits and log-sum-exp
    • Class weighting
    • Label smoothing

    Apparatus

    The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

    Entropy 0.IT.01 · Cross-entropy 0.IT.02 · The exponential family 0.PR.06 · The softmax Jacobian 0.MC.06 · Log-sum-exp 0.NU.02

    Notation

    • ℒThe loss
    • 𝔼Expectation
    • softmaxThe normalised exponential, applied row-wise unless stated
    • ℋEntropy, in nats unless bits are named
    • σThe logistic function, or a standard deviation
    • ∇Gradient operator
    • VarVariance
    • BBatch size

    Definitions

    Definition 1Loss function

    A loss ℓ : 𝒴 × 𝒴 → ℝ₊ assigns one non-negative number to a single prediction–target pair, smaller meaning better. It must be differentiable in its first argument almost everywhere, because that argument is what the optimiser moves. Nothing in the data selects it.

    • ℒThe loss

    Definition 2Reduction

    The reduction is the function turning per-example losses into the one scalar that is differentiated — almost always the arithmetic mean. The choice is invisible in the formula and decides how much each example contributes to the gradient.

    Definition 3Mean squared error

    MSE = (1/n) Σ (ŷᵢ − yᵢ)². Its derivative with respect to a residual is proportional to that residual, so a residual twice as large pulls twice as hard and one ten times as large pulls ten times as hard.

    Definition 4Mean absolute error

    MAE = (1/n) Σ |ŷᵢ − yᵢ|. Its derivative with respect to a residual is ±1 regardless of magnitude, so every example pulls equally hard whatever its error. It is not differentiable at zero.

    Definition 5Huber loss

    Huber_δ(r) = ½r² for |r| ≤ δ and δ(|r| − ½δ) otherwise. It is quadratic near zero and linear beyond δ, so it is differentiable everywhere and its derivative is bounded by δ.

    Definition 6Logit

    A logit is an unnormalised real-valued score, before any function maps it to a probability. Logits live on all of ℝ; probabilities live in (0,1). Every numerically sound loss in this chapter is computed from logits, never from probabilities.

    Definition 7Softmax

    softmax(z)ᵢ = e^{zᵢ} / Σⱼ e^{zⱼ}, mapping a logit vector to the probability simplex. It is invariant to adding a constant to every logit, which is why it can be computed after subtracting the maximum without changing the answer.

    • softmaxThe normalised exponential, applied row-wise unless stated

    Definition 8Cross-entropy

    For a target distribution p and a predicted q, H(p, q) = −Σ p(x) log q(x). With p one-hot it collapses to −log q(c) for the true class c, so all but one term of the sum vanish.

    • ℋEntropy, in nats unless bits are named

    Definition 9Label smoothing

    Label smoothing replaces the one-hot target by (1 − ε)·onehot + ε/K applied uniformly. It caps the reward for confidence: the loss is then minimised at a finite logit gap rather than at an infinite one.

    Formal results

    I.4.T1

    Losses as negative log-likelihoods

    Minimising MSE is exactly maximising the likelihood of a Gaussian observation model with fixed variance; minimising cross-entropy is exactly maximising the likelihood of a categorical model. In both cases the loss is −log p(y | x, θ) up to an additive constant and a positive scale, neither of which changes the minimiser.

    What this does not promise

    It does not promise the implied noise model is true of your data. Choosing MSE asserts symmetric, constant-variance, unbounded Gaussian errors — an assertion that is false for counts, for bounded quantities, and for anything heavy-tailed. The correspondence is a fact about the algebra, not evidence about the world, and it is most dangerous when nobody realises an assertion was made.

    I.4.T2

    Bayes-optimal predictors

    Under squared loss the risk-minimising prediction at x is the conditional mean 𝔼[Y | X = x]; under absolute loss it is a conditional median; under cross-entropy it is the true conditional distribution p(· | x). Each is derived by differentiating the expected loss and setting it to zero.

    What this does not promise

    These describe the minimiser over all measurable functions, not over the hypothesis class the model actually searches. A network trained on squared loss approaches the conditional mean only insofar as the mean lies in its class and the optimiser finds it — two conditions I.1.T1 and I.3.T2 already warned are separate. They also assume the training distribution equals the deployment one.

    Assumptions

    Each is paired with the problem that shows what its removal costs. An assumption nobody tests is a disclaimer.

    • The implied noise model of the chosen loss matches the data — Gaussian for MSE, categorical for cross-entropy. I.4.X07
    • The reduction is a plain mean, so every example contributes equally to the gradient. I.4.X08

    Propositions

    1. Prop. 1A loss is a choice about what wrong means, and its derivative is the part that matters.Nothing in the data selects a loss. What is chosen is a statement about the relative cost of errors, and the operative half of that statement is the derivative — because a loss's value is reported while its gradient is what moves the model.
    2. Prop. 2Every loss is the negative log-likelihood of some noise model, said or not.Minimising squared error is maximum likelihood under a fixed-variance Gaussian; minimising cross-entropy is maximum likelihood under a categorical. The correspondence is an identity, so choosing a loss asserts a distribution — and the assertion is most dangerous when nobody notices it was made.
    3. Prop. 3The loss and the output nonlinearity must be chosen together, because only one pairing cancels.Cross-entropy contributes a factor that is the exact reciprocal of the softmax or sigmoid derivative, so the two cancel and the gradient is p − y. Squared loss contributes no such factor, so the saturating derivative survives and drives the gradient toward zero precisely where the model is most wrong.
    4. Prop. 4The reduction is part of the loss, and it silently sets the learning rate.Turning per-example losses into one scalar is a choice that never appears in the loss's formula. Summing rather than averaging multiplies every gradient by the batch size, and dividing by the wrong count under masking changes the effective learning rate from step to step.

    Numbered equations

    1. ℓ_MSE(r) = ½r², dℓ/dr = r

      The pull grows without bound with the error.

      (I.4.1)

    2. ℓ_MAE(r) = |r|, dℓ/dr = sign(r)

      Every example pulls equally hard, however wrong it is.

      (I.4.2)

    3. Huber_δ(r) = ½r² if |r| ≤ δ, else δ(|r| − ½δ)

      Quadratic where precision matters, linear where outliers live.

      (I.4.3)

    4. ∂L/∂z = p − y

      Softmax and cross-entropy compose into the simplest gradient in the book.

      (I.4.4)

    5. BCE(z, y) = max(z, 0) − z·y + log(1 + e^{−|z|})

      The same number as the naive form, computed where the arithmetic survives.

      (I.4.5)

    6. w_c = N / (K · n_c)

      Weights that make every class contribute the same gradient mass.

      (I.4.6)

    Worked problems

    7/5 problems6/4 variants10/10 exercisesquota met, and enforced

    1. I.4.B01 — Three losses on the same five residualsnumeric▲△△
    2. I.4.B02 — Every loss is a negative log-likelihoodsymbolic▲▲△
    3. I.4.B03 — Why softmax and cross-entropy compose to p − ygradient▲▲▲
    4. I.4.B04 — The same loss, computed where the arithmetic survivesnumeric▲▲△
    5. I.4.B05 — Weights that equalise what each class contributesprobability▲▲△
    6. I.4.B06 — Where squared loss stops sending a gradientcounterexample▲▲△
    7. I.4.B07 — From a logit tensor to one scalar, with every shape namedshape▲▲△

    The whole problem set, with the exercises →

    Practice

    • Squared loss through a sigmoid stops sending gradient exactly where the model is most wrong. At logit z = −10 with target 1, MSE's gradient is 4.54e-5 while cross-entropy's is 0.99996 — a ratio of 22,029. At z = −20 the ratio is 4.85e+8. The model is as wrong as it can be and MSE reports almost nothing.
    • Naive binary cross-entropy raises an exception on ordinary logits. Computing p = sigmoid(z) then log(p) at z = −800 overflows exp(800) to infinity, giving p = 0 and log(0); at z = +800 with target 0, sigmoid rounds to exactly 1.0 and log(1 − 1) fails. The stable identity returns 800.000000 in both cases.