Every loss is a negative log-likelihood

STATEMENT

Derive squared error as the negative log-likelihood of a Gaussian observation model with fixed variance, and cross-entropy as the negative log-likelihood of a categorical one. Track every constant that appears, and state exactly which ones may be discarded and why.

GIVEN

Gaussian model. Observations are generated as $y_i = f(x_i;\theta) + \varepsilon_i$ with $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$ independent, $\sigma^2$ fixed and not learned. The density is

Categorical model. A prediction is a distribution $\vec{q} = (q_1, \dots, q_K)$ on the simplex, the observation is a class $c \in \{1,\dots,K\}$, and

$p(c \mid x, \theta) = \prod_{k=1}^{K} q_k^{\,y_k}, \qquad y_k = \mathbb{1}[k = c]$

FIND

$-\log p(\mathcal{D} \mid \theta)$ for each model, reduced to a loss, with every discarded term named.

STRATEGY

Write the likelihood of the whole dataset, take a logarithm to turn the product into a sum, negate, then separate the terms containing $\theta$ from those that do not. Only the first group can affect the minimiser, and the second group is where the constants go.

SOLUTION

Part 1 — the Gaussian case

Step 1 — the dataset likelihood. Independence turns a joint density into a product:

writing $r_i = y_i - f(x_i;\theta)$ for the $i$-th residual.

Step 2 — take the logarithm. A product becomes a sum, and $\log(ab) = \log a + \log b$ splits each factor:

The exponential and the logarithm are inverses, so the second term simplifies outright:

Step 3 — separate. The first term does not depend on $i$, so summing it $n$ times gives a single constant:

Step 4 — negate.

Step 5 — account for $A$ and $B$ precisely.

Term $A$ is additive and $\theta$-free. Adding a constant to a function shifts its graph vertically and moves no stationary point: $\nabla_\theta (g(\theta) + A) = \nabla_\theta g(\theta)$. So $A$ may be dropped without changing the minimiser or any gradient.

Term $B$ is a positive multiplicative constant. Since $\sigma^2 > 0$ we have $B > 0$, and $\arg\min_\theta\, Bg(\theta) = \arg\min_\theta g(\theta)$ for any $B > 0$. So $B$ may be dropped from the objective. It may not be dropped from the gradient if the learning rate is fixed, because $\nabla(Bg) = B\nabla g$ — dropping $B$ rescales every step by $1/B$, which is a change of effective learning rate and nothing more.

Step 6 — conclude.

$\blacksquare$ Minimising squared error is maximum likelihood under a fixed-variance Gaussian.

Part 2 — the categorical case

Step 7 — one example. The indicator exponent means all but one factor is raised to the power zero:

$p(c\mid x,\theta) = \prod_{k=1}^{K} q_k^{\,y_k} = q_c$

Step 8 — take the logarithm and negate.

using $\log(a^b) = b\log a$ on each factor. The right-hand side is exactly the cross-entropy $H(\vec{y}, \vec{q})$ of Definition 8.

Step 9 — the dataset. By independence again,

$-\log p(\mathcal{D}\mid\theta) = \sum_{i=1}^{n} H(\vec{y}_i, \vec{q}_i)$

$\blacksquare$

Note the asymmetry with Part 1: there is no constant to discard. The categorical density has no normalising factor outside the probabilities themselves, because $\sum_k q_k = 1$ is built into the parameterisation. Every term of the cross-entropy depends on $\theta$.

What the correspondence costs

Reading Steps 1–6 backwards is the uncomfortable direction. If minimising MSE is maximum likelihood under a Gaussian, then choosing MSE asserts a Gaussian, whether or not anyone intended to assert anything. Three properties come with that assertion:

Symmetry. $p(\varepsilon) = p(-\varepsilon)$, so over-prediction and under-prediction cost the same. False whenever the two have different consequences.

Constant variance. One $\sigma^2$ for every $x$. False whenever the noise scales with the signal, which is the ordinary situation for counts, prices and concentrations.

Unbounded support. $\varepsilon$ may take any real value, so $y$ may too. False for anything bounded — a probability, a proportion, a nonnegative count.

Chapter VII.2 will meet count data where all three fail at once, and will need a negative binomial likelihood instead. The route there is exactly this derivation run forwards with a different density.

Check — sanity

The Gaussian result reproduces the known optimum. For a constant model $f = \mu$, minimising $\sum(y_i - \mu)^2$ gives $\mu = \bar{y}$, the sample mean — which is the maximum-likelihood estimate of a Gaussian mean. Two routes, one answer.

Dropping $B$ is exactly a learning-rate change. With $\sigma^2 = 1$, $B = \tfrac12$. Training on $\sum r_i^2$ at rate $\eta$ and on $\tfrac12\sum r_i^2$ at rate $2\eta$ produce identical parameter sequences. This is checkable in three lines of code, and it is why the factor of $\tfrac12$ in front of squared losses is a convention rather than a claim.

The categorical result reduces to the binary case. At $K = 2$ with $q_2 = 1 - q_1$ and $\vec{y}$ one-hot, Step 8 becomes $-[y_1\log q_1 + (1-y_1)\log(1-q_1)]$, which is binary cross-entropy — the identity established in I.1.B04, recovered here as a special case rather than assumed.

The units are consistent. A log-likelihood is dimensionless (a log of a probability), and so is cross-entropy. But $\sum r_i^2$ carries the square of $y$‘s units — the mismatch is absorbed by $1/(2\sigma^2)$, whose units are the inverse square of $y$‘s. Discarding $B$ therefore discards the dimensional bookkeeping too, which is a small reason MSE values are hard to interpret across problems.