Normalisation changes the computation the optimiser differentiates; regularisation changes which fit training prefers.
Batch and layer normalisation, their backward pass, and the preferences built into weight decay, dropout and early stopping.
How this chapter is built
M3Load-bearing
The content is mathematics. Understanding is demonstrated by computation, not recall.
basics11/11what the words mean
concept2/2what to picture
theory4/4why it works, and when it does not
mathematics15/15derive it, then compute it
practice9/9build it, break it, read the papers
Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.
Chapter I.7 chose a step from a gradient. That gradient still depends on how activations are scaled, and reducing training loss still does not choose a good prediction rule for new data. This chapter separates those problems. Normalisation can also have a regularising effect, but that overlap does not make the mechanisms interchangeable.
Fig. 8 — Seven points fitted exactly and fitted loosely. Regularisation does not improve the fit — it decides which fit you get when many are available.
What this chapter covers
The reduction axes are part of the definition
Batch statistics, learned scale and learned offset
Training and inference are different batch-normalisation computations
Layer normalisation without neighbours in the batch
The backward pass through a mean and a variance
What epsilon preserves, and what it changes
L2 penalties, ridge regression and shrinkage
Why AdamW separates decay from the gradient
Inverted dropout: expectation is preserved, variance is not
Early stopping and label-preserving augmentation
Apparatus
The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.
λA regularisation coefficient, or an eigenvalue where the context is linear algebra
VarVariance
𝔼Expectation
ηLearning rate
∇Gradient operator
Definitions
Definition 1Normalisation group
The entries used together to compute a mean and a variance. For a matrix with examples in rows and features in columns, dense batch normalisation uses a column; layer normalisation over features uses a row. Changing the group changes the function even when the formula is unchanged.
Definition 2Batch normalisation
For each feature during training, subtract the minibatch mean and divide by the square root of the minibatch variance plus a positive epsilon. Then apply a learned scale gamma and offset beta. The variance in this forward computation divides by the group size, not by one less.
Definition 3Learned affine parameters
The scale gamma and offset beta applied after standardisation. They let training choose the useful output scale and location; they are parameters learned by gradients, unlike the mean and variance computed from activations.
Definition 4Running statistics
Stored estimates of a feature's activation mean and variance, collected during training and used by ordinary batch normalisation at inference. They are model state, not parameters with gradients. The estimator convention and averaging rule must be recorded with the checkpoint.
Definition 5Layer normalisation
Standardisation over specified features of one example, followed by learned affine parameters. In the dense case each row supplies its own statistics. The layer uses the same rule during training and inference and needs no running estimates.
Definition 6Regularised objective
The data loss plus a preference term: J(theta) = L_data(theta) + lambda R(theta), with lambda non-negative. A soft penalty discourages some parameter settings; it need not remove them from the hypothesis class or improve the training fit.
Definition 7L2 penalty
One half the squared Euclidean norm of the penalised weights, multiplied by lambda. Its gradient is lambda times those weights. Which parameters are included matters: biases and normalisation scales need not share the same penalty.
Definition 8Decoupled weight decay
A separate subtraction of eta times lambda times the current weights, outside the optimiser's gradient transformation. It equals an L2-penalised update for plain gradient descent with a matched coefficient. The identity does not generally survive momentum or adaptive moments.
Definition 9Inverted dropout
Multiply an activation by an independent Bernoulli mask with keep probability q, then divide by q during training. At inference use the activation unchanged. For a fixed activation its conditional mean is preserved but its variance increases.
Definition 10Early stopping
Select an earlier checkpoint using held-out validation performance. This limits how far training follows the data, rather than adding an explicit penalty. Selecting the checkpoint on the test set spends the independence needed for a final evaluation.
Formal results
I.8.T1
The normalisation Jacobian couples every entry in a group
For a group of m scalars, variance v computed with divisor m, r = sqrt(v + epsilon), and standardised values xhat, the derivative is J_ij = (delta_ij - 1/m - xhat_i xhat_j/m)/r. Thus J times the all-ones vector is zero. For batch normalisation with one shared gamma, an upstream gradient g gives dx_i = (gamma/r)(g_i - mean(g) - xhat_i mean(g xhat)). I.8.B02 derives every term.
What this does not promise
This is a derivative of the training computation, not a proof that training converges or generalises. With positive epsilon the centred scaling direction is not exactly removed. Layer normalisation may have a different gamma for each feature, so multiply each upstream entry by its own gamma before applying this Jacobian.
I.8.T2
Ridge has a unique minimiser even when the design is rank deficient
For J(w) = ||Xw - y||²/(2n) + lambda ||w||²/2 and lambda > 0, the Hessian XᵀX/n + lambda I is positive definite. The unique minimiser solves (XᵀX/n + lambda I)w = Xᵀy/n. Each positive-curvature eigendirection is shrunk relative to least squares by a/(a + lambda), where a is its eigenvalue.
What this does not promise
The result is for a linear predictor with a quadratic penalty on all coordinates shown. It does not make a neural-network objective convex, and it does not say that shrinking a genuine signal improves test error. The scale of lambda changes if the data loss is summed rather than averaged.
I.8.T3
Inverted dropout preserves the first moment, not the function
For a fixed activation h, independent mask M with probability q of being one, and htilde = Mh/q, E[htilde | h] = h and Var(htilde | h) = h²(1 - q)/q. These follow by enumerating the two possible mask values.
What this does not promise
A nonlinear downstream function need not satisfy E[f(htilde)] = f(h). The deterministic inference network is not an exact average of all masked networks, and preserving the activation's mean is not a generalisation guarantee.
Assumptions
Each is paired with the problem that shows what its removal costs. An assumption nobody tests is a disclaimer.
The normalisation group contains variation worth retaining. A dense batch-normalisation group of one scalar has zero variance and loses that scalar's input dependence.I.8.X05
Frozen running statistics describe the activations encountered at inference. A changed input distribution can break that agreement without producing a runtime error.I.8.X06
A dropout mask is sampled independently of the activation, with a strictly positive keep probability. Otherwise the stated conditional expectation need not hold.I.8.X07
An augmentation declared label-preserving actually preserves the target.I.8.X09
For a dense batch-normalisation layer receiving one example per group, subtracting the mean subtracts the input itself. The standardised activation is zero, the output is beta, and the input gradient is zero. A spatial batch-normalisation group may still contain many pixels with batch size one, so the relevant count is the reduction size, not the batch label alone.
Inverted dropout at q = 0.5 keeps an activation h as either zero or 2h during training. Applying the same random mask at inference makes repeated predictions differ; multiplying the inference activation by q as well incorrectly halves its expected scale.
Normalising over the last dimension of a channels-first image tensor standardises width, not channels. The code may run and return the expected shape while implementing a different model. I.8.B04 makes the axes explicit before counting any parameters.
L2 inside Adam changes both moment estimates. A separate shrinkage after the adaptive step does not. Copying a decay coefficient between the two names does not reproduce the same training rule; I.8.X04 gives a coordinate-level counterexample.