A model is a parameterised function, and everything training does is change the parameters.
A model is a function with a second argument, and learning searches that argument.
How this chapter is built
M2Substantive
The derivations are the chapter. A reader who skips the algebra has not learned it.
basics9/9what the words mean
concept2/2what to picture
theory2/2why it works, and when it does not
mathematics9/9derive it, then compute it
practice6/6build it, break it, read the papers
Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.
Before any of the machinery, the reader needs one frame: what object is being built, what can move in it, and what cannot. Without it, every later derivation is a manipulation of symbols whose referents are unclear.
Fig. 1 — A model as a function of two arguments. The input comes from the world; the parameters are the only part training is allowed to change.
What this chapter covers
What is a model?
Parameters
Functions and representations
Tensors
Dimensions and shapes
Forward computation
Loss functions
Optimisation
Generalisation
Apparatus
The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.
XA batch of token representations, T×d, rows are tokens
𝒟A dataset, as a set of examples
𝔼Expectation
ℝThe reals
θ̂An estimate, as against the quantity it estimates
Definitions
Definition 1Model
A model is a function f : 𝒳 × Θ → 𝒴 of two arguments. The first is an input supplied by the world; the second is a parameter vector supplied by the modeller. Fixing θ gives an ordinary function of x, written f(·; θ).
θAll parameters of a model, taken together
xAn input vector
Definition 2Parameter
A parameter is one component of θ. It is a number the training procedure is permitted to change. Anything the procedure may not change — the number of layers, the choice of nonlinearity, the learning rate — is a hyperparameter and is not part of θ.
NParameter count
Definition 3Hypothesis class
The hypothesis class ℱ = { f(·; θ) : θ ∈ Θ } is the set of functions reachable by varying the parameters. Choosing an architecture is choosing ℱ; training is searching inside it. No amount of training reaches a function outside ℱ.
Definition 4Tensor
A tensor is a rectangular array of numbers together with an interpretation of its axes. Its rank is the number of axes and its shape is the tuple of their lengths. Two tensors with identical shape may still be different objects, because the shape does not record what the axes mean.
XA batch of token representations, T×d, rows are tokens
ℝThe reals
Definition 5Forward computation
The forward computation is the evaluation of f(x; θ) at fixed θ, carried out as a finite sequence of tensor operations. Each operation's output shape is determined by its input shapes alone, which is why a shape error is detectable without running the computation.
Definition 6Loss
A loss ℓ : 𝒴 × 𝒴 → ℝ assigns a single non-negative number to one prediction–target pair, smaller meaning better. It is a choice, not a discovery: nothing in the data specifies it.
ℒThe loss
Definition 7Empirical risk
The empirical risk of θ on a dataset 𝒟 = {(xᵢ, yᵢ)}ⁿ is R̂(θ) = (1/n) Σ ℓ(f(xᵢ; θ), yᵢ) — the average loss over the examples actually held. It is a number that can be computed.
𝒟A dataset, as a set of examples
θ̂An estimate, as against the quantity it estimates
Definition 8Generalisation gap
The generalisation gap is R(θ) − R̂(θ), where R(θ) = 𝔼[ℓ(f(X; θ), Y)] is the expected loss under the distribution the data was drawn from. R̂ is computable and R is not, which is the entire difficulty of the subject.
𝔼Expectation
Formal results
I.1.T1
Empirical risk minimisation
Training selects θ̂ = argmin over Θ of R̂(θ). Under i.i.d. sampling R̂(θ) is an unbiased estimator of R(θ) for each fixed θ, so minimising the computable quantity is a principled proxy for minimising the one that matters.
What this does not promise
Unbiasedness holds for a θ fixed in advance, not for the θ̂ chosen by looking at the same data. R̂(θ̂) is biased downwards, and the bias grows with the size of the hypothesis class — which is exactly why a training loss of zero is not evidence of anything. It also promises nothing about whether the minimum is findable: existence of an optimum is not reachability by gradient descent.
I.1.T2
Bias–variance decomposition
For squared loss and a fixed input x, the expected error over training sets decomposes exactly as 𝔼[(ŷ − y)²] = (bias)² + variance + irreducible noise. The three terms are non-negative and additive, so reducing one at the cost of another may not help.
What this does not promise
It is not a recipe for choosing capacity: none of the three terms is observable from a single training run, and the decomposition is exact only for squared loss — for cross-entropy no such clean split exists. It also says nothing about which direction a given architectural change moves the terms, and modern over-parameterised networks are the standard counterexample to reading it as a U-shaped curve.
Assumptions
Each is paired with the problem that shows what its removal costs. An assumption nobody tests is a disclaimer.
The examples are drawn independently from a single fixed distribution, and the test data comes from that same distribution.I.1.X05
The loss is a faithful statement of the objective — minimising it is what the modeller actually wants.I.1.X06
A model with more parameters than examples can drive the empirical risk to exactly zero while the expected risk stays at chance. Fit a 10-parameter polynomial to 10 points with random labels: training MSE is 0 and test accuracy is 50% on a balanced binary task. The training number is not a weak signal here — it is no signal.
A shape error that broadcasts instead of failing. Adding a bias of shape (n,) to activations of shape (n, 1) yields an (n, n) tensor rather than an error, and the loss still decreases, so the bug survives training and shows up only as an accuracy ceiling nobody can explain.