The gradient supplies a direction and never a distance, and every optimiser is a different answer to how far to go.
Descent is a local decision made repeatedly; optimisers differ in how much of the past they remember.
How this chapter is built
M3Load-bearing
The content is mathematics. Understanding is demonstrated by computation, not recall.
basics11/11what the words mean
concept2/2what to picture
theory4/4why it works, and when it does not
mathematics15/15derive it, then compute it
practice9/9build it, break it, read the papers
Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.
A gradient is not a training procedure. Step size, curvature, noise from minibatching and the schedule over time each decide whether the same gradients converge, oscillate, or diverge.
Fig. 7 — The same descent under three learning rates. Too small and it creeps, too large and it climbs the far wall; the gradient supplies the direction, never the distance.
What this chapter covers
The update rule, and what it leaves unspecified
Descent on a quadratic: the exact stability threshold
Condition number, and why one rate cannot suit two directions
The minibatch gradient as an estimator
Variance, batch size, and square-root returns
Momentum as an exponential moving average
Per-coordinate scaling: AdaGrad to Adam
Bias correction, and what it is correcting
Schedules: warm-up, decay, and the budget they encode
Weight decay, and where it belongs
Apparatus
The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.
βA momentum coefficient in an optimiser; the KL weight in a preference objective
BBatch size
θAll parameters of a model, taken together
∇Gradient operator
VarVariance
ℒThe loss
Definitions
Definition 1Gradient descent
The update θ ← θ − η∇ℒ(θ), applied repeatedly. It is a local rule: the gradient describes an infinitesimal neighbourhood of the current point and says nothing about how far the surface continues in that direction, which is the whole reason a step size has to be supplied from outside.
θAll parameters of a model, taken together
∇Gradient operator
Definition 2Learning rate
The scalar η converting a direction into a displacement. It carries the units: ∇ℒ has units of loss per parameter, so η has units of parameter squared per loss. This is why a rate that works for one parameterisation of a model need not work for another, and why the number is not transferable between architectures on its own.
ηLearning rate
Definition 3Minibatch gradient
The average gradient over B examples drawn from the data, used in place of the average over all of it. It is an unbiased estimator of the full-batch gradient — its expectation is exactly right — and each step descends a different sampled function rather than a noisy version of one fixed function.
BBatch size
Definition 4Batch gradient noise
The variance of that estimator, σ²/B for per-example variance σ². The standard error falls as 1/√B, so quadrupling the batch halves the noise at four times the arithmetic. The returns are square-root, which is the single most important fact about choosing a batch size.
VarVariance
Definition 5Condition number
For a quadratic loss with Hessian eigenvalues λ, the ratio κ = λ_max/λ_min. It measures how much more curved the steepest direction is than the flattest, and it is the quantity that decides how badly a single scalar step size serves a problem: one rate must be small enough for λ_max and is then κ times too small for λ_min.
Definition 6Momentum
The update v ← βv + ∇ℒ, θ ← θ − ηv, which replaces the current gradient with an exponential moving average of the recent ones. The average has effective length 1/(1−β) steps — β = 0.9 remembers about ten — and it cancels the oscillating component of the gradient in high-curvature directions while accumulating the consistent component in flat ones.
βA momentum coefficient in an optimiser; the KL weight in a preference objective
Definition 7Adam
Momentum on the gradient and on its square, used together: a first moment m for direction and a second moment v for scale, with the step m̂/(√v̂ + ε). Dividing by a per-coordinate estimate of gradient magnitude means every coordinate moves by roughly η regardless of its gradient's size, which is what makes one learning rate serviceable across parameters of very different scales.
Definition 8Bias correction
The division of a moving average by 1 − βᵗ at step t. An average initialised at zero is biased towards zero, badly so at the start: after one step with β₂ = 0.999 the raw second moment is a thousandth of its target. Without the correction the first steps are enormous, which is a large part of what a warm-up otherwise has to absorb.
Definition 9Learning-rate schedule
A function of step count multiplying η. The usual shape is a linear warm-up to a peak then a decay to near zero, and both halves are parameterised by the total budget — a cosine written for 100 000 steps and stopped at 50 000 leaves the model at half the peak rate, having never settled.
Formal results
I.7.T1
Exact convergence of gradient descent on a quadratic (Cauchy 1847)
For ℒ(θ) = ½(θ − θ*)ᵀH(θ − θ*) with H symmetric positive definite, gradient descent gives θ_{t+1} − θ* = (I − ηH)(θ_t − θ*). In the eigenbasis each coordinate is multiplied by 1 − ηλ_i each step, so the iteration converges if and only if 0 < η < 2/λ_max, and the optimal rate η = 2/(λ_max + λ_min) achieves a per-step contraction of (κ − 1)/(κ + 1).
What this does not promise
It is a statement about a quadratic, and a neural network loss is not one — it is non-convex, its Hessian changes with θ, and its eigenvalues span many orders of magnitude with a long tail. The result is a local model valid near a minimum, and the practical use of the threshold is qualitative: there is a sharp edge, crossing it diverges rather than degrades, and the edge is set by the largest curvature rather than by the typical one. It also says nothing about which minimum is reached, or whether the one reached generalises.
I.7.T2
Momentum improves the rate to its square root (Polyak 1964; Nesterov 1983)
On the same quadratic, heavy-ball momentum with tuned η and β attains a per-step contraction of (√κ − 1)/(√κ + 1) against gradient descent's (κ − 1)/(κ + 1). For κ = 10⁴ that is the difference between needing about 10⁴ steps and about 10² — the acceleration is in the exponent, not the constant.
What this does not promise
The rate is attained only at the tuned β, and heavy-ball momentum can fail to converge on some strongly convex problems where Nesterov's variant does not. The bound is asymptotic and says nothing about the first few steps, which is where the moving average is least informative. And it assumes exact gradients: with minibatch noise the accelerated rate degrades towards the noise floor, so on a stochastic problem the advantage is real but smaller than the deterministic bound suggests.
I.7.T3
Adam can fail to converge (Reddi, Kale and Kumar 2018)
There exist convex problems on which Adam does not converge to the optimum, for every fixed β₁, β₂ and step size. The cause is that v̂_t can decrease between steps, so the effective learning rate can grow, and the proof of convergence used in the original paper does not hold. AMSGrad restores convergence by taking a running maximum of v̂.
What this does not promise
The counterexample is a constructed convex problem, not a network, and AMSGrad has not displaced Adam in practice because the failure it fixes is rarely the binding one. What the result establishes is narrower and more useful than 'Adam is broken': the convergence guarantee people believed they had did not exist, so Adam's standing rests on evidence rather than on a theorem, and it should be argued for that way.
Assumptions
Each is paired with the problem that shows what its removal costs. An assumption nobody tests is a disclaimer.
Minibatches are drawn independently of the parameters, so the gradient estimate is unbiased. Sampling without reshuffling, or ordering the data by label, breaks this and the estimator becomes biased in a way no learning rate corrects.I.7.X05
The learning rate is below the stability threshold set by the largest curvature, which is a property of the loss at the current point rather than a fixed number for the model.I.7.X06
The gradient is finite and the loss is differentiable along the path taken. A single non-finite value propagates to every parameter through the moment estimates and persists after the input that caused it has gone.I.7.X07
A learning rate above 2/λ_max diverges rather than converging slowly, and the boundary is sharp. On a quadratic with λ_max = 20 the iteration converges at η = 0.09 and diverges at η = 0.11, with nothing gradual in between: below the threshold the error contracts geometrically, above it the error grows geometrically, and the loss reaches infinity in a few dozen steps. This is why a rate that 'nearly worked' is not evidence of being close — the failure is a change of sign in an exponent, not a matter of degree.
Adam's state is three times the parameters, and it is what exhausts a training device after the activations. A model with N parameters in fp32 holds 4N bytes of weights, 4N of gradients, and 8N of moment estimates: four copies, 16N bytes, before a single activation is stored. For a 1.5-billion-parameter model that is 24 GB of optimiser state alone. Switching from Adam to plain SGD with momentum halves that term, which is a real option that the framing of Adam as a default obscures.
A single non-finite gradient poisons the moment estimates and outlives its cause. One overflow makes m and v non-finite; every subsequent step divides by a non-finite v̂, so every parameter becomes non-finite, and the model produces NaN on every input thereafter. Skipping the bad step is not enough — the state must be repaired or reset, and the usual symptom is a loss that goes to NaN several steps after the batch that caused it, which sends people looking at the wrong data.