Md. Asif Uddin

Chapter 7 I.7

Optimisation

The gradient supplies a direction and never a distance, and every optimiser is a different answer to how far to go.

Descent is a local decision made repeatedly; optimisers differ in how much of the past they remember.

How this chapter is built

M3Load-bearing

The content is mathematics. Understanding is demonstrated by computation, not recall.

basics3/11what the words mean
concept2/2what to picture
theory0/4why it works, and when it does not
mathematics0/15derive it, then compute it
practice0/9build it, break it, read the papers

Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

Before you start

The problem

A gradient is not a training procedure. Step size, curvature, noise from minibatching and the schedule over time each decide whether the same gradients converge, oscillate, or diverge.

One step, three learning ratesThe same quadratic bowl drawn three times. With too small a step the parameters creep toward the minimum. With a well-chosen step they reach it. With too large a step they overshoot the minimum and climb the far wall.too smallcreepswell chosenarrivestoo largediverges
Fig. 7 — The same descent under three learning rates. Too small and it creeps, too large and it climbs the far wall; the gradient supplies the direction, never the distance.

Apparatus

The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

Convexity 0.OP.01 · Gradient descent 0.OP.02 · Momentum 0.OP.03 · Variance and covariance 0.PR.03 · Estimators, bias and variance 0.ST.01

Notation

  • ηLearning rate
  • βA momentum coefficient in an optimiser; the KL weight in a preference objective
  • BBatch size
  • θAll parameters of a model, taken together
  • ∇Gradient operator
  • VarVariance
  • ℒThe loss

Propositions

  1. Prop. 1The gradient supplies the direction and never the distance.Descent knows which way is downhill locally and nothing about how far downhill continues. The learning rate is the missing half, and no optimiser removes the need to choose it.
  2. Prop. 2The batch size is a statement about gradient noise, not only about speed.A batch gradient is an estimate of the true one, with variance falling as one over the batch size. Choosing a batch is choosing how noisy each step is, and the noise is not purely a defect.
  3. Prop. 3A schedule is tuned to a step count, and does not survive being copied.The learning rate rises because the optimiser starts with no statistics and falls because a large step near the end discards what the earlier ones found. Both parts depend on the total budget.

Worked problems

0/5 problems0/4 variants0/10 exercisesowes 15 more

Not yet written. At M3 this chapter owes 5 worked problems across 4 distinct variants, and 10 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.