Chapter VIII
Training Deep Models
Turning an architecture into a model that has learned something.
What this chapter covers
- Dataset
- Batch
- Epoch
- Optimisation
- Learning-rate schedules
- Weight decay
- Gradient clipping
- Mixed precision
- Checkpoints
- Evaluation
Propositions
- Prop. 1The batch size is a statement about gradient noise, not only about speed.A batch gradient is an estimate of the true one, with variance falling as one over the batch size. Choosing a batch is choosing how noisy each step is, and the noise is not purely a defect.
- Prop. 2A schedule is tuned to a step count, and does not survive being copied.The learning rate rises because the optimiser starts with no statistics and falls because a large step near the end discards what the earlier ones found. Both parts depend on the total budget.
- Prop. 3Weight decay and gradient clipping guard different failures and do not substitute for each other.Clipping bounds the size of a step and does nothing on almost every one. Decay pulls every parameter toward zero on every step. One is about the update, the other about where the parameters end up.
- Prop. 4Mixed precision is a question of range before it is a question of speed.Half precision has ample precision and a narrow exponent. Small gradients fall through its floor and round to zero, which is what loss scaling exists to prevent.
- Prop. 5A number means what the split lets it mean.The unit of the split has to be the unit the claim is about. Split below that unit and the held-out score measures recall of things already seen, however carefully everything else was done.