Md. Asif Uddin

Chapter 8 I.8

Normalisation and Regularisation

Normalisation and regularisation do two different jobs, and keeping them adjacent is what makes the difference visible.

Normalisation conditions the optimisation; regularisation constrains the hypothesis class.

How this chapter is built

M3Load-bearing

The content is mathematics. Understanding is demonstrated by computation, not recall.

basics3/11what the words mean
concept2/2what to picture
theory0/4why it works, and when it does not
mathematics0/15derive it, then compute it
practice0/9build it, break it, read the papers

Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

Before you start

The problem

Two mechanisms are routinely confused because both are switched on late and both make the curve look better. One changes the geometry the optimiser walks; the other changes which solution is preferred among those that fit.

An exact fit and a constrained oneSeven data points fitted by a curve that passes exactly through every one, and by a straighter curve that misses all of them slightly. A held-out point lies far from the exact curve and close to the constrained one.the same seven points, fitted twiceheld outpasses through every pointpasses through none of themRegularisation does notimprove the fit. It decideswhich fit you get whenseveral are available.
Fig. 8 — Seven points fitted exactly and fitted loosely. Regularisation does not improve the fit — it decides which fit you get when many are available.

Apparatus

The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

Variance and covariance 0.PR.03 · Estimators, bias and variance 0.ST.01 · Conditioning and stability 0.NU.03

Notation

  • λA regularisation coefficient, or an eigenvalue where the context is linear algebra
  • VarVariance
  • 𝔼Expectation
  • ηLearning rate
  • ∇Gradient operator

Propositions

  1. Prop. 1Regularisation does not improve the fit; it decides which fit you get.When many parameter settings explain the training data equally well, regularisation is the statement of which of them to prefer. It is an assumption about the world, expressed as a penalty.

Worked problems

0/5 problems0/4 variants0/10 exercisesowes 15 more

Not yet written. At M3 this chapter owes 5 worked problems across 4 distinct variants, and 10 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.