Md. Asif Uddin

Chapter 4 I.4

Loss Functions

Every loss is a negative log-likelihood under some noise model, and choosing a loss is choosing that model.

A loss is the sentence 'here is what wrong means', written so a derivative exists.

How this chapter is built

M3Load-bearing

The content is mathematics. Understanding is demonstrated by computation, not recall.

basics3/11what the words mean
concept2/2what to picture
theory0/4why it works, and when it does not
mathematics0/15derive it, then compute it
practice0/9build it, break it, read the papers

Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

Before you start

The problem

Training needs a scalar to descend. Which scalar is not a detail: it fixes the implied noise model, the Bayes-optimal predictor, and whether the gradient survives at the point where the model is most wrong.

A loss surface, drawn as contoursNested contour rings around a deep minimum on the left and a shallower one on the right, with a saddle between them. A dashed line runs from a starting point downhill into the nearer basin, not necessarily the better one.loss surface over two parametersgloballocalstartThe loss is the only statement of the objective the optimiser can read. What it omits is not optimised.
Fig. 4 — A loss surface drawn as contours, with a deep basin and a shallower one. Descent finds the nearer minimum, which is not always the better one.

Apparatus

The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

Entropy 0.IT.01 · Cross-entropy 0.IT.02 · The exponential family 0.PR.06 · The softmax Jacobian 0.MC.06 · Log-sum-exp 0.NU.02

Notation

  • ℒThe loss
  • 𝔼Expectation
  • softmaxThe normalised exponential, applied row-wise unless stated
  • ℋEntropy, in nats unless bits are named
  • σThe logistic function, or a standard deviation

Propositions

Not yet written. The topics above are the plan for this chapter; each will become a proposition with its own figure.

Worked problems

0/5 problems0/4 variants0/10 exercisesowes 15 more

Not yet written. At M3 this chapter owes 5 worked problems across 4 distinct variants, and 10 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.