Md. Asif Uddin

Chapter I

Computation and Representation

What a model is, and what it is made of.

What this chapter covers

  • What is a model?
  • Parameters
  • Functions and representations
  • Tensors
  • Dimensions and shapes
  • Forward computation
  • Loss functions
  • Optimisation
  • Generalisation

Propositions

  1. Prop. 1A model is a function with a second argument.A model is a map from inputs to outputs that also takes a set of parameters. Fixing the parameters gives you a particular function; training is the search for which one.
  2. Prop. 2A tensor carries its meaning in its shape.A tensor is an array with named axes. The numbers alone say nothing; the shape is the claim about what those numbers are, and every bug in a pipeline is a disagreement about it.
  3. Prop. 3Learning changes parameters, not rules.Training does not write new logic. It moves numbers inside a structure that was fixed before the first example arrived, which is why the architecture decides what is learnable at all.
  4. Prop. 4The loss is the only place the objective is actually stated.An optimiser reads the loss and nothing else. Whatever the objective fails to mention is not optimised, however clearly it was intended.
  5. Prop. 5Generalisation is not memorisation, and the training loss cannot tell them apart.A model that fits its training data perfectly has demonstrated capacity, not understanding. Only performance on data the model has never influenced is evidence of anything.