Chapter II
Neural Networks
Layers, non-linearity, and how gradients move through them.
What this chapter covers
- Neurons
- Linear layers
- Activation functions
- Multilayer perceptrons
- Forward propagation
- Backpropagation
- Gradients
- Optimisers
- Learning rates
- Regularisation
Propositions
- Prop. 1A linear layer performs an affine transformation, and nothing more.A dense layer multiplies by a matrix and adds a vector. Every property of it — what it can express, how it composes, why it needs help — follows from that one sentence.
- Prop. 2Non-linearity is what makes depth worth having.Without an activation between them, stacked layers collapse into one. The activation is not a detail of the architecture — it is the reason the architecture has layers.
- Prop. 3Backpropagation is the chain rule with the intermediate results kept.There is no separate learning algorithm inside a network. There is a graph, one traversal forwards to compute values, and one backwards to accumulate derivatives.
- Prop. 4The gradient supplies the direction and never the distance.Descent knows which way is downhill locally and nothing about how far downhill continues. The learning rate is the missing half, and no optimiser removes the need to choose it.
- Prop. 5Regularisation does not improve the fit; it decides which fit you get.When many parameter settings explain the training data equally well, regularisation is the statement of which of them to prefer. It is an assumption about the world, expressed as a penalty.