Md. Asif Uddin

Chapter 5 I.5

The Multilayer Perceptron

An MLP is a composition of affine maps and nonlinearities, and depth can buy exponentially in width.

Alternating affine maps and nonlinearities, each layer re-describing the data for the next.

How this chapter is built

M3Load-bearing

The content is mathematics. Understanding is demonstrated by computation, not recall.

basics3/11what the words mean
concept2/2what to picture
theory0/4why it works, and when it does not
mathematics0/15derive it, then compute it
practice0/9build it, break it, read the papers

Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

Before you start

The problem

The perceptron cannot solve XOR and one nonlinearity cannot be applied to nothing. Composing them is the smallest change that makes the failure go away, and the first architecture worth counting parameters for.

Composition with and without a non-linearityOn the left, two linear layers composed produce a single straight line: the stack collapses to one layer. On the right, the same two layers with a rectifier between them produce a piecewise-linear curve with three segments.linear ∘ linearlinear ∘ relu ∘ linearone line, whatever the depthkinkkinkeach unit contributes a fold
Fig. 5 — Two linear layers composed give one line whatever the depth. The same two layers with a rectifier between them give a piecewise curve, one fold per unit.

Apparatus

The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

Vectors, matrices and the row-major convention 0.LA.01 · The derivative of a linear map 0.MC.04 · Convexity 0.OP.01

Notation

  • WA weight matrix
  • bA bias vector
  • xAn input vector
  • dModel width
  • LNumber of layers
  • NParameter count

Propositions

  1. Prop. 1A linear layer performs an affine transformation, and nothing more.A dense layer multiplies by a matrix and adds a vector. Every property of it — what it can express, how it composes, why it needs help — follows from that one sentence.

Worked problems

0/5 problems0/4 variants0/10 exercisesowes 15 more

Not yet written. At M3 this chapter owes 5 worked problems across 4 distinct variants, and 10 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.