Md. Asif Uddin

Chapter 9 I.9

Recurrent Neural Networks

A recurrent network compresses everything it has read into one fixed-width state, and the gradient through that compression is a long product.

One layer applied over and over, carrying a summary forward.

How this chapter is built

M3Load-bearing

The content is mathematics. Understanding is demonstrated by computation, not recall.

basics3/11what the words mean
concept2/2what to picture
theory0/4why it works, and when it does not
mathematics0/15derive it, then compute it
practice0/9build it, break it, read the papers

Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

Before you start

The problem

An MLP takes a fixed-size input. Sequences are not fixed-size, and padding them wastes the structure that makes them sequences. Sharing one layer across time is the first answer, and it introduces the failure that motivates everything after it.

A recurrence unrolled over five stepsFive inputs entering five copies of the same cell, each passing a fixed-width hidden state to the next. Everything read so far has to fit through that one state, whose width does not grow with the sequence.the same cell, five timesx₁fh1y1x₂fh2y2x₃fh3y3x₄fh4y4x₅fy5The state has a fixed width. Step 5 reaches step 1 only through everything between it and step 1.
Fig. 9 — A recurrence unrolled. Every step passes a fixed-width state to the next, so the path between two distant positions is as long as the distance between them.

What this chapter covers

  • Sequences
  • RNNs
  • LSTMs
  • GRUs
  • CNNs for sequences
  • Long-range dependencies
  • Why transformers emerged

Apparatus

The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

Rank, eigenvalues and the singular value decomposition 0.LA.04 · The chain rule 0.MC.03 · Floating point 0.NU.01

Notation

  • TSequence length in tokens
  • dModel width
  • WA weight matrix
  • JA Jacobian matrix
  • ∇Gradient operator
  • λA regularisation coefficient, or an eigenvalue where the context is linear algebra

Propositions

  1. Prop. 1A recurrent network compresses everything it has read into one fixed-width state.A recurrence carries a vector forward one step at a time. Its width does not grow with the sequence, so the whole of the past has to fit through it at every step.
  2. Prop. 2A gradient carried through many steps is a long product, and long products do not survive.Backpropagating through a recurrence multiplies the same Jacobian repeatedly. Below one it decays to nothing; above one it grows without bound. Only exactly one is stable, and nothing holds it there.
  3. Prop. 3A convolution over a sequence buys parallelism and pays for range with depth.Every position is computed at once, but the window is fixed. Widening it needs more layers, so range becomes a depth budget rather than a running state.

Worked problems

0/5 problems0/4 variants0/10 exercisesowes 15 more

Not yet written. At M3 this chapter owes 5 worked problems across 4 distinct variants, and 10 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.