Md. Asif Uddin

Chapter 2 II.2

From Recurrence to Attention

Attention was the answer to a specific bottleneck, not a general improvement.

A fixed-size summary is a bottleneck; let the decoder look back at everything instead.

How this chapter is built

M2Substantive

The derivations are the chapter. A reader who skips the algebra has not learned it.

basics3/9what the words mean
concept2/2what to picture
theory0/2why it works, and when it does not
mathematics0/9derive it, then compute it
practice0/6build it, break it, read the papers

Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

Before you start

The problem

Book I ended with a gated recurrence still failing on a long dependency. The failure is not a training problem but an architectural one: a fixed-width state cannot carry an unbounded source. Naming the bottleneck precisely is what makes the repair obvious.

Four ways to read a sequence, by path length and parallelismA table of four architectures. Recurrent models connect distant positions through a path that grows with distance and cannot be parallelised. Convolution shortens the path logarithmically and parallelises. Attention connects any two positions in one step, in parallel, at quadratic cost.architecturepath between two positionsover the sequenceRNNO(n)sequentialLSTM / GRUO(n)sequentialCNNO(log n)parallelAttentionO(1)parallelAttention is not a better idea than recurrence in the abstract. It is the trade that pays when thehardware is parallel and the sequences are short enough to afford n².
Fig. 2 — Four ways to read a sequence, placed by path length and parallelism. Attention is not a better idea in the abstract; it is the trade that pays on parallel hardware.

Apparatus

The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

Entropy 0.IT.01 · Inner products, norms and cosine similarity 0.LA.03

Notation

  • TSequence length in tokens
  • dModel width
  • WA weight matrix
  • softmaxThe normalised exponential, applied row-wise unless stated

Propositions

  1. Prop. 1Attention was the answer to a specific bottleneck, not a general improvement.It gives constant path length between any two positions with full parallelism, and pays for both in memory. Read as a trade rather than an advance, it explains what it is good for.

Worked problems

0/3 problems0/3 variants0/6 exercisesowes 9 more

Not yet written. At M2 this chapter owes 3 worked problems across 3 distinct variants, and 6 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.