Chapter IV
Sequence Representation
What came before attention, and why it was replaced.
What this chapter covers
- Sequences
- RNNs
- LSTMs
- GRUs
- CNNs for sequences
- Long-range dependencies
- Why transformers emerged
Propositions
- Prop. 1A recurrent network compresses everything it has read into one fixed-width state.A recurrence carries a vector forward one step at a time. Its width does not grow with the sequence, so the whole of the past has to fit through it at every step.
- Prop. 2A gradient carried through many steps is a long product, and long products do not survive.Backpropagating through a recurrence multiplies the same Jacobian repeatedly. Below one it decays to nothing; above one it grows without bound. Only exactly one is stable, and nothing holds it there.
- Prop. 3A gate gives the gradient a road home that no weight matrix stands on.The LSTM cell state passes through multiplication and addition only. With the forget gate near one, the path is close to the identity, and a gradient crossing it is barely attenuated.
- Prop. 4A convolution over a sequence buys parallelism and pays for range with depth.Every position is computed at once, but the window is fixed. Widening it needs more layers, so range becomes a depth budget rather than a running state.
- Prop. 5Attention was the answer to a specific bottleneck, not a general improvement.It gives constant path length between any two positions with full parallelism, and pays for both in memory. Read as a trade rather than an advance, it explains what it is good for.