Md. Asif Uddin

Chapter 10 I.10

Gated Recurrence: LSTM and GRU

A gate gives the gradient a road home that no weight matrix stands on, which slows the decay without abolishing it.

Add a path along which information travels unchanged unless something explicitly erases it.

How this chapter is built

M3Load-bearing

The content is mathematics. Understanding is demonstrated by computation, not recall.

basics2/11what the words mean
concept2/2what to picture
theory0/4why it works, and when it does not
mathematics0/15derive it, then compute it
practice0/9build it, break it, read the papers

Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

Before you start

The problem

The vanishing gradient of I.9 is a property of repeated multiplication. The repair is not a better initialisation but a different architecture: an additive path whose derivative is a gate value rather than a Jacobian product.

A gated cell and its uninterrupted carry lineA horizontal line across the top carries the cell state from left to right, crossed only by one multiplication and one addition. Below it, a forget gate and an input gate produce the numbers that do the multiplying and adding.the carry linec(t−1)c(t)×+forget gateσ → [0, 1]input gatewhat to writeh(t−1), x(t) — both gates read the same two thingsWith the forget gate near one the carry line is near the identity, and a gradient crossing it barely decays.
Fig. 10 — A gated cell. The carry line crosses only a multiplication and an addition, so with the forget gate near one the gradient has a route home that no weight matrix attenuates.

Apparatus

The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

The chain rule 0.MC.03 · Floating point 0.NU.01

Notation

  • TSequence length in tokens
  • dModel width
  • WA weight matrix
  • σThe logistic function, or a standard deviation
  • ∇Gradient operator

Propositions

  1. Prop. 1A gate gives the gradient a road home that no weight matrix stands on.The LSTM cell state passes through multiplication and addition only. With the forget gate near one, the path is close to the identity, and a gradient crossing it is barely attenuated.

Worked problems

0/5 problems0/4 variants0/10 exercisesowes 15 more

Not yet written. At M3 this chapter owes 5 worked problems across 4 distinct variants, and 10 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.