A gate gives the gradient a road home that no weight matrix stands on, which slows the decay without abolishing it.
Add a path along which information travels unchanged unless something explicitly erases it.
How this chapter is built
M3Load-bearing
The content is mathematics. Understanding is demonstrated by computation, not recall.
basics11/11what the words mean
concept2/2what to picture
theory4/4why it works, and when it does not
mathematics15/15derive it, then compute it
practice9/9build it, break it, read the papers
Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.
The vanishing gradient of I.9 is a property of repeated multiplication. The repair is not a better initialisation but a different architecture: an additive path whose derivative is a gate value rather than a Jacobian product. We build the LSTM and the GRU, differentiate them fully, and measure how much of the problem the gate actually removes.
Fig. 10 — An LSTM cell. The carry line crosses only a multiplication by the forget gate and an addition of what the input gate admits, so with the forget gate near one the gradient has a route home that no weight matrix attenuates. The output gate decides how much of the cell becomes visible.
What this chapter covers
Gates as learned, input-dependent multipliers
The LSTM cell state and its carry line
Forget, input and output gates, and the candidate
The constant error carousel and why the forget gate was added
The full Jacobian of a gated cell, block by block
Forget values, memory horizons and bias initialisation
Growth that gating does not prevent
Backpropagation through an LSTM cell
The GRU, its two conventions and its cost
Gated paths across depth: highway and residual connections
Apparatus
The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.
A vector of values in (0, 1), computed by a sigmoid from the current input and preceding state, that multiplies another vector coordinate by coordinate. It is a learned soft multiplier, never exactly zero or one, and it changes with the input.
Definition 2Cell state
The LSTM's second state vector c_t in R^H, carried from step to step alongside h_t. It is not exposed directly: the network's output reads h_t = o_t elementwise tanh(c_t). Its coordinates are not confined to (-1, 1).
Definition 3Candidate
The proposed new content, a tanh of an affine function of x_t and h_(t-1). In the LSTM the input gate scales it before it is added to the cell state; in the GRU the update gate weighs it against the old state.
Definition 4Forget gate
The gate f_t that multiplies the preceding cell state. A value near one keeps that coordinate; a value near zero discards it. It was added to the LSTM by Gers, Schmidhuber and Cummins in 2000.
Definition 5Input and output gates
The input gate i_t scales the candidate before it is written into the cell state. The output gate o_t scales the squashed cell state before it becomes the visible hidden state. Neither changes what the forget gate keeps.
Definition 6Carry line
The path from c_(t-1) to c_t through one elementwise multiplication and one addition. Holding the gate values and the candidate fixed, its Jacobian is diag(f_t). No weight matrix lies on it.
Definition 7Constant error carousel
The 1997 LSTM's self-connected cell with weight fixed at one and no forget gate. Along it the error signal is neither scaled up nor down. With a forget gate, the factor at each step becomes f_t.
Definition 8Memory horizon
For a constant forget value f in (0, 1), the number of steps -1/ln f over which the carry line's sensitivity falls by a factor of e. It lies between f/(1-f) and 1/(1-f).
Definition 9Update gate
The GRU gate z_t that interpolates between the preceding state and the candidate. Which side it multiplies is a convention: PyTorch and Cho and colleagues keep the old state when z is near one, Chung and colleagues write the candidate.
Definition 10Reset gate
The GRU gate r_t that scales the preceding state's contribution to the candidate. It may act before the recurrent matrix, as in the 2014 paper, or after it, as in PyTorch and cuDNN-compatible layers. The two forms are different functions.
Formal results
I.10.T1
The carry line is the only path free of recurrent weights
Treat (c_t, h_t) as the LSTM's state. In the block Jacobian of one step, the cell-to-cell block is diag(f_t). Expand d c_T/d c_s as a sum over paths through these blocks. The path that stays on the cell line at every step contributes diag(f_(s+1) elementwise ... elementwise f_T). Every other path passes through at least one block M_k = d c_k/d h_(k-1), which contains W_f, W_i and W_c.
What this does not promise
This does not bound the full derivative by the carry-line term. The other terms can add to it, cancel it or amplify it. In I.10.B02 the carry line alone gives 0.5625 while the full derivative is 1.3176.
I.10.T2
A constant forget value sets a memory horizon
If f_t = f in (0, 1) at every step, the carry-line sensitivity after n steps is f^n = exp(-n/m) with m = -1/ln f, and f/(1-f) < m < 1/(1-f). If f = sigma(b), then 1/(1-f) = 1 + e^b, and b = ln(m - 1) makes 1/(1-f) exactly m.
What this does not promise
Real forget values depend on the input and the state, so a bias fixes only the initial regime. The horizon of the carry line is not the horizon of the task; the output gate can hide a memory and the input gate can overwrite it.
I.10.T3
The gates, not a squashing function, bound the state
If every forget value is at most phi < 1 and |i_t tilde-c_t| < 1, then |c_t| <= phi^t |c_0| + (1 - phi^t)/(1 - phi) coordinatewise. With phi = 1 the bound is |c_0| + t. A GRU written as a convex combination of its old state and a tanh candidate keeps every coordinate in [-1, 1] when it starts there.
What this does not promise
A bounded state does not make the gradient bounded. Gradients still multiply through recurrent weights, and a large cell state saturates tanh(c_t), which shrinks the gradient into the cell through the output path.
Assumptions
Each is paired with the problem that shows what its removal costs. An assumption nobody tests is a disclaimer.
Forget-gate biases start in the remembering regime when long memory is needed. PyTorch's default initialisation does not do this, and its two bias vectors must be counted together.I.10.X02
The GRU equations in the text match the implementation whose weights are loaded: which side z multiplies, and whether r acts before or after the recurrent matrix.I.10.X06
Unbounded streams processed without reset have forget values bounded away from one, or the cell state can grow without limit.I.10.X03
A padded or skipped step freezes both h and c. Freezing h alone lets the cell state change.I.10.X05
Initialising the forget bias through the wrong slice. PyTorch orders LSTM gates (i, f, g, o), so for H = 5 the forget rows are 5 to 9; rows 0 to 4 set the input gate. Setting the forget slice of both bias_ih and bias_hh to one makes the effective bias two: sigma(2) = 0.880797, a horizon of 8.389 rather than 3.718.
Loading GRU weights trained with the reset gate after the matrix into a layer that applies it before. In I.10.B04 the same weights, state and gates give candidate (-0.124353, 0.358357) one way and (-0.358357, 0.124353) the other.
Reading the update gate with the wrong convention. With z = 0.9 for ten steps, one convention keeps 0.348678 of the old state and the other keeps 1e-10.
Running an LSTM without a forget gate over an unbroken stream. A constant write of 0.1 drives the cell to 20 after 200 steps; float64 then rounds tanh(20) to one, and the derivative of h with respect to c computed as 1 - tanh² is exactly zero.
Masking a padded step by freezing only h. In I.10.X05 the cell state moves from 1 to 0.8 during padding, and the next real step reads the wrong memory.
Dropping gradient clipping because the network is gated. In I.10.X09 a forget gate of one half still lets a perturbation grow by a factor 1.625 per step.
LSTM: A Search Space OdysseyGreff, Srivastava, Koutník, Steunebrink and Schmidhuber, 2017 · IEEE Transactions on Neural Networks and Learning Systems 28(10)