Problems

STATEMENT

Run a scalar LSTM through three inputs and separate what the input gate writes from what the forget gate keeps.

GIVEN

A scalar cell follows (I.10.1) and (I.10.2) with every recurrent weight zero. The gates read only their biases: $b_i=0$, $b_f=\ln3$, $b_o=\ln3$, and $u_i=u_f=u_o=0$. The candidate is $\tilde c_t=\tanh(x_t\ln3)$, so $u_c=\ln3$ and $b_c=0$. Start from $c_0=h_0=0$ and feed $(1,0,0)$.

FIND

The three gate values, the candidates, all three cell and hidden states, the third cell state when $b_f=\ln99$ instead, and the number of trainable scalars.

STRATEGY

Evaluate the gates first; they are constants here. Then apply the cell update one step at a time and read each hidden state through the output gate.

SOLUTION

Since $\sigma(\ln a)=a/(1+a)$, the gates are $i=\sigma(0)=1/2$ and $f=o=\sigma(\ln3)=3/4$. The first candidate is

and the later candidates are $\tanh(0)=0$.

The cell update writes once and then only keeps:

The hidden states are $h_t=\tfrac34\tanh(c_t)$: $0.284962$, $0.218484$ and $0.165959$.

With $b_f=\ln99$ the forget gate is $0.99$, and $c_3=0.4\cdot0.99^2=0.39204$. The input gate decided how much was written; the forget gate alone decided how much of it survived two silent steps.

Each of the four blocks has one input weight, one recurrent weight and one bias, so the scalar cell has $12$ trainable scalars. The plain scalar cell of I.9.B01 had three before its readout.

Check — sanity

Each hidden state is smaller than its cell state. For $c>0$, $\tanh c<c$, and the output gate multiplies by a further three quarters. The hidden state is a view of the cell, not a copy of it.

STATEMENT

Derive the backward pass of an LSTM step, identify the carry line in it, and measure how much of a real derivative the carry line accounts for.

GIVEN

The cell of (I.10.1) and (I.10.2), with a loss that may read any $\vec h_t$. For the numerical part, use the scalar cell of I.10.B01 with one change: the candidate also reads the hidden state, $\tilde c_t=\tanh(x_t\ln3+2h_{t-1})$. Inputs are $(1,0,0)$ from zero states.

FIND

The recursions for $\bar{\vec g}_{c,t}$ and the four pre-activation derivatives; the derivatives sent to $\vec h_{t-1}$ and $\vec c_{t-1}$; the parameter gradients; and, in the scalar case, $\partial c_3/\partial c_1$ compared with its carry-line term.

STRATEGY

Draw the arrows leaving $\vec c_t$ and $\vec h_t$, add the returns, and only then differentiate the activations. For the scalar part, notice that a constant output gate makes $h_t$ a function of $c_t$.

SOLUTION

Arrows out of the cell. The cell state $\vec c_t$ is used twice: by $\vec h_t=\vec o_t\odot\tanh(\vec c_t)$, and by the next cell update, where it is multiplied by $\vec f_{t+1}$. Reverse mode adds the two returns:

Here $\bar{\vec g}_{h,t}$ is already the total for $\vec h_t$: its readout term plus everything step $t+1$ returned through its gates.

Pre-activations. Write $\vec a_{k,t}$ for the argument of the gate or candidate $k$. Differentiating $\vec h_t=\vec o_t\odot\tanh\vec c_t$ with respect to $\vec a_{o,t}$, and $\vec c_t=\vec f_t\odot\vec c_{t-1}+\vec i_t\odot\tilde{\vec c}_t$ with respect to the other three,

This is (I.10.6). The output gate is the only one that reads $\bar{\vec g}_{h,t}$ directly; the other three see the cell’s total.

What goes back one step. Every pre-activation contains $\mat W_k\vec h_{t-1}$, so the recurrent return to $\vec h_{t-1}$ is $\sum_{k\in\{i,f,o,c\}}\mat W_k^{\mathsf T}\vec\delta_{k,t}$. The cell one step back receives $\bar{\vec g}_{c,t}\odot\vec f_t$, the carry line, which is the second term of the next $\bar{\vec g}_{c,t-1}$. At $t=1$ these two quantities are the derivatives with respect to $\vec h_0$ and $\vec c_0$.

Parameters. Each block is used at every step, so its gradients are sums, as in (I.9.4):

The scalar case. With constant gates $i=1/2$, $f=o=3/4$, the hidden state $h_t=\tfrac34\tanh c_t$ is a function of $c_t$, so the state is $c$ alone and the one-step derivative is

The forward pass gives $c_1=0.4$, $h_1=0.284962$, then $\tilde c_2=\tanh(0.569924)$ and onwards. Evaluating along that trajectory, $k_2=1.221326$ and $k_3=1.078827$, so

Expanding $k_3k_2$ gives four terms. Only $f\cdot f$ avoids the recurrent weight; the other three each contain it at least once, as I.10.T1 says. Here they contribute $0.755100$, more than the carry line.

The reproduction script perturbs $c_1$ by $10^{-6}$, reruns steps two and three, and matches $1.317600$ by central differences. It then checks the vector recursions above on a two-unit cell with non-symmetric recurrent weights, losses at three of four steps and nonzero initial states. All twelve parameter blocks, $\vec c_0$ and $\vec h_0$ agree with finite differences.

Check — sanity

Set every recurrent weight to zero. Then $\mat M_t=0$, the hidden-state return vanishes, and the derivative between cell states is exactly the product of forget values. In the scalar case $k_t=f$ and $\partial c_3/\partial c_1=0.5625$.

STATEMENT

Turn a constant forget value into a number of steps, bound that number, and find the forget value and bias a given horizon requires.

GIVEN

One coordinate’s forget value is $f\in(0,1)$ at every step. The carry-line factor after $n$ steps is $f^n$. The memory horizon is $m=-1/\ln f$.

FIND

(a) $f^{100}$ and $f^{1000}$ for $f=0.99$, with its horizon and half-life. (b) Bounds on $m$ in terms of $f$. (c) The smallest $f$ with $m\geq100$, and the bias that produces it when the gate’s other inputs are zero. (d) The behaviour of $m$ as $f\to1$ and as the bias grows.

STRATEGY

Write $f^n=e^{n\ln f}$ and use $\ln x\leq x-1$ twice. For the bias, invert the sigmoid.

SOLUTION

(a) Directly, $0.99^{100}=0.366032$ and $0.99^{1000}=4.317125\times10^{-5}$. The horizon is $-1/\ln0.99=99.499162$ steps. The half-life solves $f^n=1/2$, so $n_{1/2}=\ln2/(-\ln f)=m\ln2=68.967564$ steps.

(b) The inequality $\ln x\leq x-1$ is strict for $x\neq1$. At $x=f$ it gives $-\ln f>1-f$. At $x=1/f$ it gives $\ln(1/f)<1/f-1$, that is, $-\ln f<(1-f)/f$. Taking reciprocals,

The two bounds differ by exactly one, so $1/(1-f)$ gives the horizon to within a step. For $f=0.99$ they are $99$ and $100$.

(c) The condition $m\geq100$ is $-\ln f\leq1/100$, so $f\geq e^{-1/100}=0.990050$. A gate with only its bias has $f=\sigma(b)$, so $b=\ln\big(f/(1-f)\big)=4.600166$.

(d) As $f\to1$, $-\ln f\to0^+$ and $m\to\infty$; by (b), $m$ grows like $1/(1-f)$. With $f=\sigma(b)$, (I.10.5) gives $1/(1-f)=1+e^b$. Each extra unit of bias multiplies the horizon by about $e$, but no finite bias reaches $f=1$.

Check — sanity

At $f=1/2$ the bounds give $1<m<2$, and indeed $m=1/\ln2\approx1.44$. A zero forget bias forgets most of a value within two steps.

STATEMENT

Show that applying the reset gate before or after the recurrent matrix gives different GRUs, and find exactly when the two agree.

GIVEN

A two-unit GRU step with zero input contribution and zero candidate biases. The recurrent candidate matrix swaps coordinates, $\mat W_n=\begin{bmatrix}0&1\\1&0\end{bmatrix}$. The preceding state is $\vec h_{t-1}=(0.5,-0.5)$. The reset pre-activations are $(\ln3,-\ln3)$ and the update pre-activations are zero. Use PyTorch’s update, $\vec h_t=(1-\vec z_t)\odot\vec n_t+\vec z_t\odot\vec h_{t-1}$, for both.

FIND

The gates, both candidates and both new states; then a condition on $\mat W_n$ and $\vec r_t$ under which the two placements agree for every state.

STRATEGY

Compute $\mat W(\vec r\odot\vec h)$ and $\vec r\odot(\mat W\vec h)$ separately. For the general condition, write both as products with $\operatorname{diag}(\vec r)$ and compare entries.

SOLUTION

The gates are $\vec r=(\sigma(\ln3),\sigma(-\ln3))=(0.75,0.25)$ and $\vec z=(0.5,0.5)$. The update gate is the same in both models, so any difference comes from the candidate.

Reset before the matrix, as in Cho and colleagues: $\vec r\odot\vec h=(0.375,-0.125)$, and the swap gives $(-0.125,0.375)$. Hence

Reset after the matrix, as in PyTorch: $\mat W\vec h=(-0.5,0.5)$, then $\vec r\odot\mat W\vec h=(-0.375,0.125)$, and

With $z=1/2$, the new states are the averages of candidate and old state: $(0.187823,-0.070821)$ before and $(0.070821,-0.187823)$ after. Same weights, same state, same gates, and every coordinate differs by more than $0.1$.

When they agree. The candidates’ recurrent parts are $\mat W\operatorname{diag}(\vec r)\vec h$ and $\operatorname{diag}(\vec r)\mat W\vec h$. They agree for every $\vec h$ exactly when the matrices are equal. Their $(j,k)$ entries are $W_{jk}r_k$ and $r_jW_{jk}$, so the condition is

A weight joining two coordinates may be nonzero only if their reset values are equal. A uniform reset, $\vec r=\rho\vec 1$, satisfies it for every $\mat W$; a diagonal $\mat W$ satisfies it for every $\vec r$. The swap matrix with unequal resets satisfies neither, which is why this example separates them.

Biases add a second difference. In the after form the recurrent bias $\vec b_{hn}$ is inside the reset, $\vec r\odot(\mat W\vec h+\vec b_{hn})$, so the reset scales it. In the before form a bias added outside is not scaled. Even scalar models differ when that bias is nonzero.

Check — sanity

The swap matrix moves each coordinate’s reset to the other side. Reset before: coordinate one’s value, scaled by $0.75$, lands in coordinate two. Reset after: coordinate two’s value lands in coordinate one and is scaled there by $0.75$. The mirror-image answers are what that bookkeeping predicts.

STATEMENT

Audit the shapes, parameter counts, arithmetic and training storage of a tanh RNN, a GRU and an LSTM, then find a fair width for comparing two of them.

GIVEN

Input width $D=3$, hidden width $H=5$. Count one bias per gate as in (I.10.8), then PyTorch’s two bias vectors per block. For storage, take $N=4$ sequences of length $T=6$ in float32, as in I.9.B04. Assume the plain cell stores $\vec h_t$ for the backward pass and the LSTM stores $\vec i_t,\vec f_t,\vec o_t,\tilde{\vec c}_t,\vec c_t,\vec h_t$.

FIND

(a) PyTorch’s LSTM weight and bias shapes, and the rows of the forget gate. (b) Parameter counts for all three cells under both bias conventions, with the GRU’s reset-after form. (c) Multiply–accumulates per step for one sequence. (d) Stored activation bytes. (e) The GRU width whose one-bias count equals the LSTM’s at $H=5$.

STRATEGY

Each gated block is a plain recurrent layer’s affine map with its own weights. Count one block, then multiply.

SOLUTION

(a) PyTorch stacks the four LSTM blocks in the order input, forget, candidate, output. The input weights are $4H\times D=20\times3$, the recurrent weights $4H\times H=20\times5$, and each bias vector has $4H=20$ entries. Counting from zero, the forget gate occupies rows $5$ to $9$. Rows $0$ to $4$ are the input gate. For a GRU the order is reset, update, new, so the blocks are $3H$ rows tall.

(b) One block has $H(D+H+1)=5\cdot9=45$ parameters with one bias. So

The reset-after GRU keeps $\vec b_{hn}$ inside the reset, which cannot be merged with $\vec b_n$: $135+5=140$. PyTorch keeps two bias vectors per block, $H(D+H)+2H$ each: $40+10=50$, then $150$ and $200$.

(c) One affine map from $(\vec x_t,\vec h_{t-1})$ costs $H(D+H)=40$ multiply–accumulates. The LSTM computes four, $160$ per step; the GRU three, $120$; the plain cell one, $40$. The elementwise gate products add $O(H)$ per step and do not change the leading count.

(d) The plain cell stores $NTH=120$ numbers, $480$ bytes. The LSTM, under the stated scheme, stores six vectors per step: $6\cdot120=720$ numbers, $2880$ bytes. Other schemes store $\tanh\vec c_t$ or recompute gates in the backward pass. Name the scheme before quoting a ratio.

(e) A GRU of width $H'$ at $D=3$ has $3H'(H'+4)$ parameters. Setting this to $180$ gives $H'^2+4H'-60=0$, whose positive root is $H'=6$: $3\cdot6\cdot10=180$ exactly. Its arithmetic is $3\cdot6\cdot9=162$ multiply–accumulates per step, against the LSTM’s $160$. In PyTorch’s convention the two counts are $198$ and $200$.

A plain RNN cannot match exactly: $H'(H'+4)=180$ has no integer root, and widths $11$ and $12$ give $165$ and $192$.

Check — sanity

Setting $D=0$ removes every input weight: the one-bias LSTM count becomes $4H(H+1)=120$, four copies of a plain recurrent map with bias. Counts should always factor this way.