Write once, then keep

STATEMENT

Run a scalar LSTM through three inputs and separate what the input gate writes from what the forget gate keeps.

GIVEN

A scalar cell follows (I.10.1) and (I.10.2) with every recurrent weight zero. The gates read only their biases: $b_i=0$, $b_f=\ln3$, $b_o=\ln3$, and $u_i=u_f=u_o=0$. The candidate is $\tilde c_t=\tanh(x_t\ln3)$, so $u_c=\ln3$ and $b_c=0$. Start from $c_0=h_0=0$ and feed $(1,0,0)$.

FIND

The three gate values, the candidates, all three cell and hidden states, the third cell state when $b_f=\ln99$ instead, and the number of trainable scalars.

STRATEGY

Evaluate the gates first; they are constants here. Then apply the cell update one step at a time and read each hidden state through the output gate.

SOLUTION

Since $\sigma(\ln a)=a/(1+a)$, the gates are $i=\sigma(0)=1/2$ and $f=o=\sigma(\ln3)=3/4$. The first candidate is

and the later candidates are $\tanh(0)=0$.

The cell update writes once and then only keeps:

The hidden states are $h_t=\tfrac34\tanh(c_t)$: $0.284962$, $0.218484$ and $0.165959$.

With $b_f=\ln99$ the forget gate is $0.99$, and $c_3=0.4\cdot0.99^2=0.39204$. The input gate decided how much was written; the forget gate alone decided how much of it survived two silent steps.

Each of the four blocks has one input weight, one recurrent weight and one bias, so the scalar cell has $12$ trainable scalars. The plain scalar cell of I.9.B01 had three before its readout.

Check — sanity

Each hidden state is smaller than its cell state. For $c>0$, $\tanh c<c$, and the output gate multiplies by a further three quarters. The hidden state is a view of the cell, not a copy of it.