Run a shared cell through three inputs

STATEMENT

Compute a recurrent forward pass without inventing a new weight for each step.

GIVEN

The scalar linear cell $h_t=ux_t+wh_{t-1}+b$ has $u=1$, $w=1/2$, $b=0$ and $h_0=0$. Inputs are $(1,2,0)$. The final readout is $z_3=vh_3+c$, with $v=2$, $c=0$, and target $y=1$. Use $\mathcal L=(z_3-y)^2/2$.

FIND

All three states, the final prediction, loss, and number of trainable scalars.

STRATEGY

Keep one copy of each parameter. Substitute the preceding state at every step, then apply the readout only to the last state.

SOLUTION

This is (I.9.1) with an identity activation instead of tanh:

There are five trainable scalars: $u,w,b,v,c$. The initial state is specified as a constant, not learned. Sequence length changes the number of operations, not this parameter count.

A short implementation reproduces the arithmetic:

xs = [1.0, 2.0, 0.0]
u, w, b, v, c = 1.0, 0.5, 0.0, 2.0, 0.0
h = 0.0
history = []
for x in xs:
    h = u*x + w*h + b
    history.append(h)
prediction = v*h + c
loss = 0.5*(prediction - 1.0)**2
print(history)
print(prediction, loss)

It prints [1.0, 2.5, 1.25] and 2.5 1.125. The reproduction script also runs the tanh example from the proposition, so the identity activation is not silently treated as a nonlinear RNN.

Check — sanity

The third input is zero, but the third state is not zero: the recurrent term carries half the preceding state forward. With $w=0$, this cell would forget the preceding input at each step.