Separate parameters, states and matrix products

STATEMENT

Audit a batched recurrent model before confusing sequence length with model size.

GIVEN

A single-layer, unidirectional tanh RNN uses batch size $N=4$, sequence length $T=6$, input width $D=3$, hidden width $H=5$ and output width $C=2$. Use (I.9.1), one combined hidden bias, a fixed zero initial state, and a readout at every step. Store a batch of inputs as $(N,T,D)$.

FIND

Tensor shapes, trainable parameter count, main matrix-product multiply-accumulates, and storage for the $T$ hidden states in float32.

STRATEGY

Keep the column-vector mathematical convention for a single example. For batched rows, transpose the weight matrices in the implementation. Count each shared matrix once.

SOLUTION

The row-batch form of the preactivation is $X_tU^{\mathsf T}+H_{t-1}W^{\mathsf T}+b$. Writing $X_tU$ would try to multiply $(4,3)$ by $(5,3)$, whose inner dimensions disagree.

The parameter count is

Each example-step needs $HD+H^2+CH=50$ multiply-accumulates for the main matrix products. There are $NT=24$ example-steps, giving $1200$ such multiply-accumulates. Counting each multiplication and addition separately gives about $2400$ operations for those products only. Bias additions, activations, losses and backward work are excluded.

The hidden outputs contain $NTH=120$ scalars. Four bytes each gives $480$ bytes. This is not total training memory: inputs, readouts, other saved values, parameters, gradients and optimiser state also need storage. Streaming inference can instead retain the current $NH=20$ state scalars, or $80$ bytes for that state alone.

Check — sanity

Doubling $T$ doubles these products and stored hidden outputs, but leaves the parameter count unchanged. Doubling $H$ makes the recurrent matrix four times as large.