I.9.X09

A backward encoder can see the answer

numeric▲▲△

A backward linear encoder obeys $\overleftarrow h_t=x_t+0.5\overleftarrow h_{t+1}$ with terminal state $\overleftarrow h_4=0$ . Compute its state at position one for $(1,2,3)$ , then for $(1,2,7)$ . Could a representation containing that state be used unchanged to predict the next token causally after reading only position one?

Hint

Evaluate from position three to position one.

Solution

For $(1,2,3)$ the backward states at positions three, two and one are $3$ , $3.5$ and $2.75$ . For $(1,2,7)$ they are $7$ , $5.5$ and $3.75$ .

Position one’s representation changes even though the observed prefix $(1)$ is identical. It depends on the future. Using it unchanged for causal next-token prediction leaks information that would be unavailable at generation time.

A bidirectional encoder is appropriate when the full input is already observed, such as tagging a completed sentence. The objection is to using future context for a task whose information boundary forbids it, not to bidirectional encoding itself.

Draws on

I.9.P03 — A convolution over a sequence pays for range with depth and dilation.