Chapter 9 · I.9

Problems

A chapter that uses mathematics has to teach that mathematics by making the reader compute. A chapter that only displays equations has failed, however correct the equations are.

M3Load-bearing

5/5 problems5/4 variants10/10 exercisesquota met, and enforced

The contract

5 worked problems, minimum.
4 distinct variants, and no variant more than half of them.
10 exercises, every one with a published solution.
At least one numeric problem — present.
At least one symbolic problem — present.
At least one limit or counterexample problem — present.
At least one complexity or shape problem — present.
At least one ▲▲▲ problem — present.

Numerical instantiationSymbolic derivationConstructed failureDimensional algebraDifferentiation

Problem I.9.B01

Run a shared cell through three inputs

numeric▲△△

Exact values where possible; six decimal places for non-exact numerical results.

STATEMENT

Compute a recurrent forward pass without inventing a new weight for each step.

GIVEN

The scalar linear cell $h_t=ux_t+wh_{t-1}+b$ has $u=1$ , $w=1/2$ , $b=0$ and $h_0=0$ . Inputs are $(1,2,0)$ . The final readout is $z_3=vh_3+c$ , with $v=2$ , $c=0$ , and target $y=1$ . Use $\mathcal L=(z_3-y)^2/2$ .

FIND

All three states, the final prediction, loss, and number of trainable scalars.

STRATEGY

Keep one copy of each parameter. Substitute the preceding state at every step, then apply the readout only to the last state.

SOLUTION

This is (I.9.1) with an identity activation instead of tanh:

\begin{aligned} h_1&=1+(1/2)0=1,\\ h_2&=2+(1/2)1=5/2,\\ h_3&=0+(1/2)(5/2)=5/4,\\ z_3&=2(5/4)=5/2,\\ \mathcal L&=\tfrac12(5/2-1)^2=9/8. \end{aligned}

There are five trainable scalars: $u,w,b,v,c$ . The initial state is specified as a constant, not learned. Sequence length changes the number of operations, not this parameter count.

A short implementation reproduces the arithmetic:

xs = [1.0, 2.0, 0.0]
u, w, b, v, c = 1.0, 0.5, 0.0, 2.0, 0.0
h = 0.0
history = []
for x in xs:
    h = u*x + w*h + b
    history.append(h)
prediction = v*h + c
loss = 0.5*(prediction - 1.0)**2
print(history)
print(prediction, loss)

It prints [1.0, 2.5, 1.25] and 2.5 1.125. The reproduction script also runs the tanh example from the proposition, so the identity activation is not silently treated as a nonlinear RNN.

Answer

The states are $(1,2.5,1.25)$ , the prediction is $2.5$ , the loss is $1.125$ , and the model has five trainable scalars.

Check — sanity

The third input is zero, but the third state is not zero: the recurrent term carries half the preceding state forward. With $w=0$ , this cell would forget the preceding input at each step.

Check — numeric · i-9-recurrence-checks.py

The complete reproduction script uses only the Python standard library. From a checkout of the repository, run:

python3 scripts/snippets/i-9-recurrence-checks.py

It checks all fifteen problems, including analytic derivatives against central finite differences. The first line reports the three states above and loss $1.125000$ . The nonlinear forward example ends with states $0$ , $0.462117$ , $0.842886$ and $0.398145$ .

Executed in CI. The digits above are the digits it printed.

Where this breaks

Putting tanh back into the cell changes these numbers. It also introduces activation derivatives into the backward pass; the linear example is a controlled special case, not a replacement for those factors.

Variation

Reverse the first two inputs. The states become $(2,2,1)$ , giving prediction $2$ and loss $1/2$ . Shared parameters do not imply order invariance.

Problem I.9.B02

Every use of a weight contributes

symbolic▲▲▲

Exact values where possible; six decimal places for non-exact numerical results.

STATEMENT

Derive backpropagation through time and compute every parameter gradient for the forward pass in I.9.B01.

GIVEN

Use (I.9.1) and a sum of per-step losses for the derivation. For the numerical case use the linear scalar cell, inputs $(1,2,0)$ , $h_0=0$ , $(u,w,b,v,c)=(1,1/2,0,2,0)$ , and final target one.

FIND

The recurrent backward rule, the shared-weight accumulation rule, all five scalar parameter gradients, input derivatives and initial-state derivative.

STRATEGY

First add the two incoming backward signals at each state. Differentiate the activation, then accumulate the local parameter contributions. Specialise to the identity activation only after deriving the tanh rule.

SOLUTION

The readout at time $t$ contributes $\mat V^{\mathsf T}\vec e_t$ to its state. The next cell contributes $\mat W^{\mathsf T}\vec\delta_{t+1}$ . Multiplying their sum by the coordinatewise tanh derivative gives (I.9.3).

For a shared recurrent-weight entry $W_{ij}$ , the local differential is $d a_{t,i}=h_{t-1,j}\,dW_{ij}$ . Summing all uses gives

\frac{\partial\mathcal L}{\partial W_{ij}} =\sum_t\delta_{t,i}h_{t-1,j}.

This is the entrywise derivation of (I.9.4), not an average over time. If the objective is an average, its factor is already in $\vec e_t$ .

In the linear example, $z_3-y=3/2$ and the activation derivative is one. Only the final readout has a loss. Starting from the end,

\begin{aligned} \delta_3&=v(z_3-y)=3,\\ \delta_2&=w\delta_3=3/2,\\ \delta_1&=w\delta_2=3/4. \end{aligned}

The local contributions to $w$ are therefore $0$ , $3/2$ and $15/2$ :

\begin{aligned} \partial_w\mathcal L &=\delta_1h_0+\delta_2h_1+\delta_3h_2=9,\\ \partial_u\mathcal L &=\delta_1x_1+\delta_2x_2+\delta_3x_3=15/4,\\ \partial_b\mathcal L&=\delta_1+\delta_2+\delta_3=21/4,\\ \partial_v\mathcal L&=(z_3-y)h_3=15/8,\\ \partial_c\mathcal L&=z_3-y=3/2. \end{aligned}

Input derivatives are $u\delta_t$ , giving $(3/4,3/2,3)$ . The initial-state derivative is $w\delta_1=3/8$ . We compute the latter even though this example does not optimise $h_0$ .

For an independent algebraic check, with $b=c=h_0=0$ the final state is $h_3=u(w^2+2w)$ . Thus $\partial_w z_3=vu(2w+2)=6$ , and $\partial_w\mathcal L=(3/2)6=9$ .

The reproduction script perturbs each parameter by $10^{-6}$ , recomputes the full forward loss and compares central differences with the analytic derivative. It additionally checks tanh cells with losses at multiple steps and a two-coordinate state with non-symmetric recurrent weights. The latter catches transpose errors that a scalar example cannot.

The script and run command are in the numeric check on I.9.B01.

Answer

In the order $(u,w,b,v,c)$ the gradient is $(3.75,9,5.25,1.875,1.5)$ . All three time steps contribute to the same parameter objects.

Check — sanity

Since $h_0=0$ , the first local $w$ contribution is zero, but the first input still receives a nonzero derivative. “This use contributes zero” does not mean “this time step is disconnected.”

Where this breaks

Updating parameters inside the backward sweep would mix derivatives of different forward computations. Accumulate first, then take one optimiser step. A detached boundary also removes terms and must be tested as a different graph.

Variation

If all three time steps had local output losses, replace each zero local readout derivative by its actual value in (I.9.3). The future term remains; local supervision does not replace it.

Problem I.9.B03

Small eigenvalues can hide a large transient

counterexample▲▲▲

Exact values where possible; six decimal places for non-exact numerical results.

STATEMENT

Disprove the claim that recurrent eigenvalues below one prevent every finite-time amplification.

GIVEN

Consider the linear state update $\vec h_t=\mat W\vec h_{t-1}$ with

\mat W=\begin{bmatrix}1/2&4\\0&1/2\end{bmatrix}.

Both eigenvalues are $1/2$ . Use Euclidean vector norms and let $\vec e_2=(0,1)^{\mathsf T}$ .

FIND

A formula for $\mat W^n$ , its effect on $\vec e_2$ , and why this does not contradict eventual decay or the norm bound (I.9.6).

STRATEGY

Write $\mat W=(1/2)\mat I+\mat N$ . Since $\mat N^2=0$ , only two terms survive in the binomial expansion.

SOLUTION

For integer $n\geq1$ ,

\begin{aligned} \mat W^n &=2^{-n}\mat I+n\,2^{-(n-1)}\mat N\\ &=\begin{bmatrix} 2^{-n}&4n\,2^{-(n-1)}\\ 0&2^{-n} \end{bmatrix}. \end{aligned}

It follows that

\begin{aligned} \mat W\vec e_2&=(4,1/2)^{\mathsf T},\\ \mat W^2\vec e_2&=(4,1/4)^{\mathsf T}. \end{aligned}

Their norms are approximately $4.031129$ and $4.007805$ , although the initial vector has norm one. At ten steps the first coordinate is $0.078125$ and the second is $0.0009765625$ .

The matrix is non-normal: it does not commute with its transpose. Its eigenvalue magnitudes describe the eventual exponential rate, not a uniform bound on every finite power. The off-diagonal term introduces a factor of $n$ before the exponential decay wins. Both $2^{-n}$ and $n2^{-(n-1)}$ tend to zero, so $\mat W^n$ eventually tends to the zero matrix.

Backward propagation uses $(\mat W^n)^{\mathsf T}$ . Its spectral norm equals that of $\mat W^n$ , so a backward direction can be amplified too. For example, $\mat W^{\mathsf T}(1,0)^{\mathsf T}=(1/2,4)^{\mathsf T}$ .

There is no conflict with (I.9.6). Indeed $\|\mat W\|_2\geq\|\mat W\vec e_2\|_2>4$ . The eigenvalue $1/2$ is not a valid replacement for the factor norm in that inequality.

Answer

Eigenvalues inside the unit circle permit substantial transient amplification. Here a unit perturbation grows by more than four in one step, then ultimately decays.

Check — sanity

At $n=1$ the power formula returns the supplied matrix. At $n=2$ its off-diagonal entry is four, matching direct multiplication.

Where this breaks

For a nonlinear cell the temporal Jacobians can change at every step. A formula for powers of one fixed matrix is then insufficient. Even a large product norm only guarantees an amplified direction, not that the actual loss gradient points along it.

Variation

Set the off-diagonal entry to zero. The matrix becomes $(1/2)\mat I$ , and every vector contracts by $2^{-n}$ after $n$ steps. Identify the term responsible for the transient in the original example.

Problem I.9.B04

Separate parameters, states and matrix products

shape▲▲△

Exact values where possible; six decimal places for non-exact numerical results.

STATEMENT

Audit a batched recurrent model before confusing sequence length with model size.

GIVEN

A single-layer, unidirectional tanh RNN uses batch size $N=4$ , sequence length $T=6$ , input width $D=3$ , hidden width $H=5$ and output width $C=2$ . Use (I.9.1), one combined hidden bias, a fixed zero initial state, and a readout at every step. Store a batch of inputs as $(N,T,D)$ .

FIND

Tensor shapes, trainable parameter count, main matrix-product multiply-accumulates, and storage for the $T$ hidden states in float32.

STRATEGY

Keep the column-vector mathematical convention for a single example. For batched rows, transpose the weight matrices in the implementation. Count each shared matrix once.

SOLUTION

Quantity	Shape
Inputs	$(4,6,3)$
One input slice $X_t$	$(4,3)$
$U$ , $W$ , $b$	$(5,3)$ , $(5,5)$ , $(5)$
One state batch $H_t$	$(4,5)$
All hidden outputs	$(4,6,5)$
$V$ , $c$	$(2,5)$ , $(2)$
All readouts	$(4,6,2)$

The row-batch form of the preactivation is $X_tU^{\mathsf T}+H_{t-1}W^{\mathsf T}+b$ . Writing $X_tU$ would try to multiply $(4,3)$ by $(5,3)$ , whose inner dimensions disagree.

The parameter count is

\begin{aligned} HD+H^2+H+CH+C &=15+25+5+10+2\\ &=57. \end{aligned}

Each example-step needs $HD+H^2+CH=50$ multiply-accumulates for the main matrix products. There are $NT=24$ example-steps, giving $1200$ such multiply-accumulates. Counting each multiplication and addition separately gives about $2400$ operations for those products only. Bias additions, activations, losses and backward work are excluded.

The hidden outputs contain $NTH=120$ scalars. Four bytes each gives $480$ bytes. This is not total training memory: inputs, readouts, other saved values, parameters, gradients and optimiser state also need storage. Streaming inference can instead retain the current $NH=20$ state scalars, or $80$ bytes for that state alone.

Answer

There are $57$ parameters, $1200$ main-product multiply-accumulates, and $480$ bytes of hidden-output storage under the stated conventions. All output logits have shape $(4,6,2)$ .

Check — sanity

Doubling $T$ doubles these products and stored hidden outputs, but leaves the parameter count unchanged. Doubling $H$ makes the recurrent matrix four times as large.

Where this breaks

A two-bias implementation adds another $H=5$ parameters, giving $62$ . A learned initial state adds its own parameters. Bidirectionality and additional layers change the calculation again; the number 57 is not a universal RNN count.

Variation

Suppose only the final state is read out. The recurrent products still cost $NT(HD+H^2)=960$ multiply-accumulates, but readout products cost only $NCH=40$ , for a total of $1000$ . The parameter count is unchanged.

Problem I.9.B05

Same prediction, different backward graph

gradient▲▲△

Exact values where possible; six decimal places for non-exact numerical results.

STATEMENT

Compare full backpropagation with a one-step truncation that detaches the state after the second input.

GIVEN

Use I.9.B01’s linear scalar model: $(u,w,b,v,c)=(1,1/2,0,2,0)$ , inputs $(1,2,0)$ , initial state zero, and final target one. The full forward states are $(1,5/2,5/4)$ . There is only a final output loss.

FIND

The prediction and loss in both computations, and their derivatives with respect to $w$ and $u$ . Explain the correct finite-difference check for each.

STRATEGY

After detaching $h_2$ , replace its computation by the constant $5/2$ . Differentiate the remaining one-step function and compare it with I.9.B02.

SOLUTION

The full graph has $h_3=u(w^2+2w)$ at $b=0$ . At the stated parameters its gradients are $\partial_w\mathcal L=9$ and $\partial_u\mathcal L=15/4$ .

The detached graph instead evaluates

\begin{aligned} h_3&=u\cdot0+w\cdot(5/2)+b,\\ z_3&=vh_3+c. \end{aligned}

At the original parameters this gives the same $h_3=5/4$ , $z_3=5/2$ and loss $9/8$ . Its last-state derivative is still $\delta_3=3$ , but there is no earlier parameter path:

\begin{aligned} \partial_w\mathcal L_{\mathrm{trunc}}&=3(5/2)=15/2,\\ \partial_u\mathcal L_{\mathrm{trunc}}&=3\cdot0=0. \end{aligned}

The derivative with respect to the boundary value itself would be $w\delta_3=3/2$ . Detachment stops it from propagating into the calculation that produced that value.

To finite-difference the full graph, perturb $w$ or $u$ and recompute every state. To finite-difference the truncated graph, first compute the boundary state at the original parameters, freeze it at $5/2$ , then perturb the parameters only in the final chunk.

Recomputing the boundary during the truncated check reattaches the dependency you intended to remove. Its result would be the full gradient, not a failure of the truncated backward implementation.

Answer

Both predictions are $2.5$ and both losses are $1.125$ . Full $(\partial_w,\partial_u)=(9,3.75)$ ; truncated $(\partial_w,\partial_u)=(7.5,0)$ .

Check — sanity

Resetting instead of detaching would set the boundary to zero. Because $x_3=b=c=0$ , that would give prediction zero and loss $1/2$ , a different forward computation. Detach preserves the value; reset replaces it.

Where this breaks

This comparison uses one fixed parameter setting and one final loss. Chunk-wise updates may also change parameters between chunks; they are not identical to a single full-sequence update even if boundary derivatives happen to be small.

Variation

Keep the final two time steps after detaching $h_1$ . The $w$ gradient happens to remain nine because the removed first-step contribution is zero, but the $u$ gradient drops from $3.75$ to $3$ . Matching one parameter’s derivative does not establish that the whole backward graph matches.

Exercises

Every one has a published solution. A hidden solution is a solution; a missing one is an abandonment.

I.9.X01Shared weights do not erase ordernumeric▲△△

For $h_t=x_t+wh_{t-1}$ with $h_0=0$ and $w=1/2$ , compare the final states for inputs $(1,2)$ and $(2,1)$ . Derive when two inputs $(a,b)$ can be swapped without changing the final state.

Hint

Write the two final states in terms of $a$ , $b$ and $w$ before substituting.

Solution

The forward order gives $h_2=2+(1/2)1=2.5$ . The reverse gives $h_2=1+(1/2)2=2$ .

Generally the difference is

(b+wa)-(a+wb)=(1-w)(b-a).

It is zero when $a=b$ or $w=1$ . With $w=1$ , this particular linear cell sums its inputs, so order is lost. For other $w$ , sharing the transition does not make the representation permutation-invariant. Even $w=0$ retains the final input and therefore distinguishes these two orders.

A nonlinear recurrent cell has a different equality condition; do not transfer the special sum-cell result to every RNN.

I.9.X02Do not forget the activation derivativegradient▲△△

A scalar tanh cell has current output $h_t=0.6$ , recurrent weight $w=0.5$ , and total upstream state derivative $q_t=5$ . Compute $\delta_t$ and the backward contribution to $h_{t-1}$ . Repeat for $h_t=0.99$ .

Hint

Use $1-h_t^2$ , then multiply by $w$ to cross the recurrent edge.

Solution

For $h_t=0.6$ ,

\begin{aligned} \delta_t&=5(1-0.6^2)=3.2,\\ \partial_{h_{t-1}}\mathcal L\big|_{\text{via }t} &=0.5(3.2)=1.6. \end{aligned}

For $h_t=0.99$ , the activation factor is $0.0199$ . Thus $\delta_t=0.0995$ and the returned contribution is $0.04975$ . The weight and upstream signal are unchanged; saturation alone greatly reduces what travels back.

If the earlier state has its own local loss, that separate derivative must also be added. These numbers describe the path through step $t$ , not every possible contribution to the earlier state.

I.9.X07Clip the whole gradient oncenumeric▲△△

A two-parameter gradient is $g=(6,8)$ . Use global norm threshold $\tau=5$ and plain gradient-descent step size $\eta=0.1$ . Find the clipped gradient and update norm. Compare clipping each coordinate to the interval $[-5,5]$ .

Hint

The original Euclidean norm is ten. Global clipping uses one scale factor for both entries.

Solution

Equation (I.9.7) scales both coordinates by $5/10$ . The clipped gradient is $(3,4)$ , with norm five. The update is $(-0.3,-0.4)$ , with norm $0.5=\eta\tau$ .

Coordinate clipping instead gives $(5,5)$ , whose norm is $\sqrt{50}\approx7.071068$ . It changes direction and exceeds the proposed global threshold. It is a valid different rule, not an implementation of global norm clipping.

When the original norm is already below five, the global rule leaves it unchanged. A zero vector also stays zero; handle that case without dividing by zero. Neither rule repairs a non-finite gradient.

I.9.X03Expand the memory of a linear cellsymbolic▲▲△

For $h_t=wh_{t-1}+ux_t$ with fixed $h_0$ , derive a closed form for $h_T$ and for $\partial h_T/\partial w$ . Verify the derivative at $T=3$ , $h_0=0$ , $u=1$ , $w=1/2$ , and $(x_1,x_2,x_3)=(1,2,0)$ .

Hint

Expand two steps, identify the coefficient of each input, and differentiate the resulting polynomial.

Solution

Each earlier input is multiplied by $w$ once for every remaining step:

h_T=w^Th_0+u\sum_{s=1}^T w^{T-s}x_s.

The formula follows by induction: multiplying the expression for $T-1$ by $w$ raises each exponent, and adding $ux_T$ supplies the final term.

For integer $T\geq1$ ,

\frac{\partial h_T}{\partial w} =Tw^{T-1}h_0+ u\sum_{s=1}^{T-1}(T-s)w^{T-s-1}x_s.

The last input’s derivative is omitted rather than written as zero times a negative power. This polynomial formula is valid at $w=0$ too, using the ordinary constant monomial $w^0=1$ .

For the specified values, $h_3=w^2+2w=1.25$ and its derivative is $2w+2=3$ . With readout $z=2h_3$ and target one, multiplying by $(z-1)2=3$ gives the loss derivative nine from I.9.B02.

The closed form relies on a linear transition with constant parameters. Tanh prevents this distributive expansion.

I.9.X04Small does not mean rounded to zerolimit▲▲△

A scalar terminal gradient starts at one and is transported through thirty identical Jacobian factors. Compute the result for factors $0.7$ , $1$ and $1.3$ . Is the first numerically zero? Then consider thirty alternating factors $2,1/2$ .

Hint

The constant-factor example is $r^{30}$ . Pair the alternating factors before evaluating the second product.

Solution

The three magnitudes are

\begin{aligned} 0.7^{30}&\approx0.00002253934,\\ 1^{30}&=1,\\ 1.3^{30}&\approx2619.995644. \end{aligned}

The first is representable in ordinary float32 and float64 arithmetic. It may be small relative to other gradient contributions, but “small” and “underflowed to zero” are different claims.

As the gap tends to infinity, a constant factor with magnitude below one tends to zero; magnitude above one grows without bound; magnitude one preserves magnitude. A negative factor can reverse signs.

The alternating product is $(2\cdot1/2)^{15}=1$ . It disproves the blanket statement that every long product of factors different from one must vanish or explode. In real recurrent cells the factors vary with the trajectory; this exercise isolates only the constant-scalar mechanism.

I.9.X05Two large norms can make a contractioncounterexample▲▲△

Let $A=\operatorname{diag}(2,1/4)$ and $B=\operatorname{diag}(1/4,2)$ . Compute their spectral norms and the norm of the two-step product $BA$ . What happens after $n$ repetitions of this pair? Explain why the upper bound in (I.9.6) is not an equality.

Hint

Each diagonal matrix expands a different coordinate.

Solution

Both factor norms are two, since the spectral norm of a real diagonal matrix is its largest absolute diagonal entry. But

BA=\operatorname{diag}(1/2,1/2)=\tfrac12 I.

The actual product norm is $1/2$ , while the product of the factor norms is four. After $n$ pairs, every vector is multiplied by $2^{-n}$ . There is uniform contraction despite every individual factor having norm above one.

The first factor’s most expanded direction is contracted by the second. Multiplying norms forgets this alignment and gives an upper bound. A bound that permits growth does not prove growth.

These two diagonal matrices are an abstract Jacobian counterexample. It is enough to refute an inference made solely from their norms; no claim is made that every tanh trajectory realises this exact pair.

I.9.X06Padding and a new stream are different boundariesnumeric▲▲△

Use $h_t=0.5h_{t-1}+x_t$ , initially zero. An example has one real input $2$ followed by one zero padding input. Compare the final state with and without a state mask. A new independent example then begins with input $1$ . Compare resetting its initial state with wrongly carrying the previous valid state. Does detaching the carried value fix the latter?

Hint

A zero input still applies the recurrent matrix.

Solution

After the real input, $h=2$ . An ordinary zero-input update yields $0.5(2)+0=1$ . Freezing the state at the padding position keeps it at two. Masking only the padded loss does not change that unmasked update.

For the new independent example, resetting gives $h=0.5(0)+1=1$ . Wrongly carrying the valid state two gives $h=0.5(2)+1=2$ . The result depends on an unrelated example.

Detaching retains the number two. It stops a derivative, not a contaminating forward value, so the new example still produces two. Reset at independent-sequence boundaries. Carry and detach only when the next chunk continues the same stream and truncated training is intended.

I.9.X08Count histories before promising perfect recallproof▲▲△

An implementation stores eight hidden coordinates, each with at most 256 possible values. Show that it cannot uniquely encode every binary history of length 65 using only this state. Does the same argument prevent it from remembering parity?

Hint

Compare the number of representable states with the number of histories.

Solution

The state has at most $256^8=(2^8)^8=2^{64}$ distinct configurations. There are $2^{65}$ binary histories. At least two histories must reach the same state by the pigeonhole principle. Given the same future inputs, a deterministic transition and readout cannot subsequently distinguish those two histories.

Parity has only two possible answers. A two-state update can retain it exactly, switching state whenever the next bit is one. Thus the counting proof rules out lossless recall of all histories, not every task on histories.

The premise is an upper bound on representable states. It does not claim all such states are reachable or reliably distinguishable in practice. For arbitrary-precision ideal real numbers, this finite counting argument does not apply.

I.9.X09A backward encoder can see the answernumeric▲▲△

A backward linear encoder obeys $\overleftarrow h_t=x_t+0.5\overleftarrow h_{t+1}$ with terminal state $\overleftarrow h_4=0$ . Compute its state at position one for $(1,2,3)$ , then for $(1,2,7)$ . Could a representation containing that state be used unchanged to predict the next token causally after reading only position one?

Hint

Evaluate from position three to position one.

Solution

For $(1,2,3)$ the backward states at positions three, two and one are $3$ , $3.5$ and $2.75$ . For $(1,2,7)$ they are $7$ , $5.5$ and $3.75$ .

Position one’s representation changes even though the observed prefix $(1)$ is identical. It depends on the future. Using it unchanged for causal next-token prediction leaks information that would be unavailable at generation time.

A bidirectional encoder is appropriate when the full input is already observed, such as tagging a completed sentence. The objection is to using future context for a task whose information boundary forbids it, not to bidirectional encoding itself.

I.9.X10A wide span can still contain holescomplexity▲▲△

A stride-one causal convolution stack uses width three and dilations $1,2,4,\ldots$ . Compute the receptive-field span after three layers. How many such layers are needed for a span of at least 100? Now replace the first three dilations by $(2,2,2)$ . Does its span equal its number of distinct visible input positions?

Hint

Use (I.9.8), then enumerate sums of offsets in $\{0,d,2d\}$ .

Solution

For doubling dilations,

R_L=1+2(2^L-1)=2^{L+1}-1.

Three layers give $R_3=15$ . Five give $63$ , which is too small; six give $127$ , which is sufficient. For this schedule all integer offsets from zero through $R_L-1$ are reachable: adding offsets $0,d,2d$ to the previous contiguous range produces overlapping ranges at each layer.

For $(2,2,2)$ the span is $1+2(2+2+2)=13$ . But only even offsets are reachable:

\{0,2,4,6,8,10,12\}.

That is seven positions inside a span of thirteen. Every path adds even offsets, so odd offsets cannot appear, however the weights are trained.

These counts describe structural access for an interior output with enough history. Near the sequence start, part of the field is padding. Zero learned weights can further reduce actual influence. A receptive field is a set of possible dependencies, not a guarantee of information use.