Problems

STATEMENT

Compute a batch-normalisation forward pass by hand, then serve one of the same values using frozen statistics. Distinguish stored statistics from statistics computed from the current inputs.

GIVEN

One feature has training values $x=(1,2,3)$, learned scale $\gamma=1$, offset $\beta=0$, and $\varepsilon=10^{-5}$. For the evaluation calculation the saved statistics are explicitly $\mu_{\mathrm{run}}=2$ and $v_{\mathrm{run}}=2/3$. They are supplied values, not the result of a specified running-average update.

FIND

The batch mean, population-divisor variance, standardised values, their mean and variance, and the evaluation output for the singleton input $x=3$. Then compute the output if that singleton is incorrectly treated as a training batch.

STRATEGY

Use (I.8.1) for the training group. Reuse the supplied constants, not a new estimate, in the evaluation branch of (I.8.2).

SOLUTION

The mean and variance are

The denominator is $\sqrt{2/3+10^{-5}}$. The centred values are $(-1,0,1)$, so the outputs, rounded to four decimal places, are

The mean is exactly zero by symmetry. Their variance is not exactly one:

Evaluation with the supplied saved statistics gives $(3-2)/\sqrt{2/3+10^{-5}}=1.2247$. The other inputs do not need to be present.

If the singleton instead supplies its own statistics, its mean is $3$ and variance is zero. Subtracting that mean yields zero, so the output becomes $\beta=0$. This is the algebraic singleton failure. Some frameworks reject such a training group rather than returning that degenerate value.

The following 13-line implementation reproduces the two valid outputs:

from math import sqrt

def batch_norm(x, *, stats=None, gamma=1.0, beta=0.0, eps=1e-5):
    if stats is None:
        mu = sum(x) / len(x)
        var = sum((v - mu)**2 for v in x) / len(x)
    else:
        mu, var = stats
    return [gamma * (v - mu) / sqrt(var + eps) + beta for v in x]

print([round(v, 4) for v in batch_norm([1.0, 2.0, 3.0])])
print([round(v, 4) for v in batch_norm([3.0], stats=(2.0, 2.0/3.0))])

Check — sanity

The middle entry equals the mean and must map to zero before the affine transformation. The two endpoints have opposite signs. Increasing epsilon must reduce both magnitudes; setting epsilon to zero here would give $\pm\sqrt{3/2}$ because the original variance is positive.

STATEMENT

Derive the batch-normalisation backward pass including epsilon. Check it against finite differences, and identify the term lost by treating the variance as a constant.

GIVEN

A group of $m$ scalars, $\mu=m^{-1}\sum_i x_i$, $v=m^{-1}\sum_i(x_i-\mu)^2$, $r=\sqrt{v+\varepsilon}$, $\hat x_i=(x_i-\mu)/r$, and $y_i=\gamma\hat x_i+\beta$. The upstream derivative is $g_i=\partial\mathcal L/\partial y_i$. For a numerical check take $x=(-1,0,1)$, $\varepsilon=1/3$, $\gamma=2$, $\beta=0$, and $g=(1,-1,2)$.

FIND

The full Jacobian of the standardisation, input gradients, affine-parameter gradients, and the reason the input gradients sum to zero. Decide whether the gradient is also orthogonal to the centred input when epsilon is positive.

STRATEGY

Differentiate the mean first, then the variance, then the inverse standard deviation. Only then contract the Jacobian with the upstream derivative.

SOLUTION

Write $c_i=x_i-\mu$. Since $\sum_i c_i=0$,

Consequently $\partial r/\partial x_j=c_j/(mr)$. The product rule applied to $c_i r^{-1}$ gives

which is (I.8.3). Omitting the derivative of the variance loses the last term.

Contract with $\gamma g_i$ and sum over $i$:

This is (I.8.4). The parameter derivatives are

For the supplied check, $\mu=0$, $v=2/3$, and $r=1$ exactly. Thus $\hat x=(-1,0,1)$, $\overline g=2/3$, and $\overline{g\hat x}=1/3$. Substitution gives

To check an input derivative numerically, define $F(x)=\sum_i g_i y_i(x)$ and recompute all statistics after each perturbation:

The reproduction script uses $h=10^{-5}$ and asserts a maximum absolute disagreement below $10^{-8}$. It also tests a non-symmetric input, so the check does not rely only on $r=1$.

Summing the analytic gradients cancels both mean terms because $\sum_j\hat x_j=0$. Their sum is zero. A common shift in every input cannot change the normalised output.

The centred input is different. Direct multiplication gives

For nonzero centred data this vanishes only when epsilon is zero. In the numerical example $\vec c^{\mathsf T}\nabla_x\mathcal L=2/3$, not zero. Exact scale invariance would predict the wrong answer.

Check — sanity

A constant upstream gradient produces zero input gradient when gamma is shared over the batch group: it asks to change a sum that normalisation has fixed. The computation needs two reductions and one elementwise pass, not storage of an $m$-by-$m$ matrix.

STATEMENT

Refute the claim that batch normalisation during training defines a function of one example alone. Then show why layer normalisation is not a drop-in mathematical equivalent.

GIVEN

One scalar feature, $\gamma=1$, $\beta=0$, and $\varepsilon=10^{-5}$. The target example has value $1$. In batch A it is paired with $3$; in batch B it is paired with $-1$.

FIND

The output for the target example in each batch. Repeat the reasoning for layer normalisation applied to the target row $(1,3)$, and for a normalisation group containing only one scalar.

STRATEGY

Hold the target fixed and change only a member of its reduction group. The difference between the two algorithms is whether that changed value belongs to the target’s group.

SOLUTION

Batch A has mean $2$ and variance $1$. Its target output is

Batch B has mean $0$ and variance $1$, so

The unrounded magnitudes are below one. The sign change is exact. The feature, parameters and epsilon did not change. The companion example did. This is a counterexample to a sample-independent training map, not a failure of the implementation.

For layer normalisation over the target’s row $(1,3)$, its statistics remain mean $2$ and variance $1$ regardless of other rows. Its output stays $(-1,+1)/\sqrt{1+10^{-5}}$. It depends on its other feature rather than on other examples.

If the reduction group has size one, $\mu=x$ and $v=0$. The output is $\gamma(0/\sqrt\varepsilon)+\beta=\beta$ for every input. Its derivative with respect to the input is zero. Dense batch normalisation can encounter this at batch size one. Layer normalisation can encounter it when the normalised feature width is one, even with a large batch.

Check — sanity

The comparison uses the same variance in both batches, so the sign reversal comes entirely from the change of mean. Ordinary evaluation-mode batch normalisation with frozen statistics would also be independent of the companion example.

STATEMENT

Audit the axes, statistics and parameter shapes of batch and layer normalisation. An output tensor with the expected shape is not sufficient evidence that the intended axes were used.

GIVEN

First a dense matrix of shape $(N,D)=(2,3)$. Then a channels-first image tensor of shape $(N,C,H,W)=(2,3,4,4)$. Use learned affine parameters. Compare spatial batch normalisation, layer normalisation over all $(C,H,W)$, and channel-only layer normalisation performed separately at each spatial position.

FIND

The number of groups, entries per group, learned parameters and stored running-statistic scalars. Identify an implementation that silently normalises the wrong axis.

STRATEGY

Statistics have one value per reduction group. Learned affine parameters have one value per specified feature position and need not have the same shape as the statistics. Count mean and variance separately from gamma and beta.

SOLUTION

For the dense matrix:

Both return shape $(2,3)$. Their identical parameter counts hide different dependencies.

For the image tensor:

These layer-normalisation counts use an independent affine parameter for each position of the normalised shape, shared over the non-normalised axes. A deliberately tied affine parameterisation would have different counts.

The broadcast shapes of the spatial batch statistics and affine parameters are $(1,C,1,1)$. For full-example layer normalisation, the computed statistics have shape $(N,1,1,1)$, but gamma and beta each have shape $(C,H,W)$. For channel-only layer normalisation, move channels to the last axis, normalise that axis with shape $C$, then restore the original axis order.

Applying a last-axis normalisation of width $W$ directly to channels-first data instead computes a mean across each row of pixels. It returns the same overall tensor shape. Nothing about that successful return makes it channel normalisation.

Every entry participates in a constant number of reductions and elementwise operations, so forward and efficient backward arithmetic are $O(NCHW)$. The full dense Jacobian is unnecessary.

Check — sanity

Groups multiplied by entries per group must equal the tensor’s $96$ entries for each rule. Running-statistic buffers are not learned parameters, and a framework’s bookkeeping counter is separate from the mean and variance counted here.

STATEMENT

Compute dropout’s distribution exactly, rather than estimating it by sampling. Use the result to test the claim that deterministic inference is the exact average of the masked networks.

GIVEN

A fixed activation $h=2$, independent keep probability $q=1/2$, inverted dropout $\widetilde h=Mh/q$, and a downstream function $f(u)=u^2$.

FIND

The mean and variance of the masked activation, the expected squared output, and the deterministic inference prediction. Then derive the general formulas for fixed $h$ and $0<q\leq1$.

STRATEGY

Enumerate the two mask outcomes. Keep $\mathbb E[f(\widetilde h)]$ separate from $f(\mathbb E[\widetilde h])$.

SOLUTION

The first moment is $(0+4)/2=2$. The second moment is $(0+16)/2=8$. Subtracting the square of the first moment gives variance $8-2^2=4$.

At inference inverted dropout leaves $h$ unchanged, so the squared output is $f(2)=4$. But averaging the masked predictions gives $8$. The discrepancy is the activation variance, not a sampling error.

For arbitrary fixed $h$, the outcomes are zero and $h/q$:

Therefore $\operatorname{Var}(\widetilde h)=h^2(1-q)/q$, as in (I.8.8). For an affine downstream map $au+b$, expectation does commute with the map. The square gives a concrete case where it does not.

For independent masks on several fixed activations, the dropout-induced off-diagonal covariances are zero. This is conditional on the activations. It does not claim that the activations themselves are independent across data.

Check — sanity

At $q=1$ both paths coincide and the variance is zero. For fixed nonzero $h$, the variance grows without bound as $q$ tends to zero. The operation is not defined at $q=0$ by the formula used here.