The same example changes sign in another batch

STATEMENT

Refute the claim that batch normalisation during training defines a function of one example alone. Then show why layer normalisation is not a drop-in mathematical equivalent.

GIVEN

One scalar feature, $\gamma=1$, $\beta=0$, and $\varepsilon=10^{-5}$. The target example has value $1$. In batch A it is paired with $3$; in batch B it is paired with $-1$.

FIND

The output for the target example in each batch. Repeat the reasoning for layer normalisation applied to the target row $(1,3)$, and for a normalisation group containing only one scalar.

STRATEGY

Hold the target fixed and change only a member of its reduction group. The difference between the two algorithms is whether that changed value belongs to the target’s group.

SOLUTION

Batch A has mean $2$ and variance $1$. Its target output is

Batch B has mean $0$ and variance $1$, so

The unrounded magnitudes are below one. The sign change is exact. The feature, parameters and epsilon did not change. The companion example did. This is a counterexample to a sample-independent training map, not a failure of the implementation.

For layer normalisation over the target’s row $(1,3)$, its statistics remain mean $2$ and variance $1$ regardless of other rows. Its output stays $(-1,+1)/\sqrt{1+10^{-5}}$. It depends on its other feature rather than on other examples.

If the reduction group has size one, $\mu=x$ and $v=0$. The output is $\gamma(0/\sqrt\varepsilon)+\beta=\beta$ for every input. Its derivative with respect to the input is zero. Dense batch normalisation can encounter this at batch size one. Layer normalisation can encounter it when the normalised feature width is one, even with a large batch.

Check — sanity

The comparison uses the same variance in both batches, so the sign reversal comes entirely from the change of mean. Ordinary evaluation-mode batch normalisation with frozen statistics would also be independent of the companion example.