A mean-preserving mask does not preserve a prediction

STATEMENT

Compute dropout’s distribution exactly, rather than estimating it by sampling. Use the result to test the claim that deterministic inference is the exact average of the masked networks.

GIVEN

A fixed activation $h=2$, independent keep probability $q=1/2$, inverted dropout $\widetilde h=Mh/q$, and a downstream function $f(u)=u^2$.

FIND

The mean and variance of the masked activation, the expected squared output, and the deterministic inference prediction. Then derive the general formulas for fixed $h$ and $0<q\leq1$.

STRATEGY

Enumerate the two mask outcomes. Keep $\mathbb E[f(\widetilde h)]$ separate from $f(\mathbb E[\widetilde h])$.

SOLUTION

The first moment is $(0+4)/2=2$. The second moment is $(0+16)/2=8$. Subtracting the square of the first moment gives variance $8-2^2=4$.

At inference inverted dropout leaves $h$ unchanged, so the squared output is $f(2)=4$. But averaging the masked predictions gives $8$. The discrepancy is the activation variance, not a sampling error.

For arbitrary fixed $h$, the outcomes are zero and $h/q$:

Therefore $\operatorname{Var}(\widetilde h)=h^2(1-q)/q$, as in (I.8.8). For an affine downstream map $au+b$, expectation does commute with the map. The square gives a concrete case where it does not.

For independent masks on several fixed activations, the dropout-induced off-diagonal covariances are zero. This is conditional on the activations. It does not claim that the activations themselves are independent across data.

Check — sanity

At $q=1$ both paths coincide and the variance is zero. For fixed nonzero $h$, the variance grows without bound as $q$ tends to zero. The operation is not defined at $q=0$ by the formula used here.