Md. Asif Uddin
    I.8.X07

    A mask can keep the right fraction and still be biased

    limit▲▲△

    For fixed nonzero hh, examine (I.8.8) as qq tends to one and as qq tends to zero. Next let hh be equally likely to be −1-1 or 11 and keep it exactly when h=1h=1. The overall keep fraction is 1/21/2. Does dividing by that fraction preserve the mean?

    Hint

    The theorem conditions on hh. An overall keep fraction does not establish independence between the mask and the activation.

    Solution

    At q=1q=1, dropout is the identity and its induced variance is zero. As qq tends to zero through positive values, the mean remains hh but the variance h2(1−q)/qh^2(1-q)/q diverges. Rare surviving activations become arbitrarily large. The formula cannot be evaluated at q=0q=0.

    In the dependent-mask example, Mh/qMh/q is zero when h=−1h=-1 and two when h=1h=1. Its mean is one, whereas E[h]=0\mathbb E[h]=0. The mask kept half the examples but preferentially selected the positive ones. Conditional on h=−1h=-1, its keep probability is zero, not the advertised qq.

    The usual expectation calculation needs a mask sampled independently of the fixed activation, or a separately specified conditional sampling and weighting scheme. Matching an aggregate drop fraction is insufficient.

    Draws on