Md. Asif Uddin
    I.12.X08

    The best denoiser returns a conditional mean

    probability▲▲△

    A scalar input is +1+1 or −1-1 with probability one half each. It is corrupted to x~=x+ε\tilde x=x+\varepsilon with ε\varepsilon drawn from N(0,σ2)\mathcal N(0,\sigma^2).

    (a) Find the best denoiser under squared error, E[x∣x~]\E[x\mid\tilde x].

    (b) Evaluate it at x~=0.5\tilde x=0.5 for σ2=1\sigma^2=1 and for σ2=0.25\sigma^2=0.25, and at x~=0\tilde x=0.

    (c) Verify (I.12.8) for this distribution.

    (d) The output at x~=0\tilde x=0 is not a possible input. What does that say about reconstructions trained with squared error?

    Hint

    Use Bayes’ rule (Bayes' rule 0.PR.04) for the two posterior probabilities. The Gaussian normalising constants cancel in their ratio.

    Solution

    (a) The two likelihoods are proportional to e−(x~−1)2/2σ2e^{-(\tilde x-1)^2/2\sigma^2} and e−(x~+1)2/2σ2e^{-(\tilde x+1)^2/2\sigma^2}, and the priors are equal. Their ratio is e2x~/σ2e^{2\tilde x/\sigma^2}, so the posterior probabilities are

    p±=e±x~/σ2ex~/σ2+e−x~/σ2.p_\pm=\frac{e^{\pm\tilde x/\sigma^2}} {e^{\tilde x/\sigma^2}+e^{-\tilde x/\sigma^2}}.

    The conditional mean is p+−p−p_+-p_-:

    E[x∣x~]=tanh⁡ ⁣(x~/σ2).\E[x\mid\tilde x]=\tanh\!\big(\tilde x/\sigma^2\big).

    (b) At x~=0.5\tilde x=0.5: tanh⁡(0.5)=0.462117\tanh(0.5)=0.462117 for σ2=1\sigma^2=1 and tanh⁡(2)=0.964028\tanh(2)=0.964028 for σ2=0.25\sigma^2=0.25. At x~=0\tilde x=0 it is 00 for any σ2\sigma^2.

    With less noise the same observation is stronger evidence, and the denoiser commits to the nearer input. As σ2→0\sigma^2\to0 it becomes the sign of x~\tilde x. As σ2→∞\sigma^2\to\infty it flattens to the prior mean, 00.

    (c) The noised density is proportional to the sum of the two likelihoods. Differentiating its logarithm,

    ddx~log⁡pσ(x~)    =−(x~−1) p+−(x~+1) p−σ2    =tanh⁡(x~/σ2)−x~σ2.\begin{aligned} &\frac{d}{d\tilde x}\log p_\sigma(\tilde x)\\ &\;\;=\frac{-(\tilde x-1)\,p_+-(\tilde x+1)\,p_-}{\sigma^2}\\ &\;\;=\frac{\tanh(\tilde x/\sigma^2)-\tilde x}{\sigma^2}. \end{aligned}

    So x~+σ2\tilde x+\sigma^2 times the score is tanh⁡(x~/σ2)\tanh(\tilde x/\sigma^2), as (I.12.8) requires. The displacement of the denoiser points towards the nearer of the two inputs, where the noised density is higher. The script checks this with a numerical derivative at all three points of (b).

    (d) At x~=0\tilde x=0 the two inputs are equally likely. The value that minimises the expected squared error is their mean, 00, even though the input is never 00. Squared error rewards the average of the plausible answers and not any one of them.

    For images, the average of several sharp reconstructions that differ slightly is a blurred one. An autoencoder trained with squared error on inputs it cannot resolve returns that average. More training does not sharpen it, because the average is the optimum.

    Draws on