Three identities that make the arithmetic disappear

STATEMENT

Derive $\sigma'(z) = \sigma(z)\,(1 - \sigma(z))$ and $\tanh'(z) = 1 - \tanh^2(z)$ from the definitions, then show $\tanh(z) = 2\sigma(2z) - 1$. Say in each case what the identity buys.

GIVEN

$\sigma(z) = (1 + e^{-z})^{-1}$ and $\tanh(z) = (e^{z} - e^{-z})/(e^{z} + e^{-z})$.

FIND

The two derivative identities and the relation between the functions.

STRATEGY

Differentiate by the chain rule, then rewrite the result in terms of the function itself rather than of $z$ — that rewriting is the entire content of each identity.

SOLUTION

Step 1 — differentiate sigmoid. Write $\sigma = u^{-1}$ with $u = 1 + e^{-z}$, so $u' = -e^{-z}$. By the chain rule (The chain rule · 0.MC.03):

$\sigma'(z) = -u^{-2}\,u' = \frac{e^{-z}}{(1+e^{-z})^{2}}$

Step 2 — rewrite in terms of $\sigma$. Split the fraction deliberately:

For the second factor, add and subtract $1$ in the numerator:

$\frac{e^{-z}}{1+e^{-z}} = \frac{(1+e^{-z}) - 1}{1+e^{-z}} = 1 - \sigma(z)$

\boxed{\ \sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big)\ } \tag{I.3.1}

What it buys. The derivative costs nothing extra at run time. The forward pass already computed $\sigma(z)$; the backward pass needs only one multiply and one subtract, and never touches $e^{-z}$ again. That is why frameworks cache activations rather than pre-activations for these units.

Step 3 — differentiate tanh. Quotient rule on $\tanh = (e^{z}-e^{-z})/(e^{z}+e^{-z})$. Writing $N$ and $D$ for numerator and denominator, $N' = D$ and $D' = N$, so

$\tanh'(z) = \frac{N'D - ND'}{D^{2}} = \frac{D^{2} - N^{2}}{D^{2}} = 1 - \left(\frac{N}{D}\right)^{2}$

$\boxed{\ \tanh'(z) = 1 - \tanh^{2}(z)\ }$

What it buys. The same as before, and one more thing: at $z = 0$, $\tanh' = 1 - 0 = 1$. Compared with $\sigma'(0) = 1/4$, tanh passes four times as much gradient at its best point. That factor of four, compounded over depth, is the quantitative form of “tanh trains better than sigmoid”.

Step 4 — the relation between them. Start from the right-hand side:

Multiply numerator and denominator by $e^{z}$:

$= \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}} = \tanh(z)$

\boxed{\ \tanh(z) = 2\sigma(2z) - 1\ } \tag{I.3.2}

What it buys. Tanh is not a second idea. It is sigmoid, stretched vertically by $2$, shifted down by $1$, and compressed horizontally by $2$. Every property of one transfers to the other with those three transformations applied — including, by the chain rule, the derivative relation $\tanh'(z) = 4\sigma'(2z)$, which is where the factor of four came from.

Check — sanity

Numerically at $z = 0.5$. $\sigma(0.5) = 0.6225$, so (I.3.1) predicts $\sigma'(0.5) = 0.6225 \times 0.3775 = 0.2350$. Differencing numerically: $(\sigma(0.501) - \sigma(0.499))/0.002 = 0.2350$. Agreement to four digits.

The maxima are where they should be. $\sigma(1-\sigma)$ is a downward parabola in $\sigma$, maximised at $\sigma = 1/2$, i.e. $z = 0$, with value $1/4$. And $1 - \tanh^2$ is maximised where $\tanh = 0$, again $z = 0$, with value $1$. Both match the shapes plotted from I.3.B01.

The factor of four is consistent. Differentiating (I.3.2) gives $\tanh'(z) = 4\sigma'(2z)$. At $z = 0$: $1 = 4 \times 0.25$. ✓