Sigmoid and tanh, and the identity between them

Compute $\sigma$ first and get everything else from it.

Working at $z = -1$. $\sigma(-1) = 1/(1+e) = 1/3.7183 = 0.2689$. Then $\sigma' = (0.2689)(0.7311) = 0.1966$. For tanh use the identity: $2\sigma(-2) - 1 = 2(0.1192) - 1 = -0.7616$, and $\tanh' = 1 - (-0.7616)^2 = 1 - 0.5800 = 0.4200$.

The identity holds to every printed digit at all three points, which is what an exact identity should do.

The ratio at $z = 0$. $\tanh'(0)/\sigma'(0) = 1.0000/0.2500 = \mathbf{4}$.

Why that four matters. Over $L$ layers the gradient is multiplied by one derivative per layer, so at their best points a tanh stack passes $4^{L}$ times more gradient than a sigmoid stack. At $L = 10$ that is a factor of about a million. It is the entire quantitative content of the historical preference for tanh over sigmoid in hidden layers, and I.3.B02 showed it follows from the identity by the chain rule: $\tanh'(z) = 4\sigma'(2z)$.

But notice the third row. At $z = 1.5$ the two derivatives are $0.1491$ and $0.1807$ — a ratio of $1.21$, not $4$. The advantage is largest at the origin and shrinks as either function saturates. Tanh delays the problem; it does not solve it, which is why Chapter I.8 exists.