Five activations at five points

STATEMENT

Evaluate sigmoid, tanh, ReLU, LeakyReLU with slope $0.01$, and GELU at $z \in \{-2,\ -0.5,\ 0,\ 0.5,\ 2\}$. Then read four structural facts off the table that no single value shows.

GIVEN

with $\Phi$ the standard normal CDF.

FIND

A $5\times 5$ table of values, and four properties visible only across it.

STRATEGY

Compute the two exponential families first, since tanh follows from sigmoid; the two rectifiers need no arithmetic at all; GELU needs $\Phi$.

SOLUTION

Step 1 — sigmoid. $\sigma(-2) = 1/(1+e^{2}) = 1/8.3891 = 0.1192$. By the symmetry $\sigma(-z) = 1 - \sigma(z)$, $\sigma(2) = 0.8808$ without further work. $\sigma(0) = 1/2$ exactly. $\sigma(-0.5) = 1/(1+e^{0.5}) = 1/2.6487 = 0.3775$, so $\sigma(0.5) = 0.6225$.

Step 2 — tanh from sigmoid. Using $\tanh(z) = 2\sigma(2z) - 1$: $\tanh(0.5) = 2\sigma(1) - 1 = 2(0.7311) - 1 = 0.4621$, and $\tanh(2) = 2\sigma(4) - 1 = 2(0.9820) - 1 = 0.9640$. Odd symmetry gives the negatives.

Step 3 — the rectifiers. No arithmetic. ReLU zeroes the negatives and copies the positives; LeakyReLU scales the negatives by $0.01$.

Step 4 — GELU. $\Phi(-2) = 0.0228$, so $\mathrm{GELU}(-2) = (-2)(0.0228) = -0.0455$. $\Phi(0) = 0.5$ so $\mathrm{GELU}(0) = 0$. $\Phi(2) = 0.9772$, giving $1.9545$.

The table.

Four structural facts.

Only sigmoid fails to pass zero to zero. $\sigma(0) = 0.5$. Every other column has $0 \mapsto 0$. A layer of sigmoids therefore emits a nonzero mean even from centred input, and that offset compounds with depth — the historical reason tanh replaced sigmoid in hidden layers.

Sigmoid and tanh are bounded; the rectifiers are not. At $z = 2$ the bounded pair are already at $0.88$ and $0.96$, close to their ceilings, while ReLU returns $2$ and would return $200$ at $z = 200$. Boundedness is what causes saturation and also what prevents blow-up.

GELU is not monotonic. From $z = -2$ to $z = -0.5$ the output falls from $-0.0455$ to $-0.1543$. No other column does this. GELU is a soft gate, not a soft switch, and this dip is where its behaviour genuinely differs from a smoothed ReLU.

GELU is close to ReLU where it matters and different where it does not. At $z = 2$ the two differ by $0.0455$; at $z = -0.5$ they differ by $0.1543$. The whole difference lives in a band around the origin, which is exactly where gradients are decided.

Check — sanity

The symmetries hold. $\sigma(-2) + \sigma(2) = 0.1192 + 0.8808 = 1.0000$ exactly, as $\sigma(-z) = 1 - \sigma(z)$ requires. And $\tanh(-2) = -\tanh(2)$ to every printed digit.

The identity $\tanh(z) = 2\sigma(2z) - 1$ checks out. At $z = 0.5$: $2\sigma(1) - 1 = 2(0.7311) - 1 = 0.4622$, against $\tanh(0.5) = 0.4621$. The last digit differs by rounding of the intermediate, not by error.

GELU is between $0$ and $z$ for $z > 0$. $0 < 0.3457 < 0.5$ and $0 < 1.9545 < 2$. Since $\Phi \in (0,1)$ and $\mathrm{GELU} = z\Phi(z)$, that must hold — and it is the fastest way to catch a sign or scale slip.