Problems

STATEMENT

Evaluate sigmoid, tanh, ReLU, LeakyReLU with slope $0.01$, and GELU at $z \in \{-2,\ -0.5,\ 0,\ 0.5,\ 2\}$. Then read four structural facts off the table that no single value shows.

GIVEN

with $\Phi$ the standard normal CDF.

FIND

A $5\times 5$ table of values, and four properties visible only across it.

STRATEGY

Compute the two exponential families first, since tanh follows from sigmoid; the two rectifiers need no arithmetic at all; GELU needs $\Phi$.

SOLUTION

Step 1 — sigmoid. $\sigma(-2) = 1/(1+e^{2}) = 1/8.3891 = 0.1192$. By the symmetry $\sigma(-z) = 1 - \sigma(z)$, $\sigma(2) = 0.8808$ without further work. $\sigma(0) = 1/2$ exactly. $\sigma(-0.5) = 1/(1+e^{0.5}) = 1/2.6487 = 0.3775$, so $\sigma(0.5) = 0.6225$.

Step 2 — tanh from sigmoid. Using $\tanh(z) = 2\sigma(2z) - 1$: $\tanh(0.5) = 2\sigma(1) - 1 = 2(0.7311) - 1 = 0.4621$, and $\tanh(2) = 2\sigma(4) - 1 = 2(0.9820) - 1 = 0.9640$. Odd symmetry gives the negatives.

Step 3 — the rectifiers. No arithmetic. ReLU zeroes the negatives and copies the positives; LeakyReLU scales the negatives by $0.01$.

Step 4 — GELU. $\Phi(-2) = 0.0228$, so $\mathrm{GELU}(-2) = (-2)(0.0228) = -0.0455$. $\Phi(0) = 0.5$ so $\mathrm{GELU}(0) = 0$. $\Phi(2) = 0.9772$, giving $1.9545$.

The table.

Four structural facts.

Only sigmoid fails to pass zero to zero. $\sigma(0) = 0.5$. Every other column has $0 \mapsto 0$. A layer of sigmoids therefore emits a nonzero mean even from centred input, and that offset compounds with depth — the historical reason tanh replaced sigmoid in hidden layers.

Sigmoid and tanh are bounded; the rectifiers are not. At $z = 2$ the bounded pair are already at $0.88$ and $0.96$, close to their ceilings, while ReLU returns $2$ and would return $200$ at $z = 200$. Boundedness is what causes saturation and also what prevents blow-up.

GELU is not monotonic. From $z = -2$ to $z = -0.5$ the output falls from $-0.0455$ to $-0.1543$. No other column does this. GELU is a soft gate, not a soft switch, and this dip is where its behaviour genuinely differs from a smoothed ReLU.

GELU is close to ReLU where it matters and different where it does not. At $z = 2$ the two differ by $0.0455$; at $z = -0.5$ they differ by $0.1543$. The whole difference lives in a band around the origin, which is exactly where gradients are decided.

Check — sanity

The symmetries hold. $\sigma(-2) + \sigma(2) = 0.1192 + 0.8808 = 1.0000$ exactly, as $\sigma(-z) = 1 - \sigma(z)$ requires. And $\tanh(-2) = -\tanh(2)$ to every printed digit.

The identity $\tanh(z) = 2\sigma(2z) - 1$ checks out. At $z = 0.5$: $2\sigma(1) - 1 = 2(0.7311) - 1 = 0.4622$, against $\tanh(0.5) = 0.4621$. The last digit differs by rounding of the intermediate, not by error.

GELU is between $0$ and $z$ for $z > 0$. $0 < 0.3457 < 0.5$ and $0 < 1.9545 < 2$. Since $\Phi \in (0,1)$ and $\mathrm{GELU} = z\Phi(z)$, that must hold — and it is the fastest way to catch a sign or scale slip.

STATEMENT

Derive $\sigma'(z) = \sigma(z)\,(1 - \sigma(z))$ and $\tanh'(z) = 1 - \tanh^2(z)$ from the definitions, then show $\tanh(z) = 2\sigma(2z) - 1$. Say in each case what the identity buys.

GIVEN

$\sigma(z) = (1 + e^{-z})^{-1}$ and $\tanh(z) = (e^{z} - e^{-z})/(e^{z} + e^{-z})$.

FIND

The two derivative identities and the relation between the functions.

STRATEGY

Differentiate by the chain rule, then rewrite the result in terms of the function itself rather than of $z$ — that rewriting is the entire content of each identity.

SOLUTION

Step 1 — differentiate sigmoid. Write $\sigma = u^{-1}$ with $u = 1 + e^{-z}$, so $u' = -e^{-z}$. By the chain rule (The chain rule · 0.MC.03):

$\sigma'(z) = -u^{-2}\,u' = \frac{e^{-z}}{(1+e^{-z})^{2}}$

Step 2 — rewrite in terms of $\sigma$. Split the fraction deliberately:

For the second factor, add and subtract $1$ in the numerator:

$\frac{e^{-z}}{1+e^{-z}} = \frac{(1+e^{-z}) - 1}{1+e^{-z}} = 1 - \sigma(z)$

\boxed{\ \sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big)\ } \tag{I.3.1}

What it buys. The derivative costs nothing extra at run time. The forward pass already computed $\sigma(z)$; the backward pass needs only one multiply and one subtract, and never touches $e^{-z}$ again. That is why frameworks cache activations rather than pre-activations for these units.

Step 3 — differentiate tanh. Quotient rule on $\tanh = (e^{z}-e^{-z})/(e^{z}+e^{-z})$. Writing $N$ and $D$ for numerator and denominator, $N' = D$ and $D' = N$, so

$\tanh'(z) = \frac{N'D - ND'}{D^{2}} = \frac{D^{2} - N^{2}}{D^{2}} = 1 - \left(\frac{N}{D}\right)^{2}$

$\boxed{\ \tanh'(z) = 1 - \tanh^{2}(z)\ }$

What it buys. The same as before, and one more thing: at $z = 0$, $\tanh' = 1 - 0 = 1$. Compared with $\sigma'(0) = 1/4$, tanh passes four times as much gradient at its best point. That factor of four, compounded over depth, is the quantitative form of “tanh trains better than sigmoid”.

Step 4 — the relation between them. Start from the right-hand side:

Multiply numerator and denominator by $e^{z}$:

$= \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}} = \tanh(z)$

\boxed{\ \tanh(z) = 2\sigma(2z) - 1\ } \tag{I.3.2}

What it buys. Tanh is not a second idea. It is sigmoid, stretched vertically by $2$, shifted down by $1$, and compressed horizontally by $2$. Every property of one transfers to the other with those three transformations applied — including, by the chain rule, the derivative relation $\tanh'(z) = 4\sigma'(2z)$, which is where the factor of four came from.

Check — sanity

Numerically at $z = 0.5$. $\sigma(0.5) = 0.6225$, so (I.3.1) predicts $\sigma'(0.5) = 0.6225 \times 0.3775 = 0.2350$. Differencing numerically: $(\sigma(0.501) - \sigma(0.499))/0.002 = 0.2350$. Agreement to four digits.

The maxima are where they should be. $\sigma(1-\sigma)$ is a downward parabola in $\sigma$, maximised at $\sigma = 1/2$, i.e. $z = 0$, with value $1/4$. And $1 - \tanh^2$ is maximised where $\tanh = 0$, again $z = 0$, with value $1$. Both match the shapes plotted from I.3.B01.

The factor of four is consistent. Differentiating (I.3.2) gives $\tanh'(z) = 4\sigma'(2z)$. At $z = 0$: $1 = 4 \times 0.25$. ✓

STATEMENT

Find the maximum of $\sigma'$. Compute the product of ten sigmoid derivatives at $|z| = 4$. Then find the depth at which the product underflows fp16, both at $|z| = 4$ and in the best case.

GIVEN

$\sigma'(z) = \sigma(z)(1-\sigma(z))$ from I.3.B02. The smallest positive fp16 value is the subnormal $2^{-24} = 5.96\times10^{-8}$ (Floating point · 0.NU.01). A gradient reaching layer $1$ of an $L$-layer stack is multiplied by one $\sigma'$ per layer.

FIND

$\max_z \sigma'(z)$; the value of $\sigma'(4)^{10}$; and the smallest $L$ with $\sigma'^{\,L} < 2^{-24}$ at $|z| = 4$ and at $z = 0$.

STRATEGY

Maximise the derivative by treating it as a quadratic in $\sigma$ rather than in $z$ — the substitution turns calculus into inspection. Then take logarithms to turn the repeated product into a linear count.

SOLUTION

Step 1 — the maximum. Let $s = \sigma(z) \in (0,1)$. Then $\sigma' = s(1-s) = s - s^2$, a downward parabola in $s$ with vertex at $s = 1/2$:

$\max \sigma' = \tfrac12\left(1 - \tfrac12\right) = \tfrac14 = 0.2500$

and $s = 1/2$ means $z = 0$. So the best a sigmoid ever does for a gradient is divide it by four, and it does that only at one point.

Step 2 — the derivative at $|z| = 4$. $\sigma(4) = 1/(1+e^{-4}) = 0.982014$, so

$\sigma'(4) = (0.982014)(0.017986) = 0.0176627$

Fourteen times smaller than the best case, from a pre-activation that is not extreme — $z = 4$ is entirely ordinary in an unnormalised network.

Step 3 — ten layers.

$\sigma'(4)^{10} = (0.0176627)^{10} = 2.955\times10^{-18}$

For comparison the best case gives $0.25^{10} = 9.537\times10^{-7}$. Even at every layer’s single most favourable point, ten layers cost six orders of magnitude.

Step 4 — the fp16 depth. Solve $g^{L} < 2^{-24}$ by taking logarithms:

$L > \frac{-24\ln 2}{\ln g} = \frac{-16.6355}{\ln g}$

At $|z| = 4$, $\ln(0.0176627) = -4.0362$, so $L > 4.12$, giving $\boxed{L = 5}$.

At $z = 0$, $\ln(0.25) = -1.3863$, so $L > 11.999$, giving $\boxed{L = 12}$.

What the two numbers say. A sigmoid network in fp16 has a hard ceiling of about twelve layers even under conditions that never occur — every unit sitting exactly at its most favourable point on every example. In realistic conditions, with pre-activations spread over a range including $|z| = 4$, the ceiling is five. Not “training is slow past five layers”: the gradient is exactly zero, because the product is not representable and rounds to it.

This is why deep networks were not trained before roughly 2010, and it is why three separate later chapters exist. ReLU (this chapter) removes the shrinking factor entirely for active units. Normalisation (I.8) keeps $z$ near the favourable region. Residual connections (II.6) provide a path with derivative exactly $1$ that no activation stands on.

Check — sanity

The maximum is attained where the function is symmetric. $\sigma'$ inherits $\sigma(-z) = 1-\sigma(z)$, so $\sigma'(-z) = \sigma'(z)$: the derivative is even, and an even function’s extremum on $\R$ sits at $0$ unless it is bimodal. Consistent.

The logarithmic estimate agrees with the loop. The closed form gave $L > 4.12$ and the loop counted $5$; $L > 11.999$ and the loop counted $12$. Two methods, same answer.

The magnitudes bracket correctly. $2.955\times10^{-18} < 9.537\times10^{-7}$, since $0.0177 < 0.25$ and both are raised to the same power. A product of smaller numbers must be smaller.

STATEMENT

Construct explicit weights and a bias for which a ReLU unit’s gradient is exactly zero for every input in the dataset, forever. Then show LeakyReLU repairs it, and quantify how long the repair takes.

GIVEN

A ReLU unit $a = \max(0, \langle\vec{w},\vec{x}\rangle + b)$ with inputs drawn from $[0,1]^2$. Downstream the unit feeds a loss $\loss$ with $\partial \loss/\partial a$ of order $1$.

FIND

A $(\vec{w}, b)$ making the unit permanently dead; a proof that no update revives it; and the number of steps LeakyReLU needs to recover, at a stated learning rate.

STRATEGY

Make the pre-activation negative on the whole input domain, then trace the gradient backwards and observe that it is zero at the first multiplication.

SOLUTION

Step 1 — the construction. Take

$\vec{w} = (1, 1), \qquad b = -10$

For any $\vec{x} \in [0,1]^2$ the pre-activation is

$z = x_1 + x_2 - 10 \in [-10, -8]$

always negative, with a margin of at least $8$ from the kink.

Step 2 — the output is always zero. $a = \max(0, z) = 0$ for every input in the domain. The unit contributes nothing to any prediction.

Step 3 — the gradient is always zero. By the chain rule, the gradient reaching the weights is

and identically $\partial\loss/\partial b = 0$. Whatever the downstream error, it is multiplied by zero at this unit.

Step 4 — the death is permanent. Any gradient-based update has the form $\vec{w} \leftarrow \vec{w} - \eta\,\partial\loss/\partial\vec{w}$. Since the gradient is $\vec{0}$, the update is $\vec{w} \leftarrow \vec{w}$. The parameters cannot move, so $z$ stays in $[-10,-8]$, so the gradient stays zero. The state is a fixed point of training. No learning rate, no schedule, no optimiser and no amount of data changes it. Momentum does not help either: momentum accumulates past gradients, and every past gradient was also zero.

The only escapes are external — weight decay pulling $b$ toward zero, or a different unit’s weights changing the input distribution — and neither is a property of this unit’s own learning.

Step 5 — LeakyReLU repairs it. With $\varphi(z) = z$ for $z>0$ and $0.01z$ otherwise, the derivative on the negative side is $0.01$ rather than $0$. Now

$\frac{\partial \loss}{\partial b} = \frac{\partial \loss}{\partial a}\cdot 0.01$

Small, but not zero — and a small nonzero gradient applied repeatedly moves.

Step 6 — how long. Suppose $\partial\loss/\partial a = -1$ on average (the loss wants this unit’s output larger) and $\eta = 0.01$. Each step moves the bias by

$\Delta b = -\eta\,\frac{\partial\loss}{\partial b} = -(0.01)(-1)(0.01) = 10^{-4}$

To raise $b$ from $-10$ to $-1$, where the unit begins firing on part of the domain, needs

$\frac{9}{10^{-4}} = 90{,}000 \text{ steps}$

The honest reading. LeakyReLU converts impossible into slow. Ninety thousand steps is a real cost, and it is why the slope is often set to $0.1$ rather than $0.01$ — which cuts the recovery to $9{,}000$ steps — and why careful initialisation, which prevents the death in the first place, matters more than the repair.

Check — sanity

The margin is comfortable. The largest possible pre-activation is $1 + 1 - 10 = -8$, not $-0.001$. So this is not a knife-edge construction that a single unusual input could disturb — the whole domain is $8$ units from the kink.

The zero is structural, not numerical. $\mathrm{ReLU}'(z) = 0$ exactly for $z < 0$, not approximately. There is no underflow here and no precision to recover: the arithmetic is exact and the answer is exactly zero.

The LeakyReLU arithmetic is dimensionally right. $\Delta b$ is $\eta \times$ (downstream gradient) $\times$ (slope) $= 0.01 \times 1 \times 0.01$, three dimensionless factors, giving $10^{-4}$ per step in the units of $b$. And $9 / 10^{-4} = 9\times10^{4}$. ✓

STATEMENT

Prove that a network of any depth whose activations are all the identity — or any linear map — computes a single affine function. Then state exactly what the theorem does and does not rule out.

GIVEN

An $L$-layer network with $\vec{h}^{(0)} = \vec{x}$ and $\vec{h}^{(\ell)} = \vec{h}^{(\ell-1)}\mat{W}^{(\ell)} + \vec{b}^{(\ell)}$, row-major throughout, with no nonlinearity between layers.

FIND

A single $(\mat{W}, \vec{b})$ with $\vec{h}^{(L)} = \vec{x}\mat{W} + \vec{b}$ for all $\vec{x}$, together with explicit formulas for both.

STRATEGY

Induct on depth. The base case is one layer; the inductive step substitutes the hypothesis into the next layer and collects terms. The only algebra needed is that matrix multiplication distributes over addition.

SOLUTION

Base case, $L = 1$. $\vec{h}^{(1)} = \vec{x}\mat{W}^{(1)} + \vec{b}^{(1)}$, which is already of the required form with $\mat{W} = \mat{W}^{(1)}$ and $\vec{b} = \vec{b}^{(1)}$.

Inductive hypothesis. Suppose after $\ell$ layers

$\vec{h}^{(\ell)} = \vec{x}\,\mat{A}_{\ell} + \vec{c}_{\ell}$

for some $\mat{A}_\ell$ and $\vec{c}_\ell$ not depending on $\vec{x}$.

Inductive step. Apply layer $\ell+1$ and expand, using distributivity:

which is again of the required form. By induction it holds for all $L$, with

$\blacksquare$ (I.3.3)

The same result for any linear activation. If $\varphi(z) = \alpha z$ then applying it elementwise is multiplication by $\alpha\mat{I}$, which is itself a linear map and can be absorbed into the neighbouring $\mat{W}$. The proof is unchanged with $\mat{W}^{(\ell)}$ replaced by $\alpha\mat{W}^{(\ell)}$.

Even an affine activation collapses. $\varphi(z) = \alpha z + \beta$ adds a constant row, which merges into the bias. So no affine activation escapes.

What is ruled out. The hypothesis class of the deep linear network equals that of a single layer of the same input and output widths. Nothing reachable by the deep one is unreachable by the shallow one. Depth has bought zero expressiveness.

What is not ruled out — and this is the part usually stated wrongly.

The parameter counts differ. The deep network has $\sum d_\ell d_{\ell+1}$ parameters against the shallow one’s $d_0 d_L$. It is over-parameterised for what it computes, and the map from parameters to functions is many-to-one.

The rank can be lower. If some intermediate width $d_\ell$ is smaller than both $d_0$ and $d_L$, the product $\mat{W}$ has rank at most $d_\ell$ (Rank, eigenvalues and the singular value decomposition · 0.LA.04). So a deep linear network with a bottleneck is more restricted than one layer, not equal to it. This is the linear autoencoder of Chapter I.12, and its restriction is the whole point.

The optimisation differs entirely. The loss surface of the deep linear network is non-convex in its parameters even though the function class is linear. It has saddle points the single layer does not, gradient descent on it has a different implicit bias, and it converges at a different rate. Studying deep linear networks is a live research programme precisely because the trajectory is interesting while the destination is not.

Check — sanity

Shapes conform. $\mat{W}^{(1)}$ is $d_0\times d_1$, …, $\mat{W}^{(L)}$ is $d_{L-1}\times d_L$, so the product is $d_0\times d_L$ — exactly the shape a single layer from input to output would have.

A two-layer instance checks by hand. With $\mat{W}^{(1)} = \begin{bmatrix}1&2\\0&1\end{bmatrix}$, $\vec{b}^{(1)} = (1,0)$, $\mat{W}^{(2)} = \begin{bmatrix}1&0\\1&1\end{bmatrix}$, $\vec{b}^{(2)} = (0,1)$ and $\vec{x} = (1,1)$: layer by layer, $\vec{h}^{(1)} = (2,3)$ then $\vec{h}^{(2)} = (5,4)$. By the formula, $\mat{W} = \begin{bmatrix}3&2\\1&1\end{bmatrix}$ and $\vec{b} = (1,0)\mat{W}^{(2)} + (0,1) = (1,1)$, so $\vec{x}\mat{W} + \vec{b} = (4,3) + (1,1) = (5,4)$. ✓

It is consistent with I.1.B01. There, removing the ReLU changed $\hat{y}$ from $2$ to $1$. If the collapse theorem said the nonlinearity did nothing, that check would have had to give the same number. It did not, which is the same fact from the other side.

STATEMENT

Derive $\mathrm{GELU}'$, evaluate it at seven points, and compare its behaviour at the origin with ReLU’s. Then state the optimisation consequence of the difference and find where the derivative is most negative.

GIVEN

$\mathrm{GELU}(z) = z\,\Phi(z)$ with $\Phi$ the standard normal CDF and $\Phi' = \phi$, the standard normal density $\phi(z) = e^{-z^2/2}/\sqrt{2\pi}$.

FIND

A closed form for $\mathrm{GELU}'$; its values at $z \in \{-3,-1,-0.5,0,0.5,1,3\}$; the value at $z=0$ against ReLU’s; and $\min_z \mathrm{GELU}'(z)$.

STRATEGY

Product rule, then evaluate. The interesting content is entirely in what the second term does for negative $z$.

SOLUTION

Step 1 — differentiate. By the product rule on $z \cdot \Phi(z)$:

\mathrm{GELU}'(z) = \Phi(z) + z\,\phi(z) \tag{I.3.4}

Two terms with different characters. $\Phi(z) \in (0,1)$ is a soft gate rising from $0$ to $1$. The correction $z\phi(z)$ is positive for $z>0$, negative for $z<0$, and decays to zero in both tails because the Gaussian density does.

Step 2 — evaluate.

Step 3 — the origin. $\mathrm{GELU}'(0) = \Phi(0) + 0 = 0.5$, and it is a genuine two-sided derivative: GELU is $C^\infty$ everywhere, being a product of smooth functions.

ReLU has no derivative at $0$ at all. Its left slope is $0$, its right slope is $1$, and every framework silently picks a subgradient from $[0,1]$ — usually $0$, sometimes $0.5$, sometimes $1$. GELU’s $0.5$ is not a convention: it is the value.

Step 4 — the minimum. Setting $\mathrm{GELU}'' = 0$ gives $2\phi(z) + z\phi'(z) = \phi(z)(2 - z^2) = 0$, so $z = -\sqrt2 = -1.4142$ (taking the negative root). There

$\mathrm{GELU}'(-\sqrt2) = \Phi(-1.4142) + (-1.4142)\phi(-1.4142) = 0.0786 - 0.2075 = -0.1289$

The derivative goes negative. No activation studied so far does this. Between roughly $z = -\infty$ and $z \approx -0.75$, increasing the pre-activation decreases the output.

Step 5 — the optimisation consequences. Three, in order of how often they are stated correctly.

No dead region. $\mathrm{GELU}'(-10) = \Phi(-10) + (-10)\phi(-10)$, which is about $-7.6\times10^{-23}$ — vanishingly small but nonzero. So the dead-unit failure of I.3.B04 cannot occur exactly, though at $10^{-23}$ the distinction is academic and fp16 will round it to zero regardless.

No kink, so second-order methods are well defined. Anything relying on curvature — natural gradient, K-FAC, or simply a smooth loss landscape — is better behaved when the activation is twice differentiable. ReLU’s Hessian is a sum of Dirac deltas at the kinks.

The negative lobe is a real functional difference. It lets a unit express “this input is moderately against my feature” with a small negative output rather than with silence. Whether that helps is empirical, and the honest answer is that GELU’s advantage over ReLU in transformers is consistent but small, and no convincing theoretical account of it exists.

Check — sanity

The tails match ReLU. At $z = 3$, $\mathrm{GELU}' = 1.0119 \approx 1$; at $z=-3$, $-0.0119 \approx 0$. GELU is asymptotically ReLU in both directions, as its construction intends.

The derivative integrates back to the function. Between $z=0$ and $z=0.5$ the mean of $\mathrm{GELU}'$ is roughly $(0.5+0.8675)/2 = 0.684$, so the predicted rise is $0.684 \times 0.5 = 0.342$ against the actual $0.3457 - 0 = 0.3457$. Agreement to two digits from a two-point trapezoid, which is as much as that method deserves.

The minimum is where the second derivative says. $2 - z^2 = 0$ at $z = \pm\sqrt2$, and the numerical scan over $3{,}001$ points found $-0.1289$, consistent with the closed-form evaluation at $-\sqrt2$.

STATEMENT

For a transformer-scale layer, compute the memory one activation tensor occupies and the arithmetic each activation function costs. Then compare both with the matrix multiply they sit next to, and draw the conclusion.

GIVEN

Batch $B = 8$, sequence $T = 2048$, width $d = 4096$, depth $L = 32$. Elementwise costs, approximately: ReLU $1$ FLOP (a compare), LeakyReLU $2$, sigmoid $4$ (exp, add, divide), tanh $6$, GELU with erf about $12$, GELU with the tanh approximation about $8$.

FIND

The element count and memory of one activation tensor in bf16 and fp32; the total across $L$ layers; the GFLOPs each activation costs per layer; and the ratio to one $d\times d$ matrix multiply.

STRATEGY

Count elements once and reuse. Memory scales with the count; arithmetic scales with the count times a small constant; the matmul scales with the count times $d$ — and that last factor is the whole answer.

SOLUTION

Step 1 — the element count.

$B \times T \times d = 8 \times 2048 \times 4096 = 67{,}108{,}864 \approx 6.71\times10^{7}$

Step 2 — memory. At $2$ bytes per element in bf16:

$6.71\times10^{7} \times 2 = 1.342\times10^{8}\ \text{bytes} = 134.2\ \text{MB}$

and in fp32, $268.4$ MB. Across $32$ layers, storing one activation tensor per layer for the backward pass:

$32 \times 134.2\ \text{MB} = 4.29\ \text{GB (bf16)}, \qquad 8.59\ \text{GB (fp32)}$

Step 3 — arithmetic. Each function is $c$ FLOPs per element:

Step 4 — the comparison that settles it. One $d\times d$ matrix multiply on the same tensor costs $2 \times (BTd) \times d$ FLOPs:

$2 \times 6.71\times10^{7} \times 4096 = 5.50\times10^{11} = 549.8\ \text{GFLOP}$

against the costliest activation’s $0.81$ GFLOP — a ratio of about $\mathbf{680}$.

The conclusion, in two halves.

Arithmetically, activations are free. Even GELU with erf is under $0.2\%$ of the matmul it follows. Choosing ReLU over GELU to save compute is optimising the wrong term by a factor of several hundred. Any argument for ReLU on grounds of speed is, at this scale, wrong.

In memory they are not free at all. $4.29$ GB of stored activations is a substantial fraction of an $80$ GB accelerator, and it is why activation checkpointing exists (II.8.B03): recompute the activation in the backward pass rather than store it, trading that $0.81$ GFLOP — which is free — against $134$ MB per layer, which is not.

That asymmetry is the general shape of the thing. Elementwise operations are bounded by memory bandwidth, matrix operations by arithmetic. An activation reads $BTd$ values and writes $BTd$ values to do $c \cdot BTd$ FLOPs, giving an arithmetic intensity of about $c/4$ FLOPs per byte in bf16 — for ReLU that is $0.25$, against roughly $200$ for the matmul. Chapter VIII.6 turns this into a roofline argument; here it is enough to notice that the two operations live in different worlds.

Check — sanity

Memory scales with elements, not with the function. ReLU and GELU store identical tensors: $134.2$ MB either way. Only the arithmetic column varies. If memory had varied by activation, an element would have been double-counted.

bf16 is exactly half of fp32. $134.2 \times 2 = 268.4$. Two bytes against four.

The matmul ratio is $d/2$ times the activation constant. Matmul is $2 \cdot BTd \cdot d$ and the activation is $c \cdot BTd$, so the ratio is $2d/c = 8192/12 = 683$ for GELU-erf. The snippet printed $682$, differing by integer division. Consistent.