Without a nonlinearity a stack of affine maps is a single affine map, so depth buys nothing at all.
The nonlinearity is what makes depth worth having.
How this chapter is built
M3Load-bearing
The content is mathematics. Understanding is demonstrated by computation, not recall.
basics11/11what the words mean
concept2/2what to picture
theory4/4why it works, and when it does not
mathematics15/15derive it, then compute it
practice9/9build it, break it, read the papers
Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.
The reader has one perceptron and has just been shown it cannot compute XOR. The obvious repair is to stack perceptrons. This chapter shows that stacking alone achieves exactly nothing — a composition of affine maps is affine — and that the entire benefit of depth comes from what is placed between the layers.
Fig. 3 — Two linear layers composed give one line whatever the depth. The same two layers with a rectifier between them give a piecewise curve, one fold per unit.
What this chapter covers
Activation
Nonlinearity
Saturation
Sigmoid and tanh
ReLU and LeakyReLU
GELU
Dead units
Universal approximation
Apparatus
The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.
An activation function φ : ℝ → ℝ is applied elementwise to a pre-activation vector. Being elementwise is the defining property: it introduces curvature without introducing any mixing between coordinates, which is what keeps it cheap and what makes its Jacobian diagonal.
σThe logistic function, or a standard deviation
Definition 2Pre-activation
The pre-activation z is the output of the affine part of a layer, before φ is applied. The distinction matters because gradients are usually derived with respect to z and only then pushed back to the weights.
Definition 3Saturation
An activation saturates at an input z when φ′(z) ≈ 0, so that changing z barely changes the output. A saturated unit passes almost no gradient backwards, whatever the error downstream.
Definition 4Sigmoid
σ(z) = 1/(1 + e⁻ᶻ), mapping ℝ onto (0, 1) monotonically. Its derivative σ′ = σ(1 − σ) never exceeds 1/4, attained at z = 0.
Definition 5Hyperbolic tangent
tanh(z) = (eᶻ − e⁻ᶻ)/(eᶻ + e⁻ᶻ), mapping ℝ onto (−1, 1). It is a rescaled sigmoid: tanh(z) = 2σ(2z) − 1, and its derivative 1 − tanh²(z) reaches 1 at the origin.
Definition 6Rectified linear unit
ReLU(z) = max(0, z). It is piecewise linear with derivative 1 for z > 0 and 0 for z < 0, and no derivative at z = 0 — where frameworks silently choose a subgradient, usually 0.
Definition 7Dead unit
A unit is dead when its pre-activation is negative for every input in the dataset, so ReLU outputs zero always and its incoming weights receive zero gradient always. A dead unit is permanently dead: nothing in training can revive it.
Definition 8Gaussian error linear unit
GELU(z) = z·Φ(z), where Φ is the standard normal CDF. It is smooth everywhere, is not monotonic — it dips below zero for moderately negative z — and approaches ReLU as |z| grows.
Formal results
I.3.T1
Collapse of linear stacks
For any depth L, a network whose activations are all identity or any linear map computes x ↦ xW + b for a single W and b. The composition of affine maps is affine, so such a network has exactly the representational power of one layer, whatever its parameter count.
What this does not promise
It does not say the deep linear network is useless to study: its *optimisation* differs sharply from the single layer's, its loss surface has saddles the shallow one lacks, and the implicit bias of gradient descent on it is an active research topic. The theorem is about the set of functions reachable, not about the trajectory taken to reach one.
A network with one hidden layer and any non-polynomial activation can approximate any continuous function on a compact set to arbitrary accuracy, given enough hidden units.
What this does not promise
It gives no bound on the number of units, which may be exponential in the input dimension. It says nothing about whether gradient descent finds the approximating parameters — existence is not reachability. It says nothing about behaviour off the compact set. And it is not a statement about generalisation: the approximating network fits the target on that set and may be arbitrarily wrong on data drawn near it.
Assumptions
Each is paired with the problem that shows what its removal costs. An assumption nobody tests is a disclaimer.
The activation is applied elementwise, so its Jacobian is diagonal and depth does not multiply full matrices of derivatives.I.3.X08
The activation is non-polynomial, which is what universal approximation requires.I.3.X07
Sigmoid saturation kills depth. At |z| = 4 the derivative is 0.0177, so a product over ten layers is 2.96e-18 — already below fp16's smallest subnormal, 5.96e-8, by ten orders of magnitude. In fp16 a sigmoid stack underflows to exactly zero at depth 5 even in the best case at z = 0, where the derivative is 0.25.
A ReLU unit can die permanently. Give a unit weights (1, 1) and bias -10 with all inputs in [0, 1]: the pre-activation never exceeds -8, the output is always zero, and the gradient reaching its weights is always zero. No learning rate, schedule or restart recovers it, because the gradient that would move it is itself zero.