Chapter 1 · I.1

Problems

A chapter that uses mathematics has to teach that mathematics by making the reader compute. A chapter that only displays equations has failed, however correct the equations are.

M2Substantive

4/3 problems4/3 variants6/6 exercisesquota met, and enforced

The contract

3 worked problems, minimum.
3 distinct variants, and no variant more than half of them.
6 exercises, every one with a published solution.

Numerical instantiationDimensional algebraDifferentiationProbabilistic

Problem I.1.B01

A forward pass, every intermediate written down

numeric▲△△

All values are exact here; they are printed to 4 d.p. so that they line up with the reproduction snippet.

STATEMENT

A network with two inputs, two hidden units and one output is fully specified below. Carry out the forward computation by hand and report every intermediate quantity, not only the final number.

GIVEN

The input, as a row vector:

$\vec{x} = \begin{bmatrix} 1 & 2 \end{bmatrix} \in \R^{1\times 2}$

The first layer, with $\mat{W}^{(1)} \in \R^{2\times 2}$ and a bias row $\vec{b}^{(1)} \in \R^{1\times 2}$ :

\mat{W}^{(1)} = \begin{bmatrix} 1 & 0 \\ -1 & 2 \end{bmatrix}, \qquad \vec{b}^{(1)} = \begin{bmatrix} 0.5 & -1 \end{bmatrix}

The hidden nonlinearity is $\mathrm{ReLU}(z) = \max(0, z)$ , applied to each entry separately. The second layer, with $\mat{W}^{(2)} \in \R^{2\times 1}$ and $b^{(2)} \in \R$ :

\mat{W}^{(2)} = \begin{bmatrix} 2 \\ 1 \end{bmatrix}, \qquad b^{(2)} = -1

The output layer has no nonlinearity.

FIND

The pre-activations $\vec{z}^{(1)}$ (a $1\times 2$ row), the activations $\vec{a}^{(1)}$ (a $1\times 2$ row), and the output $\hat{y}$ (a scalar).

STRATEGY

Follow the definition one operation at a time and name each result, because a wrong final number is only useful if you can say which step produced it.

SOLUTION

Step 0 — fix the convention. Elementa is row-major (Vectors, matrices and the row-major convention 0.LA.01): a row of $\mat{X}$ is one example, and a layer is applied on the right as $\vec{x}\mat{W} + \vec{b}$ . The shapes must meet as $(1\times 2)(2\times 2) \to (1\times 2)$ , and they do.

Getting this wrong is the single most common error in a first hand-computation. If you write $\mat{W}\vec{x}$ instead, you are computing with $\mat{W}^{\mathsf T}$ and every number below changes.

Step 1 — the first pre-activation. Each entry of $\vec{z}^{(1)}$ is one inner product of $\vec{x}$ with one column of $\mat{W}^{(1)}$ , plus the matching bias.

Column 1 of $\mat{W}^{(1)}$ is $(1, -1)$ :

$z^{(1)}_1 = (1)(1) + (2)(-1) + 0.5 = 1 - 2 + 0.5 = -0.5$

Column 2 is $(0, 2)$ :

$z^{(1)}_2 = (1)(0) + (2)(2) + (-1) = 0 + 4 - 1 = 3$

$\vec{z}^{(1)} = \begin{bmatrix} -0.5 & 3 \end{bmatrix}$

Step 2 — the nonlinearity. ReLU acts on each entry independently. There is no mixing here at all; that is what “elementwise” means.

a^{(1)}_1 = \max(0, -0.5) = 0, \qquad a^{(1)}_2 = \max(0, 3) = 3

$\vec{a}^{(1)} = \begin{bmatrix} 0 & 3 \end{bmatrix}$

The first hidden unit has been switched off. It contributes nothing to the output, and — as Chapter I.6 will show — it also receives no gradient on this example.

Step 3 — the output layer. Shapes $(1\times 2)(2\times 1) \to (1\times 1)$ :

$\hat{y} = (0)(2) + (3)(1) + (-1) = 0 + 3 - 1 = 2$

Answer

\vec{z}^{(1)} = \begin{bmatrix} -0.5 & 3 \end{bmatrix}, \qquad \vec{a}^{(1)} = \begin{bmatrix} 0 & 3 \end{bmatrix}, \qquad \hat{y} = 2

$\vec{z}^{(1)}$ and $\vec{a}^{(1)}$ are $1\times 2$ ; $\hat{y}$ is a scalar and carries whatever units the target carries.

Check — numeric · i-1-b01-forward-pass.py

def relu(v):        return [max(0.0, t) for t in v]
def matvec(x, W, b): return [sum(x[i] * W[i][j] for i in range(len(x))) + b[j]
                             for j in range(len(b))]
z1 = matvec(x, W1, b1); a1 = relu(z1); z2 = matvec(a1, W2, b2)

Prints z1 = -0.5000 3.0000, a1 = 0.0000 3.0000, yhat = 2.0000.

Executed in CI. The digits above are the digits it printed.

Check — sanity

Three independent reasons the answer is right.

The shapes conform at every step. $(1\times2)(2\times2)\to(1\times2)$ , then elementwise, then $(1\times2)(2\times1)\to(1\times1)$ . A forward pass whose shapes conform is not necessarily correct, but one whose shapes do not conform is certainly wrong, and it costs nothing to check.

The dead unit is consistent. $a^{(1)}_1 = 0$ , so the first row of $\mat{W}^{(2)}$ — the value $2$ — cannot influence $\hat{y}$ . Change it to $200$ and recompute: still $\hat{y}=2$ . That is a real test, and it passes.

Removing the nonlinearity changes the answer. Without ReLU, $\hat{y} = (-0.5)(2) + (3)(1) - 1 = 1$ . Since $1 \neq 2$ , the nonlinearity is doing something on this input — which is the whole content of Chapter I.3.

Where this breaks

The answer depends on ReLU being applied before the second layer. Swap the order — apply the second layer to $\vec{z}^{(1)}$ and take ReLU afterwards — and $\hat{y}$ becomes $\max(0, 1) = 1$ . Two networks with identical weights and identical shapes, differing only in the order of two operations, give different outputs. The shape check cannot catch this; only reading the definition can.

Variation

Recompute with $\vec{x} = \begin{bmatrix} 2 & 1 \end{bmatrix}$ . Before doing any arithmetic, predict which hidden unit switches off — and then check whether your prediction was right and why.

Problem I.1.B02

One conformability error in five stages

shape▲△△

Shapes only; no rounding applies.

STATEMENT

A five-stage pipeline is specified below. Exactly one stage cannot be computed. Find it, say why, and state the shape the offending tensor must have for the pipeline to run.

GIVEN

A batch of $B = 32$ examples, each a flattened $28\times 28$ image:

Stage	Operation	Declared shape
0	input $\mat{X}$	$32 \times 784$
1	$\mat{X}\mat{W}^{(1)}$	$\mat{W}^{(1)}: 784 \times 256$
2	$\cdot\;\mat{W}^{(2)}$	$\mat{W}^{(2)}: 256 \times 128$
3	$\cdot\;\mat{W}^{(3)}$	$\mat{W}^{(3)}: 256 \times 64$
4	$\cdot\;\mat{W}^{(4)}$	$\mat{W}^{(4)}: 64 \times 10$

Elementwise nonlinearities sit between the stages. They do not change shape, so they can be ignored for this question — which is itself worth noticing.

FIND

The failing stage, the reason, and the corrected shape of the offending weight matrix. Then the total parameter count once it is corrected, counting weights only.

STRATEGY

Propagate the shape forward one stage at a time. A matrix product needs the inner dimensions to agree, so at each stage compare the running width with the first dimension of the next weight matrix; the first disagreement is the error.

SOLUTION

Step 1 — propagate. Write the running shape after each stage. The rule is $(a \times b)(b \times c) \to (a \times c)$ : the inner pair must match and then vanishes, while the outer pair survives.

\begin{aligned} \text{after 0:}\quad & 32 \times 784 \\ \text{after 1:}\quad & (32\times784)(784\times256) \to 32 \times 256 \\ \text{after 2:}\quad & (32\times256)(256\times128) \to 32 \times 128 \\ \text{after 3:}\quad & (32\times \mathbf{128})(\mathbf{256}\times64) \to \text{undefined} \end{aligned}

Step 2 — name the failure. At stage 3 the running width is $128$ , because stage 2 projected down to $128$ . But $\mat{W}^{(3)}$ expects an input of width $256$ . The inner dimensions are $128$ and $256$ ; they disagree, so the product is undefined.

Note the batch axis played no part. It is carried along untouched by every stage, which is why a shape error in a stack of dense layers is always a statement about widths and never about how many examples you fed in.

Step 3 — correct it. A weight matrix’s first dimension is the width it consumes and its second is the width it produces. The width consumed must be $128$ ; the width produced was intended to be $64$ and nothing contradicts that. So

$\mat{W}^{(3)} : 128 \times 64$

and the pipeline then runs $32\times784 \to 256 \to 128 \to 64 \to 10$ .

Step 4 — count the parameters. Weights only, so the sum of the products:

\begin{aligned} 784 \times 256 &= 200{,}704 \\ 256 \times 128 &= 32{,}768 \\ 128 \times 64 &= 8{,}192 \\ 64 \times 10 &= 640 \\ \hline \text{total} &= 242{,}304 \end{aligned}

Answer

Stage 3 fails: the running width is $128$ but $\mat{W}^{(3)}$ declares an input width of $256$ . The corrected shape is $\mat{W}^{(3)} : 128 \times 64$ , and the pipeline then holds $\mathbf{242{,}304}$ weights.

Adding biases of widths $256, 128, 64, 10$ would contribute a further $458$ , for $242{,}762$ parameters in total.

Check — sanity

The first layer dominates, as it must. $200{,}704$ of $242{,}304$ weights — $82.8\%$ — sit in the first matrix, because it is the only one touching the $784$ -dimensional input. Any parameter count for a network with a wide input and a narrowing stack should be dominated by its first layer; if yours is not, recheck.

The corrected chain telescopes. Reading the widths in order gives $784 \to 256 \to 128 \to 64 \to 10$ , where each weight matrix’s second dimension is the next one’s first. That chain property is the shape rule restated, and it is the fastest way to eyeball a whole architecture.

Where this breaks

The diagnosis assumes the declared stage-2 output is correct and stage 3 is at fault. Nothing in the specification proves that. If the author intended a $256$ -wide trunk throughout, the error is in $\mat{W}^{(2)}$ , which should have been $256\times256$ , and the corrected parameter count is different. A shape error localises a contradiction; it does not tell you which side to change. That judgement needs the architecture’s intent, which lives outside the shapes.

Variation

Insert a skip connection adding the stage-1 output to the stage-3 output. State the new constraint this imposes on the widths, and say which of the two repairs above it rules out.

Problem I.1.B03

The squared-loss gradient, and the optimum it points at

gradient▲▲△

All values rounded to 4 d.p.

STATEMENT

Derive the gradient of the squared loss for one-dimensional linear regression through the origin, evaluate it at $w = 1$ on three data points, and confirm that the parameter at which the gradient vanishes is the one the normal equation gives.

GIVEN

The model $\hat{y} = wx$ , with a single scalar parameter $w$ . The loss

$\loss(w) = \frac{1}{2n}\sum_{i=1}^{n}\left(w x_i - y_i\right)^2$

and the three points

$(x_1,y_1) = (1, 2), \quad (x_2,y_2) = (2, 4), \quad (x_3,y_3) = (3, 5)$

FIND

A closed form for $\mathrm{d}\loss/\mathrm{d}w$ ; its value at $w = 1$ ; the minimiser $w^\star$ ; and a demonstration that the two agree.

STRATEGY

Differentiate under the sum, then set the result to zero. Both steps are legal here because the sum is finite and each term is differentiable everywhere — worth saying once, because for the absolute-error loss the second step is not available.

SOLUTION

Step 1 — differentiate one term. Write $r_i = w x_i - y_i$ for the $i$ -th residual. By the chain rule (The chain rule 0.MC.03), with $\partial r_i / \partial w = x_i$ :

$\frac{\mathrm{d}}{\mathrm{d}w}\,\tfrac12 r_i^2 = r_i \cdot \frac{\mathrm{d} r_i}{\mathrm{d}w} = x_i\,(w x_i - y_i)$

The factor $\tfrac12$ in the loss exists precisely to cancel the $2$ that differentiating a square produces. It is a convenience, not a modelling choice, and it changes no minimiser.

Step 2 — sum. Differentiation is linear, so the derivative of the average is the average of the derivatives:

$\frac{\mathrm{d}\loss}{\mathrm{d}w} = \frac{1}{n}\sum_{i=1}^{n} x_i\,(w x_i - y_i) \tag{I.1.5}$

Read what this says: each point pulls on $w$ in proportion to its own $x_i$ and to how wrong the prediction currently is. A point with $x_i = 0$ exerts no pull at all, whatever its target.

Step 3 — evaluate at $w = 1$ . The three residuals are

r_1 = (1)(1) - 2 = -1, \quad r_2 = (1)(2) - 4 = -2, \quad r_3 = (1)(3) - 5 = -2

\frac{\mathrm{d}\loss}{\mathrm{d}w}\bigg|_{w=1} = \frac{1}{3}\Big[(1)(-1) + (2)(-2) + (3)(-2)\Big] = \frac{-1 - 4 - 6}{3} = \frac{-11}{3} = -3.6667

Negative, so increasing $w$ decreases the loss. That matches the residuals: every prediction is below its target.

Step 4 — set the gradient to zero. Because the loss is a quadratic in a single variable with a positive leading coefficient, its unique stationary point is its minimum.

\frac{1}{n}\sum_i x_i(w x_i - y_i) = 0 \;\Longleftrightarrow\; w\sum_i x_i^2 = \sum_i x_i y_i \;\Longleftrightarrow\; w^\star = \frac{\sum_i x_i y_i}{\sum_i x_i^2}

This is the normal equation for the through-origin case. It is not a separate result to be looked up: it is what step 2 becomes when set to zero.

Step 5 — evaluate.

\sum_i x_i y_i = (1)(2) + (2)(4) + (3)(5) = 2 + 8 + 15 = 25

\sum_i x_i^2 = 1 + 4 + 9 = 14

$w^\star = \frac{25}{14} = 1.7857$

Answer

\frac{\mathrm{d}\loss}{\mathrm{d}w} = \frac{1}{n}\sum_i x_i (w x_i - y_i), \qquad \frac{\mathrm{d}\loss}{\mathrm{d}w}\bigg|_{w=1} = -\frac{11}{3} = -3.6667, \qquad w^\star = \frac{25}{14} = 1.7857

The gradient is a scalar in units of loss per unit $w$ ; $w^\star$ is dimensionless here because $x$ and $y$ share units.

Check — numeric · i-1-b03-linear-regression-gradient.py

def grad(w): return sum(x * (w * x - y) for x, y in data) / len(data)
w_star = sum(x * y for x, y in data) / sum(x * x for x, y in data)

Prints grad at w=1 -3.6667, w* = Sxy/Sxx 1.7857, and grad at w* 0.0000.

Executed in CI. The digits above are the digits it printed.

Check — sanity

The gradient vanishes at the claimed optimum. Substituting $w^\star$ into (I.1.5) gives $0.0000$ to four decimal places — which is the definition of the optimum, computed independently of the closed form that produced it.

The loss falls. $\loss(1) = 1.5000$ and $\loss(w^\star) = 0.0595$ . If the “optimum” had a higher loss than an arbitrary starting point, the sign of the derivation would be wrong somewhere.

The answer is bracketed by the per-point ratios. Each point alone would give $y_i/x_i$ : that is $2$ , $2$ and $1.667$ . The least-squares fit, $1.7857$ , lies inside $[1.667, 2]$ — as any weighted compromise must. A value outside that interval would mean an arithmetic slip.

Where this breaks

The step from “gradient is zero” to “this is the minimum” uses convexity, and convexity here comes from the model being linear in the parameter — not from the loss being squared. Keep the squared loss but make the model $\hat{y} = wx$ with $w$ replaced by $\sigma(v)$ for a new parameter $v$ , and the loss in $v$ is no longer convex: stationary points can be maxima or saddles. Every network after Chapter I.5 is in that second regime, and this is the last chapter in which “set the derivative to zero” is a solution method rather than a hope.

Variation

Redo the derivation for $\hat{y} = wx + b$ with two parameters. You will get two equations in two unknowns; solve them for the same three points, and say what the extra parameter bought in terms of the final loss.

Problem I.1.B04

Cross-entropy is the negative log-likelihood

probability▲▲△

All values rounded to 4 d.p. Logarithms are natural, so losses are in nats.

STATEMENT

Show that binary cross-entropy and the negative log-likelihood of a Bernoulli model are the same quantity, then evaluate both at $p = 0.8$ for each possible label.

GIVEN

A model that outputs a single number $p \in (0,1)$ , interpreted as $P(Y = 1 \mid x)$ . The label $y \in \{0, 1\}$ . Binary cross-entropy is defined as

$\mathrm{CE}(p, y) = -\big[\, y \log p + (1-y)\log(1-p) \,\big]$

and the Bernoulli likelihood of the observed label is $P(Y = y) = p^{y}(1-p)^{1-y}$ .

FIND

An algebraic demonstration that $\mathrm{CE}(p,y) = -\log P(Y = y)$ , and the value of each at $(p, y) = (0.8, 1)$ and $(0.8, 0)$ .

STRATEGY

Take the logarithm of the likelihood and let the exponents come down. The two expressions are then identical term for term, so this is an identity rather than an approximation — which matters, because it means choosing cross-entropy is choosing a Bernoulli noise model, whether or not anyone said so.

SOLUTION

Step 1 — write the likelihood. The trick is the indicator exponent: since $y$ is $0$ or $1$ , exactly one of the two factors survives.

$P(Y = y) = p^{y}(1-p)^{1-y}$

Check both cases before going further. At $y=1$ : $p^1(1-p)^0 = p$ . At $y=0$ : $p^0(1-p)^1 = 1-p$ . Correct in both.

Step 2 — take logarithms. Using $\log(ab) = \log a + \log b$ and $\log(a^c) = c\log a$ :

\log P(Y=y) = \log\!\big(p^{y}\big) + \log\!\big((1-p)^{1-y}\big) = y\log p + (1-y)\log(1-p)

Step 3 — negate.

$-\log P(Y = y) = -\big[\,y\log p + (1-y)\log(1-p)\,\big] = \mathrm{CE}(p,y) \tag{I.1.6}$

The two definitions coincide exactly, for every $p$ and every $y$ . Nothing was approximated and no assumption was added beyond the one already made when the output was called a probability.

Step 4 — evaluate at $y = 1$ . The $(1-y)$ term vanishes:

$\mathrm{CE}(0.8, 1) = -\log 0.8 = -(-0.2231) = 0.2231$

and independently, $-\log P(Y=1) = -\log 0.8 = 0.2231$ . They agree.

Step 5 — evaluate at $y = 0$ . Now the $y$ term vanishes:

$\mathrm{CE}(0.8, 0) = -\log(1 - 0.8) = -\log 0.2 = 1.6094$

and $-\log P(Y=0) = -\log 0.2 = 1.6094$ .

The asymmetry is the point. The same prediction $p = 0.8$ costs $0.2231$ when it is right and $1.6094$ when it is wrong — seven times as much. A model is punished far more for confident error than it is rewarded for confident correctness, and that asymmetry is what drives calibration.

Answer

$\mathrm{CE}(p, y) = -\log P(Y = y) \quad\text{identically, for all } p \in (0,1),\ y \in \{0,1\}$

\mathrm{CE}(0.8, 1) = 0.2231 \text{ nats}, \qquad \mathrm{CE}(0.8, 0) = 1.6094 \text{ nats}

In bits, divide by $\log 2$ : $0.3219$ and $2.3219$ bits respectively.

Check — numeric · i-1-b04-bernoulli-cross-entropy.py

def cross_entropy(p, y): return -(y * log(p) + (1 - y) * log(1 - p))
def likelihood(p, y):    return p if y == 1 else 1 - p
ce, nll = cross_entropy(p, y), -log(likelihood(p, y))

The two are computed from separate definitions and compared, rather than one being derived from the other. It prints equal=True at every tested pair.

Executed in CI. The digits above are the digits it printed.

Check — sanity

A perfect prediction costs nothing. At $p \to 1$ with $y = 1$ , $-\log p \to 0$ . The snippet’s $p = 0.99$ row gives $0.0101$ , which is close to zero and on the right side of it.

A maximally uncertain prediction costs $\log 2$ . At $p = 0.5$ the loss is $0.6931$ nats $= 1$ bit, for either label. One bit is exactly the information in a fair coin (Entropy 0.IT.01), so the units are behaving.

The two branches sum correctly. For any $p$ , the expected cross-entropy under the model’s own distribution is $p(-\log p) + (1-p)(-\log(1-p))$ , which is the entropy of that Bernoulli. At $p = 0.8$ : $0.8(0.2231) + 0.2(1.6094) = 0.5004$ nats, and the closed-form entropy of $\mathrm{Bern}(0.8)$ is $0.5004$ . That is a genuinely independent route to the same number.

Where this breaks

The identity needs $p \in (0,1)$ strictly. At $p = 0$ with $y = 1$ the loss is $-\log 0 = +\infty$ : a model that assigns zero probability to something that then happens is infinitely wrong, and no finite gradient step recovers from it. This is not a corner case in practice — it is why the loss is computed from logits with the log-sum-exp identity (Log-sum-exp 0.NU.02) rather than from a probability that a float can round to exactly $0$ or $1$ .

Variation

Extend to $K$ classes: show that categorical cross-entropy $-\sum_k y_k \log p_k$ is the negative log-likelihood of a categorical distribution, and evaluate for $\vec{p} = (0.7, 0.2, 0.1)$ with the true class being the second.

Exercises

Every one has a published solution. A hidden solution is a solution; a missing one is an abandonment.

I.1.X01The same network, a different inputnumeric▲△△

Using the network of I.1.B01 unchanged, compute the forward pass for $\vec{x} = \begin{bmatrix} 2 & 1 \end{bmatrix}$ . Before computing, predict how many hidden units will be switched off, and say what that prediction rests on.

Hint

The first hidden unit’s pre-activation is $x_1 - x_2 + 0.5$ . Ask what sign that takes when $x_1 > x_2$ .

Solution

Prediction. The first unit computes $x_1 - x_2 + 0.5$ , which is positive whenever $x_1 > x_2 - 0.5$ . Here $2 > 0.5$ , so it stays on. The second computes $2x_2 - 1$ , positive when $x_2 > 0.5$ ; here $x_2 = 1$ , so it also stays on. Prediction: no dead units, unlike I.1.B01 where the first died.

Step 1 — pre-activations.

z^{(1)}_1 = (2)(1) + (1)(-1) + 0.5 = 2 - 1 + 0.5 = 1.5

z^{(1)}_2 = (2)(0) + (1)(2) - 1 = 0 + 2 - 1 = 1

$\vec{z}^{(1)} = \begin{bmatrix} 1.5 & 1 \end{bmatrix}$

Step 2 — ReLU. Both entries are positive, so ReLU is the identity here:

$\vec{a}^{(1)} = \begin{bmatrix} 1.5 & 1 \end{bmatrix}$

Step 3 — output.

$\hat{y} = (1.5)(2) + (1)(1) - 1 = 3 + 1 - 1 = 3$

Answer. $\vec{a}^{(1)} = (1.5,\ 1)$ , $\hat{y} = 3$ , and no unit is dead — as predicted.

What this shows. The set of active units is a function of the input, not of the weights alone. The same network is, for each input, effectively a different linear map — the one obtained by deleting the dead units. A ReLU network is a piecewise-linear function, and which piece you are on is decided at the input. That observation is the whole of Proposition I.3.P01.

I.1.X02Where the parameters actually areshape▲△△

A fully connected network has widths $100 \to 50 \to 20 \to 5$ , with a bias on every layer. Count the weights, count the biases, and state what fraction of all parameters sits in the first layer. Then say what that fraction implies for where a parameter budget should be spent.

Hint

A layer from width $a$ to width $b$ holds $ab$ weights and $b$ biases.

Solution

Weights. One matrix per layer, of shape (input width $\times$ output width):

100 \times 50 = 5{,}000, \qquad 50 \times 20 = 1{,}000, \qquad 20 \times 5 = 100

$\text{total weights} = 6{,}100$

Biases. One per output unit, so $50 + 20 + 5 = 75$ .

Total. $6{,}100 + 75 = \mathbf{6{,}175}$ parameters.

First-layer share.

$\frac{5{,}000 + 50}{6{,}175} = \frac{5{,}050}{6{,}175} = 0.8178 = 81.78\%$

What it implies. Four fifths of the network is the single matrix that meets the input. Two consequences follow directly:

Widening the input is expensive and widening the tail is cheap. Going from $100$ to $200$ inputs adds $5{,}000$ parameters — as many as the whole first layer already had. Going from $5$ outputs to $10$ adds $105$ .

Parameter count is a poor proxy for depth. This network could be made twice as deep by inserting another $20 \to 20$ layer, adding $420$ parameters — under $7\%$ — while changing the function class substantially. Counting parameters tells you about memory; it tells you very little about capacity in the sense that matters for I.1.T2.

Biases are $1.2\%$ of the total here, which is why they are routinely omitted from back-of-envelope counts. That is a rounding decision, not a claim that they do nothing.

I.1.X03What the intercept buysgradient▲▲△

Repeat the derivation of I.1.B03 for the two-parameter model $\hat{y} = wx + b$ . Derive both partial derivatives, set them to zero, solve for $w$ and $b$ on the same three points $(1,2), (2,4), (3,5)$ , and compare the optimal loss with the through-origin fit.

Hint

Setting $\partial \loss/\partial b = 0$ first gives $b$ in terms of $w$ and the means. Substituting that back leaves one equation in $w$ .

Solution

Step 1 — the two partials. With $r_i = wx_i + b - y_i$ and $\loss = \frac{1}{2n}\sum r_i^2$ :

\frac{\partial \loss}{\partial w} = \frac1n \sum_i x_i r_i, \qquad \frac{\partial \loss}{\partial b} = \frac1n \sum_i r_i

The second is the first with $x_i$ replaced by $1$ — which is exactly right, since the bias is the weight on a constant input of $1$ . That is worth remembering: a bias is not a special kind of parameter, only a weight whose feature happens to be constant.

Step 2 — the normal equations. Setting both to zero:

w\sum x_i^2 + b\sum x_i = \sum x_i y_i, \qquad w\sum x_i + nb = \sum y_i

Step 3 — the sums.

n = 3, \quad \textstyle\sum x_i = 6, \quad \sum y_i = 11, \quad \sum x_i y_i = 25, \quad \sum x_i^2 = 14

Step 4 — solve. Eliminating $b$ gives the standard form

w = \frac{n\sum x_iy_i - \sum x_i \sum y_i}{n\sum x_i^2 - (\sum x_i)^2} = \frac{3(25) - (6)(11)}{3(14) - 36} = \frac{75 - 66}{42 - 36} = \frac{9}{6} = 1.5000

and then from the second equation

$b = \frac{\sum y_i - w\sum x_i}{n} = \frac{11 - (1.5)(6)}{3} = \frac{2}{3} = 0.6667$

Step 5 — the residuals and the loss.

r_1 = +0.1667, \qquad r_2 = -0.3333, \qquad r_3 = +0.1667

$\loss = \frac{1}{6}\big(0.0278 + 0.1111 + 0.0278\big) = 0.0278$

against $0.0595$ for the through-origin fit of I.1.B03.

What the intercept bought. The loss more than halved, from $0.0595$ to $0.0278$ . But notice something sharper in the residuals: they sum to $0.1667 - 0.3333 + 0.1667 = 0.0000$ . That is not a coincidence — it is exactly what $\partial\loss/\partial b = 0$ says. A fitted intercept forces the mean residual to zero. The through-origin model has no such constraint, and its residuals do not sum to zero.

So the intercept did not merely add flexibility; it added a specific, nameable property to the fit. That is the honest way to describe what any parameter buys.

I.1.X04Cross-entropy for more than two classesprobability▲▲△

Extend I.1.B04 to $K$ classes. Show that categorical cross-entropy $-\sum_k y_k \log p_k$ , with $\vec{y}$ one-hot, is the negative log-likelihood of a categorical distribution. Then evaluate it for $\vec{p} = (0.7, 0.2, 0.1)$ for each possible true class, and compare the average of those three losses with the entropy of $\vec{p}$ .

Hint

A one-hot $\vec{y}$ turns a sum into a single selected term. For the last part, weight each loss by how often that class actually occurs under $\vec{p}$ itself.

Solution

Step 1 — the likelihood. Generalising the indicator-exponent trick of I.1.B04 from two factors to $K$ :

$P(Y = c) = \prod_{k=1}^{K} p_k^{\,y_k}, \qquad y_k = \mathbb{1}[k = c]$

Every factor with $y_k = 0$ is $p_k^0 = 1$ and drops out, leaving $p_c$ .

Step 2 — take logarithms and negate.

$-\log P(Y = c) = -\sum_{k=1}^{K} y_k \log p_k = \mathrm{CE}(\vec{p}, \vec{y})$

Identical, for the same reason as the binary case: a product of powers becomes a weighted sum of logarithms. Equation (I.1.6) is this result at $K = 2$ with $p_2 = 1 - p_1$ , so nothing new has been assumed — only an index has been allowed to run further.

Step 3 — evaluate. With $\vec{p} = (0.7, 0.2, 0.1)$ :

True class	Loss
1	$-\log 0.7 = 0.3567$
2	$-\log 0.2 = 1.6094$
3	$-\log 0.1 = 2.3026$

For the true class being the second, as asked: $\mathbf{1.6094}$ nats.

Step 4 — the comparison. The plain average of the three is $(0.3567 + 1.6094 + 2.3026)/3 = 1.4229$ nats. The entropy of $\vec{p}$ is

$\mathcal{H}(\vec{p}) = -\sum_k p_k \log p_k = 0.7(0.3567) + 0.2(1.6094) + 0.1(2.3026) = 0.8018$

These are different numbers, and the difference is instructive. The entropy is the probability-weighted average — the expected loss if the labels really were drawn from $\vec{p}$ . The plain average treats all three classes as equally likely, which is a different distribution, and its excess over the entropy is the KL divergence from uniform to $\vec{p}$ (Kullback–Leibler divergence 0.IT.03):

$1.4229 - 0.8018 = 0.6211 = \mathrm{KL}\!\left(\text{uniform} \,\|\, \vec{p}\right)$

The lesson. A model’s expected cross-entropy equals its entropy only when its predictions match the true distribution. Any excess is exactly the divergence between what the model believes and what is true, which is why cross-entropy is bounded below by the data’s own entropy and can never be trained to zero on genuinely noisy labels.

I.1.X05A split that reports a number about nothingcounterexample▲▲△

The first assumption of this chapter is that examples are drawn independently from one fixed distribution. Construct a dataset where that fails, show concretely that the reported test accuracy is then not an estimate of anything the modeller wants, and state the smallest repair.

Hint

Independence is a property of the sampling, not of the numbers. Ask what happens when a single underlying object contributes several rows.

Solution

The construction. Take $100$ patients. From each, take $10$ photographs of the same skin lesion, from slightly different angles: $1{,}000$ images, each labelled benign or malignant according to that patient’s biopsy.

Split the $1{,}000$ images uniformly at random, $800$ for training and $200$ for testing.

Why the assumption fails. The ten images from one patient are not independent draws. They share a lesion, a camera, a skin tone, a lighting condition. Under a uniform split, a given patient’s images land on both sides: with $800/1000$ training, the chance that a particular patient has all ten images in test is $(0.2)^{10} \approx 1.0 \times 10^{-7}$ . In expectation, essentially every patient in the test set also appears in training.

What the model can do. Nothing about the disease needs to be learned. A model that recognises the patient — from a freckle, the skin tone, a background corner of the image — can recall that patient’s label from training and apply it to the test images. It will score near-perfectly.

Concretely: suppose the model learns patient identity with $95\%$ accuracy and nothing else. Test accuracy is then about $95\%$ , against a base rate of, say, $50\%$ . The reported number looks like a strong result.

What it is an estimate of. It estimates performance on new photographs of patients already seen. That is a real quantity, and it is almost never the one anyone wants. The quantity wanted is performance on a new patient, and the experiment contains no evidence about it whatsoever — a model with $95\%$ reported accuracy may be at chance on a new patient, and this design cannot distinguish the two cases.

The smallest repair. Split by patient, not by image. Assign each of the $100$ patients wholly to train or wholly to test. The test set then contains no patient the model has seen, and the number estimates the quantity of interest.

The cost is that the effective sample size falls from $1{,}000$ to $100$ , so the confidence interval widens by roughly $\sqrt{10} \approx 3.2\times$ (Confidence intervals 0.ST.02). That is not a loss — the narrow interval was never real. The honest number is the wider one.

The general form. Wherever rows share a latent generator — patient, site, author, session, scanner, document — independence fails, and the repair is always to split at the level of the generator. Chapter VIII.3 makes this quantitative; Proposition I.1.P05 is where it starts.

I.1.X06Where the loss and the objective come apartlimit▲▲▲

The second assumption of this chapter is that minimising the loss is what the modeller wants. Take a task you care about, write down the loss you would actually train with, and enumerate what the second omits. Then show — with a worked case — that a model can strictly improve the loss while getting strictly worse at the objective.

This is the chapter’s open exercise. There is no single right answer, but there is a standard of rigour: the divergence must be exhibited with numbers, not asserted.

Hint

Averages hide structure. Ask what happens when a loss is averaged over a population containing a small subgroup you care about disproportionately.

Solution

A worked case. The objective: detect a rare condition, present in $1\%$ of a screened population, well enough to be useful in a clinic. The loss: average binary cross-entropy over the screened population.

Model A predicts $p = 0.01$ for every patient, always. It has learned nothing. Its average loss is the entropy of $\mathrm{Bern}(0.01)$ :

$\loss_A = -\big[0.01\log 0.01 + 0.99\log 0.99\big] = 0.0461 + 0.0099 = 0.0560$

Model B genuinely detects the condition, assigning $p = 0.60$ to the $1\%$ who have it and $p = 0.02$ to everyone else.

\loss_B = 0.01\big(-\log 0.60\big) + 0.99\big(-\log 0.98\big) = 0.01(0.5108) + 0.99(0.0202) = 0.0051 + 0.0200 = 0.0251

Model B is better on both counts here, so refine it. Model C is Model B with its confident positives softened to $p = 0.30$ , and its negatives sharpened to $p = 0.005$ :

\loss_C = 0.01(-\log 0.30) + 0.99(-\log 0.995) = 0.01(1.2040) + 0.99(0.0050) = 0.0120 + 0.0050 = 0.0170

$\loss_C = 0.0170 < \loss_B = 0.0251$ . Model C has a strictly lower loss.

But at any threshold above $0.30$ , Model C detects nothing at all, while Model B detects everything it should. On the objective — finding the condition — Model C is strictly worse, and by the loss it is strictly better.

Why. The $99\%$ contribute $99$ times more terms to the average than the $1\%$ do. A tiny improvement on the majority — $0.0202 \to 0.0050$ per example — outweighs a large degradation on the minority, $0.5108 \to 1.2040$ . The arithmetic is $0.99 \times 0.0152 = 0.0150$ saved against $0.01 \times 0.6932 = 0.0069$ lost. The average is doing exactly what an average does.

What the loss omitted. Four things, none of them visible in the number:

Who the errors fall on. An average over a population is indifferent to which subgroup absorbs the error.
The asymmetric cost of the two error types. A missed malignancy and a false alarm are both “wrong” to cross-entropy and are not remotely comparable in a clinic.
The operating point. The loss is computed over all thresholds at once; the clinic uses exactly one, and Chapter III.7 shows that is the only one that matters.
The deployment distribution. The $1\%$ prevalence is the screened population’s, not the referred population’s, and the same model faces both.

The repair, and its limits. Class weighting, focal loss, or a fixed sensitivity constraint each address point 1 or 2. None addresses all four, and each introduces its own hyperparameter that is chosen by looking at the very metric it was meant to replace.

The honest conclusion. The loss is a proxy chosen because it is differentiable, and differentiability is a property of the optimiser’s needs, not the problem’s. Every claim built on a training curve inherits that substitution. The discipline this chapter asks for is not to avoid the substitution — it is unavoidable — but to write down, once and explicitly, what it dropped.