Every diagram on this site, at its own address. All are hand-written inline SVG using the site palette, so they stay legible in both themes and paste into a slide deck without becoming a screenshot of a screenshot.
Fig. 1 — The HierarchiRetina cascade: a screening gate, lesion and vessel segmentation, and a lesion-guided grader with separate severity and gradability heads.
Fig. 3 — The axes of a tensor, named. The same numbers arranged along different axes are different data, which is why a shape is a claim about meaning and not only about storage.
Fig. 5 — A loss surface drawn as contours, with a deep basin and a shallower one. Descent finds the nearer minimum, which is not always the better one.
Fig. 8 — One number carries both facts: its sign is the predicted class and its magnitude, once divided by the norm of w, is the distance to the boundary.
Fig. 9 — An update turns the weight vector toward the point it got wrong, and the boundary swings with it. A point already correct produces no change at all.
Fig. 13 — Cross-entropy contributes the reciprocal of the output derivative, so the two cancel. Squared loss contributes nothing, so the derivative survives and shrinks the gradient exactly where the model is most wrong.
Fig. 14 — The sigmoid derivative never exceeds one quarter, and a product of them reaches the smallest half-precision number by depth twelve — in the best case that never occurs.
Fig. 17 — One unit: a weighted sum, a bias, and an activation. Everything before the activation is affine, and affine maps compose into a single affine map.
Fig. 18 — Two linear layers composed give one line whatever the depth. The same two layers with a rectifier between them give a piecewise curve, one fold per unit.
Fig. 19 — One graph traversed twice: values forwards, partial derivatives backwards. Backpropagation is the chain rule with the intermediate results kept.
Fig. 20 — The same descent under three learning rates. Too small and it creeps, too large and it climbs the far wall; the gradient supplies the direction, never the distance.
Fig. 21 — Seven points fitted exactly and fitted loosely. Regularisation does not improve the fit — it decides which fit you get when many are available.
Fig. 22 — The same matrix, different reduction groups. Batch normalisation shares statistics down a feature column; layer normalisation shares them across one example. Highlighted entries form one group; the remaining rows or columns form their own groups.
Fig. 24 — Byte-pair encoding, three merges deep. The merge list is the tokeniser; it is fixed by frequency in a corpus before training begins, and nothing in it is learned by the model.
Fig. 26 — One token in two sentences. It enters as the same vector in both, and only separates once a layer of attention has let it read its neighbours.
Fig. 27 — A recurrence unrolled. Every step passes a fixed-width state to the next, so the path between two distant positions is as long as the distance between them.
Fig. 28 — Constant scalar Jacobian factors over sixteen steps, on a logarithmic scale. These examples isolate decay and growth; varying factors and matrices also depend on their order and direction.
Fig. 29 — An LSTM cell. The carry line crosses only a multiplication by the forget gate and an addition of what the input gate admits, so with the forget gate near one the gradient has a route home that no weight matrix attenuates. The output gate decides how much of the cell becomes visible.
Fig. 30 — The carry line's own factor for constant forget values 0.9, 0.99 and 0.999 over a thousand steps, on a logarithmic scale. Each crosses the dashed 1/e line within one step of 1/(1 − f). A forget value near one slows the decay; it does not stop it.
Fig. 31 — A GRU in PyTorch's convention. The update gate keeps a share z of the old state and writes 1 − z of the candidate, so the direct derivative along the state line is diag(z). Other descriptions swap z and 1 − z, or apply the reset gate before the recurrent matrix.
Fig. 32 — Paths from fifteen inputs to one selected output through width-three centred convolutions with dilations 1, 2 and 4. A causal stack shifts this window into the past. Positions within each layer can be evaluated in parallel when the preceding layer is available.
Fig. 33 — Four ways to read a sequence, placed by path length and parallelism. Attention is not a better idea in the abstract; it is the trade that pays on parallel hardware.
Fig. 34 — Attention as a weighted average over values, with weights computed by comparing a query against every key. Nothing in the computation depends on position.
Fig. 35 — Query, key and value as three learned projections of one vector: what a token is looking for, what it advertises, and what it hands over once matched.
Fig. 36 — The same attention logits, scaled and unscaled. Without the divisor the distribution collapses onto one key and the gradient through the softmax goes flat.
Fig. 38 — The score matrix at three sequence lengths. Constant path length between any two positions is bought with an area that grows as the square of the sequence.
Fig. 39 — Permutation equivariance and its repair. Without positional encoding a reordered input yields a merely reordered output; with it, order becomes information.
Fig. 40 — Sinusoidal encoding as a bank of clocks. Fast dimensions separate neighbouring positions, slow ones separate distant regions, and together they give each position a signature.
Fig. 41 — Rotary encoding turns position into an angle. Both the query and the key are rotated, so the absolute indices cancel in the inner product and only the distance survives.
Fig. 42 — Past the training length a learned table has no row at all, and a periodic scheme has a value but no experience. A long-context claim is a claim about training, not about the encoding.
Fig. 43 — One transformer block: attention moves information between positions, the feed-forward network does the work within one, and both are wrapped in a residual addition.
Fig. 44 — One attention budget divided into four heads. Heads are slices rather than copies, so several relations are attended to at once at the cost of resolution inside each.
Fig. 45 — The residual stream. Blocks read from it and add back into it rather than replacing it, which is both why gradients reach layer one and why intermediate states can be read.
Fig. 46 — Post-norm puts a normalisation on the shortcut; pre-norm leaves the shortcut clean. The same two components in two orders, with different training behaviour.
Fig. 47 — The causal mask. Setting the future to minus infinity before the softmax is what lets every position in a sequence be trained as a separate prediction in one pass.
Fig. 48 — Encoder, decoder, and the two joined by cross-attention. Same block, same arithmetic — the masks and the wiring are the whole of the difference.
Fig. 49 — The batch gradient as an estimate of the true one. A small batch scatters widely and steps often; a large batch points true and steps rarely.
Fig. 50 — Warm-up then decay. The rise exists because the optimiser has no statistics yet; the fall exists because a large step near the end discards what the earlier ones found.
Fig. 51 — Gradient norms over fourteen steps with a clipping threshold. Clipping touches only the rare spike; weight decay pulls on every parameter every step. They guard different failures.
Fig. 52 — Representable range in three float formats. Half precision is narrow, and late-training gradients fall through its floor — which is what loss scaling exists to prevent.
Fig. 53 — The same six images split two ways. Splitting by image puts every patient on both sides; splitting by patient is the only one of the two that measures what the claim is about.
Fig. 55 — Dense features rot as a large ViT trains, because the image-level objective wants abstraction and wins. Anchoring the patch-to-patch Gram matrix to an earlier checkpoint drags them back.
Fig. 56 — A kernel slides across a masked hole and leaks the answer; sparse convolution stops it, and global response normalisation stops the channels collapsing. Neither half works alone.
Fig. 58 — What a linear probe measures, and what does the measuring. The encoder is frozen; the classifier on top is logistic regression, and its convexity is why it can serve as a ruler at all.
Fig. 59 — A dataset fingerprint resolved by heuristic rules into a whole pipeline under a memory budget. The architecture is not among the things being chosen.
Fig. 60 — Doubling the parameters on a fixed dataset bought seven tenths of a point. Raising the input resolution, with no extra parameters at all, bought six.
Fig. 61 — The whole ladder shipped at once, with the top rung held back. The gap between the open rungs and the closed flagship is the honest measure of the commitment.
Fig. 62 — The two checkpoints in one release. The headline belongs to the 2.4 trillion; the 27B is the one that fits on hardware you can buy, and it is the Apache one.
Fig. 63 — Whether the concept is present and where it is are different questions that used to interfere. Giving presence its own token and multiplying the scores is most of why the numbers moved.
Fig. 64 — At a one-million-token context, roughly a quarter of the per-token operations and a tenth of the key-value cache. Everyone quoted the parameter count.
Fig. 65 — Rate card against measured cost per task. A higher price per million tokens and a lower bill per job are not contradictory when the answers run token-efficient.
Fig. 66 — Every expert is stored; only the routed few compute. Memory tracks the big number and arithmetic tracks the small one, which is the whole reason anyone does this.
Fig. 68 — Two models at the same accuracy: one with 1.8x fewer operations and 2.7x more wall-clock time. Accelerators are bound by memory bandwidth, not arithmetic.
Fig. 69 — Context against output resolution. Every earlier model sits on one edge or the other; the claim is that the corner between them was an engineering limit rather than a law.
Fig. 70 — Co-folding says whether two molecules fit, free energy perturbation says how tightly and runs slowly. Reaching the second at the speed of the first is what changes screening.
Fig. 71 — Accuracy against pretraining set size. The paper's finding is a crossover with a threshold in it, somewhere near a hundred million images — not a verdict.
Fig. 72 — Three departures from a textbook autoencoder, each one a fact about the assay: a count likelihood, batch as a conditioning variable, and library size given its own latent.
Fig. 74 — Operators assigned by the range they cover, so attention is spent only on the sparse long-range relationships that need it. A megabase becomes affordable.
Fig. 82 — An autoencoder. The encoder maps five input numbers to two code numbers, the decoder maps them back, and the loss compares the output with the input it started from. No label enters, and with the code narrower than the input something has to be left out.
Fig. 83 — Four points and the best line through them, drawn to scale. Each corner lies at distance one from the principal axis, so the mean squared reconstruction error is one, the discarded eigenvalue. Keeping the first coordinate instead costs 2.08.
Fig. 84 — What four linear autoencoders do to the two principal directions of the same data. A bottleneck keeps one and drops the other. Noise and weight decay shrink both, and shrink the weak direction more.
Fig. 85 — A circle encoded by its angle. Reconstruction is exact everywhere, yet two neighbouring inputs sit at opposite ends of the code, and the code halfway between them decodes to the far side of the circle.
Fig. 86 — A variational autoencoder in training. The encoder returns a mean and a width, the code is drawn as μ + σε, and the loss adds the decoder's reconstruction error to the KL divergence from the encoder's distribution to the prior that generation will draw from.
Fig. 87 — The log-evidence of the point (1, 2) under probabilistic PCA, and the lower bound four encoders give it. The gap is the KL divergence from each encoder's distribution to the true posterior, and it closes only at the posterior.
Fig. 88 — The same draw written two ways. On the path, a sample has no derivative with respect to its parameters. Beside it, the noise enters from outside and the code is a sum and a product the chain rule can pass through.
Fig. 89 — Rate against distortion for the linear β-VAE on the chapter's four points, drawn from the exact solution. Raising β buys a cheaper code with a worse reconstruction; at β = 4 the code carries nothing and the decoder ignores it.
Fig. 90 — One round of message passing at node 3 of a four-node graph. The neighbours' states are summed, so their order cannot matter, and the same rule runs at every node. Renumbering the nodes renumbers the outputs and changes nothing else.
Fig. 91 — A grid convolution has one weight per relative position, so it can tell left from right. A graph convolution has no positions to weigh: each edge's coefficient comes from the degrees at its ends, and one weight matrix is shared by every neighbour.
Fig. 92 — Ten rounds of the graph convolution's averaging on the four-node graph, from the values 1, 2, 3 and 4, each divided by the square root of its degree. Nodes 1 and 2 merge after one round, and all four converge to one value at the rate set by the second eigenvalue, 0.564.
Fig. 93 — Two triangles and one hexagon. Every node has degree two and neighbours of degree two, so colour refinement cannot separate the graphs, and no message-passing network can either, although one has two triangles and the other none.