The commonplace book
Marginalia
Notes written in the margins of the work: what I thought of a book, what a model actually does when you use it, and the occasional argument that did not fit anywhere else. Kept in one ledger, numbered as they are written.
Three parts. The first is the ledger — my opinion of something after reading it. The second is a reading course: what somebody else should read, in what order, and what to take from each one. The third is the shelf those two are worked out on — the textbooks, which are not read in an order and so are not filed in one.
RecensionesWhat I thought of it
- On asking thirty-seven timesEvery hospital said no. One research group said yes. That is the dataset.
- What open actually buysThe strongest argument in biology right now for releasing everything.
- Two benchmarks, one model, opposite answersBoth claims are in print. Working out how they can both be true is more useful than picking a side.
- The baseline nobody specifiesThe most useful model in single-cell biology, and the one nobody writes about.
- The architecture won, the argument did notThe paper is careful, conditional and honest about its negative result. What spread was a slogan.
- The number a chemist can useA neural network reaching the correlation of a physics simulation, three orders of magnitude faster.
- Refusing the tradeoffThe most convincing large model in genomics right now, and the most frustrating to actually use.
- The model is fine. The framing wasn'tWhat Geneformer actually is Geneformer is a bidirectional encoder, BERT in shape, pretrained on Genecorpus-30M and later extended to around 95 million cells. The interesting design decision is the input representation. Single-cell expression data is a problem for tokenization. Counts are sparse, noisy, and vary in scale between cells for reasons that have nothing to do with biology. Feeding raw values invites the model to learn sequencing depth. Geneformer's answer is rank value encoding. Within each cell, genes are ranked by expression, but normalized first by that gene's median expression across the entire corpus. A gene that is highly expressed everywhere gets demoted. A gene that is unusually high in this particular cell rises. The result is a ranked list per cell where position encodes something closer to cell identity than to library size. Then the standard masked objective. Hide some genes in the ranking, predict them from the rest.
- EfficientNet, V1 and V2Fewer FLOPs, 2.7x slower, same accuracy. The name did a decade of damage.
- What the registers fixedDense features that no longer need apologising for.
- Llama 3.3 70B and Llama 4Ten million tokens of advertised context. At a hundred and twenty thousand, 15.6%.
- Mixture of expertsA parameter count is no longer a capability claim. It is a hardware requirement.
- Kimi K3 and K2.6They said the weights would be out by July 27. On July 27, the weights were out.
- DeepSeek V4Everyone quoted the 1.6 trillion. The number that matters is 10%.
- The Machine Learning Guide to CausalityMIT Press, 2017. It's open access, so the PDF is free, and there are companion Jupyter notebooks with coding exercises. This book connects machine learning with causal reasoning. It shows how algorithms can figure out cause and effect just by looking at regular data.
- The Starter Guide to Causal InferenceWiley, 2016. Pearl with Madelyn Glymour and Nicholas Jewell, a Berkeley biostatistician, and it shows. This is the only one of the three that behaves like a textbook. This book is a beginner-friendly guide to understanding cause and effect. It uses clear examples and simple math to help you build models and solve real-world problems.
- Beyond Correlation: A Review of Pearl’s CausalityCambridge University Press, 2nd edition 2009, first published 2000. Eleven chapters, four hundred-odd pages, no attempt to be charming. This book helps AI and statistics move past simple correlation to understand true cause and effect. Pearl provides math tools to help readers map relationships, predict outcomes, and test "what if" scenarios.
- The ladder is the argumentA polemic disguised as a popular science book, and better for it.
- SAM 3SAM 1 and 2 answered where. This one answers what, and the clever part is one token.
- Qwen3.8First Max-class Qwen released as weights. Both halves of that headline need checking.
- Qwen, the familyNot any single model. A ladder you can climb without changing vendor, and weights you can keep.
- EVA-CLIP, 8B and 18BGoing from 8.1B to 18B bought 0.7 points. The most useful thing in the paper is that number.
- nnU-Net, the 3D oneIt is not an architecture. That is the whole point, and it is why the paper gets misread.
- The instrument, not the competitorEvery linear probe in every self-supervised paper is this model. It is the ruler, and nobody calls it by name.
- Swin UNETR V2A transformer that had to be handed convolutions back before it worked properly.
- ConvNeXt V2The least exciting model I use, and it shows up in more of my projects than anything else.
- DINOv2 and DINOv3I still start projects on the three-year-old one. That is not nostalgia, it is licensing.
- SigLIP 2Models I actually use, and what annoys me about them.
Nothing of that kind yet.
LectionesWhat to read, in order
A reading course in machine learning, published in parts of three to five papers. Every paper appears exactly once, so finishing a part means something. 12 parts so far, 54 papers.
- FoundationsWhat a deep network is, and the four papers that settled it.4
- The TransformerOne architecture, and the three papers that showed what it could carry.4
- AlignmentTurning a model that predicts text into one that answers you.5
- Generating before the language modelFour ways to learn a distribution you can sample from.4
- Vision after the TransformerWhat happened to computer vision once attention arrived.5
- Language and vision in one modelHow a model comes to see and talk about the same thing.5
- Position, memory and the cost of attentionAttention is quadratic and the cache does not fit. Five papers about that.5
- Fine-tuning without the computeThree papers, five years, and the reason you can tune a large model on one GPU.3
- Sparsity and scaleWhy a model can have 671 billion parameters and use 37 billion of them.5
- ReasoningFour papers in which nothing about the model changes and the answers get better.4
- RetrievalGiving a model access to things it was not trained on.5
- AgentsWhere the model stops answering and starts doing.5
The whole course, with what each part is for
BibliothecaThe books behind it
Textbooks rather than papers, and kept apart from the course for a reason: a part of the Lectiones is a week of reading in an order that matters, while a textbook is opened at the chapter you need and closed again. 20 books on 4 shelves.