Book III

Large Language Model (LLM)

Transformers, attention mechanisms, prompt engineering, fine-tuning.

Move from the transformer block to modern language models: how they are trained, scaled, aligned and made to fail.

9 chapters0 propositions written

Chapter ILanguage ModellingPrediction as repeated conditional probability.Probability · Conditional probability · Next-token prediction · Cross-entropy · Perplexitynot yet written
Chapter IIAutoregressive TransformersDecoder-only models, and what makes them generate.Causal masking · Decoder-only architecture · GPT-style models · Context windows · KV cachenot yet written
Chapter IIIPretrainingWhat it takes to train one.Corpus construction · Data filtering · Token budgets · Compute · Scaling · Distributed trainingnot yet written
Chapter IVScalingWhat more parameters, data and compute actually buy.Parameter count · Data · Compute · Scaling laws · Compute and data trade-offs · Inference scalingnot yet written
Chapter VInstruction FollowingFrom a text predictor to something that answers.Instruction tuning · Supervised fine-tuning · Preference optimisation · RLHF · DPO · Alignmentnot yet written
Chapter VIPrompting and In-Context LearningLearning without weight updates.Zero-shot · Few-shot · Chain-of-thought · Structured prompting · Tool use · Reasoning promptsnot yet written
Chapter VIIEfficient AdaptationChanging a model you cannot afford to retrain.Fine-tuning · LoRA · QLoRA · Adapters · Quantisation · Pruning · Distillationnot yet written
Chapter VIIILLM FailureThe parts that do not work.Hallucination · Bias · Context limitations · Reasoning failures · Benchmark contamination · Distribution shift · Evaluation problemsnot yet written
Chapter IXLLM SystemsA model is not yet a system.RAG · Embeddings · Vector databases · Agents · Tools · Memory · Inference systemsnot yet written