Md. Asif Uddin

    Book I

    Neural networks

    the perceptron, backpropagation, optimisation, convolution, recurrence, latent variables, graphs

    How a neural network computes and how it learns. Fourteen chapters, the largest book here, and the one every other book borrows from.

    14 chapters51 propositions written

    Read first

    Mathematics assumed — follow these when a step stops making sense.

    Vectors, matrices and the row-major convention 0.LA.01 · The gradient, and the layout convention 0.MC.01 · The chain rule 0.MC.03 · The chain rule for matrix products 0.MC.07 · Expectation 0.PR.02 · Gradient descent 0.OP.02

    1. Chapter 1I.1Computation and RepresentationA model is a parameterised function, and everything training does is change the parameters.What is a model? · Parameters · Functions and representations · Tensors · Dimensions and shapes · Forward computation · Loss functions · Optimisation · GeneralisationM25 written

      4/3 problems4/3 variants6/6 exercisesquota met, and enforced

    2. Chapter 2I.2The PerceptronA perceptron is a hyperplane that moves toward every point it misclassifies, and it converges only if such a hyperplane exists.The perceptron · Weight vector and bias · Decision boundary · Signed distance and margin · Linear separability · The perceptron learning rule · The convergence theorem · XORM34 written

      7/5 problems6/4 variants10/10 exercisesquota met, and enforced

    3. Chapter 3I.3Activation FunctionsWithout a nonlinearity a stack of affine maps is a single affine map, so depth buys nothing at all.Activation · Nonlinearity · Saturation · Sigmoid and tanh · ReLU and LeakyReLU · GELU · Dead units · Universal approximationM34 written

      7/5 problems7/4 variants10/10 exercisesquota met, and enforced

    4. Chapter 4I.4Loss FunctionsEvery loss is a negative log-likelihood under some noise model, and choosing a loss is choosing that model whether or not anyone says so.Loss and risk · Mean squared error · Mean absolute error · Huber loss · Binary cross-entropy · Categorical cross-entropy · Logits and log-sum-exp · Class weighting · Label smoothingM34 written

      7/5 problems6/4 variants10/10 exercisesquota met, and enforced

    5. Chapter 5I.5The Multilayer PerceptronAn MLP is a composition of affine maps and nonlinearities, and depth can buy exponentially in width.The affine layer · Composition, depth and width · Hidden representations · Parameter and shape accounting · Activation patterns and linear regions · Counting regions: width adds, depth multiplies · The depth–width trade-off · Minimal width and the rank bottleneck · Permutation symmetry of hidden unitsM34 written

      7/5 problems7/4 variants10/10 exercisesquota met, and enforced

    6. Chapter 6I.6Backpropagation and Computational GraphsBackpropagation is the chain rule applied to a computational graph, and reverse mode returns every partial derivative of a scalar output in one traversal.The graph is the program · The forward pass, and what it must keep · The chain rule along a path · Local Jacobians · Reverse accumulation: one traversal, every partial · Forward mode, and when it is the cheaper one · Fan-out and gradient accumulation · Vector–Jacobian products: never forming J · The layer recurrence for an MLP · The memory bill, and checkpointingM33 written

      5/5 problems5/4 variants10/10 exercisesquota met, and enforced

    7. Chapter 7I.7OptimisationThe gradient supplies a direction and never a distance, and every optimiser is a different answer to how far to go.The update rule, and what it leaves unspecified · Descent on a quadratic: the exact stability threshold · Condition number, and why one rate cannot suit two directions · The minibatch gradient as an estimator · Variance, batch size, and square-root returns · Momentum as an exponential moving average · Per-coordinate scaling: AdaGrad to Adam · Bias correction, and what it is correcting · Schedules: warm-up, decay, and the budget they encode · Weight decay, and where it belongsM33 written

      5/5 problems5/4 variants10/10 exercisesquota met, and enforced

    8. Chapter 8I.8Normalisation and RegularisationNormalisation changes the computation the optimiser differentiates; regularisation changes which fit training prefers.The reduction axes are part of the definition · Batch statistics, learned scale and learned offset · Training and inference are different batch-normalisation computations · Layer normalisation without neighbours in the batch · The backward pass through a mean and a variance · What epsilon preserves, and what it changes · L2 penalties, ridge regression and shrinkage · Why AdamW separates decay from the gradient · Inverted dropout: expectation is preserved, variance is not · Early stopping and label-preserving augmentationM32 written

      5/5 problems5/4 variants10/10 exercisesquota met, and enforced

    9. Chapter 9I.9Recurrent Neural NetworksA recurrent network carries a fixed-width state through a sequence. Learning what to remember requires differentiating every use of the shared weights.One transition, shared across time · Sequence outputs, final-state outputs and causal prediction · Unrolling the graph without copying the parameters · Backpropagation through time and summed parameter gradients · Products of state-dependent Jacobians · Vanishing gradients, saturation and transient amplification · Gradient norm clipping and truncated backpropagation · Padding masks, state resets and sequence boundaries · Parameter counts, training memory and streaming inference · Dilated convolutions and the path to attentionM33 written

      5/5 problems5/4 variants10/10 exercisesquota met, and enforced

    10. Chapter 10I.10Gated Recurrence: LSTM and GRUA gate gives the gradient a road home that no weight matrix stands on, which slows the decay without abolishing it.Gates as learned, input-dependent multipliers · The LSTM cell state and its carry line · Forget, input and output gates, and the candidate · The constant error carousel and why the forget gate was added · The full Jacobian of a gated cell, block by block · Forget values, memory horizons and bias initialisation · Growth that gating does not prevent · Backpropagation through an LSTM cell · The GRU, its two conventions and its cost · Gated paths across depth: highway and residual connectionsM33 written

      5/5 problems5/4 variants10/10 exercisesquota met, and enforced

    11. Chapter 11I.11Convolutional Neural NetworksA convolution reuses one local rule across a grid. Its efficiency comes from restricting connectivity and sharing parameters, not from ignoring spatial structure.Cross-correlation, kernels and feature maps · Input channels, output channels and groups · Parameter counts and multiply-accumulates · Padding, stride, dilation and exact output shapes · Backpropagation as accumulation over overlapping windows · Transposed convolution is an adjoint, not an inverse · Translation equivariance and its boundary conditions · Receptive-field size, spacing and effective influence · Max pooling, average pooling and their derivatives · LeNet, AlexNet, VGG, ResNet and EfficientNetM34 written

      5/5 problems5/4 variants10/10 exercisesquota met, and enforced

    12. Chapter 12I.12AutoencodersAn autoencoder learns a representation by reconstruction alone. Under a linear map it recovers the principal subspace and nothing finer; every other autoencoder is defined by the constraint that stops it copying.Reconstruction as a supervised problem whose target is the input · Encoder, decoder, code and the reconstruction surface · Bottlenecks, rank and what an undercomplete code must discard · Gradients of a linear autoencoder, untied and tied · The principal-subspace theorem, proved by two least-squares steps · Why the code's coordinates are not principal components · Saddles, plateaus and what weight decay does to the symmetry · Denoising as shrinkage, and the optimal denoiser as a score · Contractive and sparse penalties, and the scale they must fix · Dense and convolutional autoencoder shapes and parameter counts · What reconstruction error does not measureM34 written

      5/5 problems5/4 variants10/10 exercisesquota met, and enforced

    13. Chapter 13I.13Variational AutoencodersThe ELBO bounds the log-evidence, and the gap is exactly the divergence between the approximate and the true posterior. Its second term prices the code, and a code priced too high is ignored.Latent-variable models: prior, decoder distribution, evidence and posterior · Probabilistic PCA, the linear model where every quantity has a closed form · Why the evidence cannot be estimated by sampling the prior · The evidence lower bound, derived twice, and the exact size of its gap · The Gaussian KL in closed form, and its derivatives · Amortised inference: one encoder in place of one optimisation per example · The reparameterisation trick and the score-function estimator · Variance of gradient estimators, and where each one fails · β, rate and distortion, solved exactly for a linear model · Posterior collapse, and the holes that a prior does not fillM34 written

      5/5 problems5/4 variants10/10 exercisesquota met, and enforced

    14. Chapter 14I.14Graph Neural NetworksMessage passing is permutation-equivariant, and no amount of it can distinguish graphs that the 1-WL test cannot.Node features, adjacency and what a relabelling does to them · Permutation equivariance of layers and invariance of readouts · Message passing: messages, aggregators, updates · Receptive fields on a graph: k layers see k hops · The graph Laplacian, spectral filters and the graph convolution · Why a graph convolution cannot tell its neighbours apart · Over-smoothing: repeated averaging and the second eigenvalue · Colour refinement and the limit it sets on message passing · Sum, mean and max aggregators, and which can count · Sparse computation, batching graphs and neighbour samplingM34 written

      5/5 problems5/4 variants10/10 exercisesquota met, and enforced

    Problem setWhere the chapters have to be used together8 cross-chapter problems, each combining results from two or more chapters of this book.

    Practical connection

    Linear regression, then an MLP, then a reverse-mode autodiff engine, then a convolution — all in NumPy, no framework.

    Your implementation must reproduce the hand-computed numbers to four decimal places before you move on. A reader who skipped the derivations cannot pass their own unit test.

    Verified againstI.5.B01 · I.6.B02 · I.11.B01