Md. Asif Uddin

Book I

Neural networks

the perceptron, backpropagation, optimisation, convolution, recurrence, latent variables, graphs

How a neural network computes and how it learns. Fourteen chapters, the largest book here, and the one every other book borrows from.

14 chapters27 propositions written

Read first

Mathematics assumed — follow these when a step stops making sense.

Vectors, matrices and the row-major convention 0.LA.01 · The gradient, and the layout convention 0.MC.01 · The chain rule 0.MC.03 · The chain rule for matrix products 0.MC.07 · Expectation 0.PR.02 · Gradient descent 0.OP.02

  1. Chapter 1I.1Computation and RepresentationA model is a parameterised function, and everything training does is change the parameters.What is a model? · Parameters · Functions and representations · Tensors · Dimensions and shapes · Forward computation · Loss functions · Optimisation · GeneralisationM25 written

    4/3 problems4/3 variants6/6 exercisesquota met, and enforced

  2. Chapter 2I.2The PerceptronA perceptron is a hyperplane that moves toward every point it misclassifies, and it converges only if such a hyperplane exists.The perceptron · Weight vector and bias · Decision boundary · Signed distance and margin · Linear separability · The perceptron learning rule · The convergence theorem · XORM34 written

    7/5 problems6/4 variants10/10 exercisesquota met, and enforced

  3. Chapter 3I.3Activation FunctionsWithout a nonlinearity a stack of affine maps is a single affine map, so depth buys nothing at all.Activation · Nonlinearity · Saturation · Sigmoid and tanh · ReLU and LeakyReLU · GELU · Dead units · Universal approximationM34 written

    7/5 problems7/4 variants10/10 exercisesquota met, and enforced

  4. Chapter 4I.4Loss FunctionsEvery loss is a negative log-likelihood under some noise model, and choosing a loss is choosing that model.M3not yet written

    0/5 problems0/4 variants0/10 exercisesowes 15 more

  5. Chapter 5I.5The Multilayer PerceptronAn MLP is a composition of affine maps and nonlinearities, and depth can buy exponentially in width.M31 written

    0/5 problems0/4 variants0/10 exercisesowes 15 more

  6. Chapter 6I.6Backpropagation and Computational GraphsBackpropagation is the chain rule applied to a computational graph, and reverse mode returns every partial derivative of a scalar output in one traversal.M31 written

    0/5 problems0/4 variants0/10 exercisesowes 15 more

  7. Chapter 7I.7OptimisationThe gradient supplies a direction and never a distance, and every optimiser is a different answer to how far to go.M33 written

    0/5 problems0/4 variants0/10 exercisesowes 15 more

  8. Chapter 8I.8Normalisation and RegularisationNormalisation and regularisation do two different jobs, and keeping them adjacent is what makes the difference visible.M31 written

    0/5 problems0/4 variants0/10 exercisesowes 15 more

  9. Chapter 9I.9Recurrent Neural NetworksA recurrent network compresses everything it has read into one fixed-width state, and the gradient through that compression is a long product.Sequences · RNNs · LSTMs · GRUs · CNNs for sequences · Long-range dependencies · Why transformers emergedM33 written

    0/5 problems0/4 variants0/10 exercisesowes 15 more

  10. Chapter 10I.10Gated Recurrence: LSTM and GRUA gate gives the gradient a road home that no weight matrix stands on, which slows the decay without abolishing it.M31 written

    0/5 problems0/4 variants0/10 exercisesowes 15 more

  11. Chapter 11I.11Convolutional Neural NetworksA convolution applies one small filter at every position, because a feature's meaning does not depend on where it sits.kernels · convolution · padding · stride · pooling · feature maps · CNN architectures · LeNet · AlexNet · VGG · ResNet · EfficientNetM34 written

    0/5 problems0/4 variants0/10 exercisesowes 15 more

  12. Chapter 12I.12AutoencodersAn autoencoder learns a representation by reconstruction alone, and under a linear map it recovers the PCA subspace.M2not yet written

    0/3 problems0/3 variants0/6 exercisesowes 9 more

  13. Chapter 13I.13Variational AutoencodersThe ELBO bounds the log-evidence, and the gap is exactly the divergence between the approximate and the true posterior.M3not yet written

    0/5 problems0/4 variants0/10 exercisesowes 15 more

  14. Chapter 14I.14Graph Neural NetworksMessage passing is permutation-equivariant, and no amount of it can distinguish graphs that the 1-WL test cannot.M3not yet written

    0/5 problems0/4 variants0/10 exercisesowes 15 more

Problem setWhere the chapters have to be used togetherNot yet written. A book with load-bearing chapters owes at least eight cross-chapter problems and two that reach back into an earlier book.

Practical connection

Linear regression, then an MLP, then a reverse-mode autodiff engine, then a convolution — all in NumPy, no framework.

Your implementation must reproduce the hand-computed numbers to four decimal places before you move on. A reader who skipped the derivations cannot pass their own unit test.

Verified againstI.5.B01 · I.6.B02 · I.11.B01