Md. Asif Uddin

Chapter 6 I.6

Backpropagation and Computational Graphs

Backpropagation is the chain rule applied to a computational graph, and reverse mode returns every partial derivative of a scalar output in one traversal.

The graph is the program, and the backward pass is a second traversal that reuses the first.

How this chapter is built

M3Load-bearing

The content is mathematics. Understanding is demonstrated by computation, not recall.

basics3/11what the words mean
concept2/2what to picture
theory0/4why it works, and when it does not
mathematics0/15derive it, then compute it
practice0/9build it, break it, read the papers

Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

Before you start

The problem

An MLP has parameters and a loss but no way to connect them. Every gradient computed anywhere in Elementa is computed by the rules derived here, which is why this is the chapter the rest of the corpus borrows from most.

One graph, traversed forwards then backwardsFive nodes in a row: input, a linear step, an activation, a second linear step and the loss. Arrows along the top run left to right carrying values. Arrows underneath run right to left carrying partial derivatives, each one multiplied into the next by the chain rule.forward — valuesbackward — gradientsxu = Wxh = σ(u)ŷ = VhL∂L/∂·∂L/∂·∂L/∂·∂L/∂·Each backward arrow is one local derivative multiplied into what arrived from the right.The cost of the backward pass is the cost of the forward pass, within a small constant.
Fig. 6 — One graph traversed twice: values forwards, partial derivatives backwards. Backpropagation is the chain rule with the intermediate results kept.

Apparatus

The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

The gradient, and the layout convention 0.MC.01 · The Jacobian 0.MC.02 · The chain rule 0.MC.03 · The derivative of a linear map 0.MC.04 · The chain rule for matrix products 0.MC.07 · The identity table 0.MC.08

Notation

  • ∇Gradient operator
  • JA Jacobian matrix
  • WA weight matrix
  • bA bias vector
  • ℒThe loss
  • IThe identity matrix

Propositions

  1. Prop. 1Backpropagation is the chain rule with the intermediate results kept.There is no separate learning algorithm inside a network. There is a graph, one traversal forwards to compute values, and one backwards to accumulate derivatives.

Worked problems

0/5 problems0/4 variants0/10 exercisesowes 15 more

Not yet written. At M3 this chapter owes 5 worked problems across 4 distinct variants, and 10 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.