Square-root returns, and where they stop being worth buying

By (I.7.2) noise falls as $1/\sqrt{B}$ while cost rises as $B$. Form the product and read off which direction it moves.

The trade.

Quadrupling $B$ halves the noise and costs four times the arithmetic, at every scale. The last column — noise per unit of compute — doubles each time, which is the trade stated in the least flattering way that is still true: measured per FLOP, a large batch is strictly worse at reducing noise.

Why anyone uses large batches anyway. The table prices arithmetic, and arithmetic is not the binding constraint on modern hardware. A batch of 256 does not take 256 times as long as a batch of 1 — it takes barely longer, because the device was idle. The square-root law is about statistical efficiency; large batches win on hardware efficiency, and the two pull in opposite directions.

Where it stops. Comparing noise against the signal it corrupts:

Once the noise is small against the gradient, further averaging refines a number that was already accurate enough to step with. That boundary is McCandlish and colleagues’ critical batch size, and the useful thing about it is that it is a property of the problem at a point in training, not of the hardware — it generally grows as training proceeds and the gradient shrinks.

What the budget does not decide. With a fixed budget of $10^6$ example-visits:

Same compute, three different training runs: many noisy steps or few clean ones. The budget constrains their product and says nothing about which is better — that is decided by where the problem sits relative to its critical batch, and by whether the noise is doing anything useful, which by I.7.P02 it sometimes is.