A transformer block mixes across positions, then computes within one.

Demonstration

A block is two sublayers, each wrapped in a residual connection:

x ← x + Attention(Norm(x))
x ← x + MLP(Norm(x))

The division of labour is exact and worth stating in those terms. Attention is the only operation in the entire model that lets one position see another. Remove it and the transformer becomes a stack of independent per-token MLPs. The feed-forward network never sees another position. The same two matrices are applied to every token independently, which is why it is properly called position-wise.

The MLP is usually where most of the parameters live. The inner width is conventionally four times the model width, so with d = 4096 the block holds about 134M parameters in the MLP against about 67M in attention. The common mental image of a transformer as “mostly attention” is wrong by parameter count in every standard configuration.

What the MLP does with them is a live research question with a persuasive partial answer: read the first matrix as a set of keys and the second as a set of values, and the layer behaves like a memory that fires on particular input patterns and writes particular things into the stream. Geva and colleagues found neurons that activate on identifiable patterns — a period followed by a capitalised word, a specific factual context — and whose output rows push the prediction in a corresponding direction.

Gated variants (SwiGLU and relatives) replace the single activation with an elementwise product of two projections, one gating the other. They are standard in current models and generally reduce the inner width to about 8/3 d to hold the parameter count constant.

Corollary

Mix, then think. Almost every transformer variant is a change to one half of that sentence: the efficient-attention literature modifies the mixing, the mixture-of-experts literature modifies the thinking by routing each token to a subset of MLPs. Knowing which half a paper touches is most of understanding it.