the basics · lab 2 of 2
How a Model Thinks
Attention connects tokens to each other, and it is built from one primitive: the matrix multiplication. Build the primitive, run one full attention computation by hand, then tour every attention design up to the delta-rule scheme in Kimi K3.
back to the contentsThe three stations:
lab 2: how a model thinks
How a Model Thinks
Embeddings give every token a list of numbers, but a list alone knows nothing about its neighbors. The mechanism that connects tokens to each other is attention, and it is built from one primitive: the matrix multiplication. We will build that primitive first, then run one complete attention computation by hand, and finish with a tour of every attention design used in modern models, from the 2017 original to the delta-rule scheme inside Kimi K3.
4.The one operation everything uses
A matrix is just a grid of numbers. Multiplying two grids means: for each cell of the answer, walk along one row of the first grid and one column of the second, multiply the pairs, and add them up. That is it. It sounds trivial, and it is. The surprise is that nearly everything a model does, including attention itself, is this operation at enormous scale. A modern GPU spends almost its whole life doing it.
Lab 2a: what a matrix calculation is
Click any cell of the answer grid. The worked example shows every multiplication that produced it.
A (2×3)
B (3×2)
C (2×2), click a cell
cell C[1][1] = row 1 of A · column 1 of B:
1 × 7 + 2 × 9 + 3 × 11 = 58
three multiplications and two additions per cell. This exact operation, at enormous scale, is nearly all of what a GPU does while running a model.
5.The attention equation, by hand
Take the sentence the cat sat on the mat, plus a new word: it, as in it was tired. A human knows “it” means the cat. Attention is the mechanism that lets the model reach the same conclusion using nothing but the matrix product you just learned. Each token plays three roles:
A query (what am I looking for?), a key (what do I offer for matching?), and a value (what do I contribute if chosen?). The query of it is compared with the key of every earlier token, the scores are turned into percentages that sum to one, and the values are blended by those percentages. Five stages, each one small. Step through them:
Lab 2b: attention, computed in place
Pick the query word, then click Next step: each dot product, the scaling, the softmax, and the final blend appear one by one, in place.
That is the entire mechanism. Softmax(Q·Kᵀ/√d)·V is not hiding anything deeper than what you just computed: score, normalize, blend. When a model guide says “the indexer scores every token” or “the KV cache stores keys and values,” you now know exactly which quantities those are.
6.Every kind of attention, and what each one costs
The version you just computed has a flaw that gets worse by the word: every token must be compared with every earlier token, forever. At a 1-million-token context that is about a trillion comparisons per layer, and every token's keys and values must be remembered. Everything that follows is a different bargain with those two bills. Click through the family in order; each design exists to fix the cost the previous one left behind.
Lab 2c: the attention family tree
Click through the six designs in order. Each one attacks the cost of the previous one; none escapes every tradeoff.
the 2017 starting point. Everything after exists to cut its bill.
Full / MHA (2017)
Multi-Head Attention
Every token reads every earlier token. The original design, still the accuracy ceiling.
reads per query
n per query (all of history)
memory bill
grows with context: every token stores its key and value for every layer
what it buys
- + Exact recall of any detail
- + Simple, well understood
- + Nothing is ever thrown away
what it costs
- - Quadratic compute: 1M tokens = ~1 trillion comparisons per layer
- - Cache eats memory at long context
used by: GPT-3, Llama 2, and every early transformer
the lab 2b computation, rerun under Full / MHA rules:
every token read: the baseline dot products for the query of it
cat: 3.90 ÷ 2 = 1.95 → 67.5%
sat: 1.60 ÷ 2 = 0.80 → 21.4%
mat: 0.30 ÷ 2 = 0.15 → 11.2%
the two bills at a 1M-token context (relative, per layer):
compute per token
memory for history
No design wins both bars. The models in the autopsies mix designs precisely because of this: Kimi K3 pairs 69 KDA layers (tiny memory) with 24 Gated MLA layers (exact recall); GLM-5.3-Flash pairs 34 KDA layers with 11 sparse layers.
If you retain one sentence from this lab, make it this one: there is no best attention, there is only a budget. The models on the dissection table are interesting precisely because of how they spend that budget, and you are now equipped to read every tradeoff they make.