the foundations, zero to one
The Basics
Two labs, no prerequisites. First: how a model reads text. Then: how it decides what to pay attention to. Every number in every lab is real; work through them in order.
How a Model Reads
Chop text into tokens with byte-pair encoding, look up embeddings, and learn how a model decides that two words are similar.
open the lab~16 min
How a Model Thinks
Matrix multiplication, the attention equation computed by hand with query, key, and value, and every attention variant up to KDA.
open the lab~19 min
still in the notebook
Mixture of Experts
How 288 specialists and a router replace one giant feed-forward network.
The KV cache
What models remember during a conversation, and why it grows.
Quantization (FP8, FP4)
Squeezing weights into fewer bits without breaking the model.
How models are trained
Prediction, GPUs, and trillions of tokens.