the basics · lab 1 of 2
How a Model Reads
A language model cannot read letters. Everything it will ever see is a number, and the journey from text to numbers happens in two steps: tokens, then embeddings. Every number in this lab is real; work through the three stations in order.
back to the contentsThe three stations:
lab 1: how a model reads
How a Model Reads
A language model cannot read letters. Everything it will ever see is a number, and the journey from text to numbers happens in two steps. First the text is chopped into pieces called tokens. Then each token is swapped for a long list of numbers called an embedding. This lab builds both steps from scratch, and ends with the trick that makes embeddings useful: measuring how similar two meanings are.
1.From letters to tokens
The problem: a model has a fixed table of entries (its vocabulary) but language never stops producing new words. The fix is to learn the pieces instead of the words. The algorithm, byte-pair encoding (BPE), is almost embarrassingly simple: start with single characters, count which two neighboring pieces appear together most often, and glue them together. Repeat millions of times. Frequent words become single tokens; rare words stay as readable fragments.
Work through it yourself. Each click of Learn a merge performs one real fusion on a nine-word corpus, exactly as the algorithm would.
Lab 1a: teach a tokenizer from scratch
Click Learn a merge repeatedly. Each click fuses the most frequent adjacent pair in the corpus.
learned vocabulary so far (11 tokens): the 11 characters that appear in this corpus, plus every merged pair:
now tokenize a word with what the tokenizer knows (training words in bold):
lowest + end-marker → l · o · w · e · s · t · _
Notice what the merges bought us. After five rounds, lowest reads as two tokens instead of seven, and a word the corpus never contained, like slowly, still tokenizes into sensible pieces. That is the whole trick that lets a 154,000-token vocabulary cover all of English, code, and most of the internet.
2.From tokens to numbers
Now the model needs to turn the token king into something it can do arithmetic with. It keeps a giant table: one row per vocabulary entry, where each row is a vector of numbers (4,096 of them in GLM-5.3-Flash; we use 4 here so you can see every digit). Finding a token's embedding is literally looking up its row. The interesting question is what those numbers mean and how the model compares two of them. The answer is one operation you already know from grade school: multiply matching positions and add everything up.
Lab 1b: an embedding is just a row of numbers
Pick two words. The table multiplies their vectors dimension by dimension; that is the whole calculation.
Above 0.9: these vectors point in nearly the same direction. The model treats these words as close.
That score, the cosine similarity, is the model's only notion of “these two words are related.” There is no dictionary anywhere in the model. There are only vectors that point in similar directions.
3.What did the numbers actually learn?
Nobody tells the embedding table that king and queen are related. During training the model sees billions of sentences and is repeatedly asked to predict the next word. To be good at that, it is forced to place words used in similar contexts at similar coordinates. king and queen appear before of England, wore a crown, and ruled for in the same ways, so their vectors drift together. Words that never share contexts (say, banana and king) end up far apart, even though both are perfectly ordinary words.
This gives us a genuinely useful test for whether two concepts are similar, and it has nothing to do with definitions. Throw a word at the map below: it lands next to the words the model would score as its closest neighbors.
Lab 1c: the dart map
1. Pick a word: its 2D position is its embedding. 2. Click anywhere on the map to throw. 3. Read the distances.
dashed teal circles are equal-distance lines around your word: everything on one ring is equally similar to it.
Two things on that map are worth remembering forever. First, opposites land close together (happy next to sad) because similarity measures interchangeability of context, not sameness of meaning. Second, the map has countries nobody designed: feelings, fruits, vehicles, programming languages. The geography emerged from text statistics alone. Keep both facts in mind and half of every model guide on this site becomes common sense.