GLM-5.3-FLASH

Attention is the part of a transformer, a neural network made of repeated layers, that lets the next token use relevant information from earlier tokens. A token is the unit a model reads and writes. It may be a whole short word, part of a longer word, or punctuation. To read a complete prompt, ordinary full attention considers relationships between every token position and every other position. If the prompt has n tokens, that work grows roughly as n², called quadratic growth. Doubling the context therefore makes about four times as many pairwise comparisons: 128,000 tokens need about 16 billion comparisons, while 1 million tokens need about 1 trillion. GLM stands for General Language Model. GLM-5.3 Flash combines two ways to make that growth manageable: most layers carry forward a compact summary, while a smaller set can still retrieve precise details from the full history.

How can a layer remember without rereading?

Thirty-four of Flash's 45 layers use Kimi Delta Attention (KDA), a linear-attention design from Moonshot's Kimi Linear research. A full-attention layer keeps a separate record for each token it has seen. A linear-attention layer carries one fixed-size state forward instead. In KDA, each attention head, an independent pattern-matching channel inside a layer, maintains a 128×128 grid of numbers. Each arriving token updates that grid once, so the work per new token stays constant as the conversation becomes longer.

A simple running state can only accumulate, which makes it hard to replace an outdated association. KDA uses a delta rule to edit its state. A key is an address-like representation used for matching, and a value is the information stored with that key. Its update rule is:

S_t = (I − β·k kᵀ) · Diag(α_t) · S_{t−1} + β·k vᵀ
      └── erase old note for this key ──┘   └ write ┘
         each of 128 channels decays at
         its own learned, data-driven rate

Here S_t is the state after token t. I is the identity matrix, β controls the update strength, and Diag(α_t) puts the learned decay rates on a matrix diagonal. The update weakens an old record that matches the new key, lets each channel decay at its own learned rate α, then writes the new key and value. Different channels can therefore preserve information over different spans. A fixed state is still a summary, so it is useful for broad context but cannot guarantee an exact lookup of every detail. In Flash, the 34 KDA layers use a fixed 71.3 megabytes (MB) of state whether the context holds 1,000 or 1 million tokens.

The matrix above shows the trade from the inside. A query row in full attention is a long bar of reads: useful tokens bright, irrelevant ones dim, but everything is read and scored. The sparse design keeps that matrix and trims it to the few cells the indexer ranks highest. KDA replaces the whole row with a single teal bar: one summary read. Move the query around and watch the read count under the matrix change; then switch to the KDA view and see it stay at 1. That single read is why 34 layers of Flash cost the same at 1,000 tokens as at 1 million, and also why those layers alone cannot answer an exact needle query. The 11 sparse layers exist to cover that gap.

How can a layer search the whole history selectively?

A fixed state can retain the gist of a long exchange, but an exact error code from a file seen 900,000 tokens earlier needs access to individual tokens. Flash reserves 11 sparse layers for that job. Each sparse layer has a small scout called the lightning indexer . It assigns a cheap relevance score to every past token, keeps the top 2,048, and runs full attention only on that selected set. At a 1-million-token context, 2,048 tokens are about 0.2% of the history. The indexer still sees the whole history; the expensive comparison is limited to the selected tokens.

The GLM-5 report tested these choices by retraining a 9-billion-parameter model with four lower-cost attention schemes and measuring long-context recall. At 128,000 tokens, full attention scored 75.3. A naive sliding window scored 44.9 because it cannot reach tokens outside its recent fixed region. Smarter window placement recovered much of that loss. The report's DeepSeek Sparse Attention (DSA)-style indexer can score any past token before selecting its top candidates, so the architecture does not permanently exclude a position just because it is old. That is the narrower meaning of “lossless by construction”: it preserves a route to every token, while the indexer still has to rank the useful ones well. The table also includes a Gated DeltaNet (GDN)-style linear baseline, which keeps a fixed-size state rather than directly retrieving individual past tokens.

The 11 sparse layers also use No Positional Encoding (NoPE), shown in the configuration as qk_rope_head_dim: 0. Attention normally needs some signal about token order. A common choice is Rotary Position Embedding (RoPE), which adds position to the query and key representations. Flash's sparse layers omit that positional tail. KDA decay already carries a signal of recency, and the indexer has its own positional signal. Removing RoPE from these sparse layers avoids an extra positional component while the model is trained for a 1-million-token window.

Drag the slider, then compare the cost