GLM-5.3-Flash: The Memory Squeeze

Long context has a memory cost as well as a compute cost. In an attention layer, every past token produces a key, used to measure relevance, and a value, the information recovered after a match. A server stores those vectors in a key-value (KV) cache after it reads the prompt. Later tokens can reuse the stored vectors instead of rebuilding the earlier conversation. The tradeoff is that each active conversation owns a cache entry for every token in every layer that needs one. Flash separates summarization from lookup: 34 Kimi Delta Attention (KDA) layers hold fixed state, while 11 sparse multi-head latent attention (MLA) layers keep smaller token-specific records. In the model name, GLM means General Language Model. Reading the supplied prompt and constructing its first cache is called prefill. Generation after that is called decoding. During decoding, the cache lets the model compare a new token with the earlier sequence without running all earlier tokens through every layer again. The saved work is useful, but the saved vectors remain in memory until that sequence finishes.

Start with a conventional 45-layer dense model using 8-bit floating point (FP8). Every token needs both a key and a value in each layer. With 8 KV heads of 128 dimensions, that is two vectors × 45 layers × 8 heads × 128 numbers, or about 92 kilobytes (KB) per token at one byte per FP8 number. A thousand tokens add about 92 megabytes (MB). The two accounts for the key and value pair; the remaining factors account for repeated layers, separate attention heads, and the coordinates in each vector. This is a calculation for one sequence's cache only. It does not include model weights or other memory a serving system needs. At 1 million tokens, the same cache arithmetic reaches about 92 gigabytes (GB):

100K-token chat:    ~9.2  GB of cache
1M-token session:   ~92   GB of cache  before the model's weights!
                                     (weights themselves: 320+ GB)

At 1 million tokens, the cache alone is about 92 GB before the model's 320+ GB of weights. Unlike weights, which are shared across requests, a KV cache grows with each sequence and is separate for each active conversation. The question is not only how many numbers each token needs. It is also how many layers must retain those numbers at all. A busy server may keep many sequences active while they wait for the next output token. Each sequence brings its own growing cache, so cache capacity determines both the maximum context length and how many requests can share the machine at the same time.

Why do only 11 layers keep token-level notes?

The 34 KDA layers hold a fixed 71.3 MB state, so they add zero per-token growth. Only the 11 sparse MLA layers maintain a per-token cache. MLA compresses each token's key and value information into one latent vector, where latent means an internal compact representation, rather than storing full vectors for every head. Flash's latent has 512 numbers. It also uses No Positional Encoding (NoPE), so there is no separate positional tail. The lower-bound calculation is 11 × 512 = 5,632 numbers per token, or about 5.5 KB at FP8 versus about 92 KB for the dense layout. That is roughly twenty-fold smaller before counting the indexer's separate keys. The distinction is architectural: KDA carries one evolving summary for a layer, while sparse MLA retains a compact record for each token that a later query may need to retrieve. The model therefore spends its token-level memory only where it expects exact retrieval to matter.

Selecting tokens also has a memory cost. The lightning indexer retains keys for every token it may score. Flash's IndexPool compresses 4 indexer key vectors into 1 through weighted pooling, which quarters that part of the indexer storage. The configuration documents the 4-to-1 pooling, but not the exact cache persistence for every indexer layer. That missing detail matters because a calculation can be exact about the 512-number MLA latent and still be incomplete about the indexer. Treat the figures in this chapter as a transparent floor, not a claim that every byte on a server has been counted.

Why does numerical precision change the memory bill?

FP8 uses one byte per number. Bfloat16 (BF16) uses two, so the cache doubles in size when all else is equal. Flash serves its cache and weights in FP8 on the published setup. Z.ai reports a 4.44× smaller KV cache than GLM-5.3. A bottom-up estimate from the configuration ranges from 4.6× to 8× depending on which indexer storage is included. The exact indexer persistence is not published, so the aggregate 4.44× comparison is the firmer public claim. Precision changes the byte count, not the number of cached tokens or layers. It therefore multiplies the architectural saving rather than replacing it: fewer token-level records first, fewer bytes per remaining record second. The calculator compares that cache requirement with random-access memory (RAM) capacities. It also uses tensor parallelism (TP), a way to split model computation across processors, when it names a serving configuration.

Compare the multi-head attention (MHA) baseline, then drag the context slider

Compare KV cache sizes

Choose a design, then drag context length to compare memory use.

context length:1M

5.8 GBKV cache for one 1M-token conversation

RTX 4090

Mac 64GB

H200

Ascend node ×8

1,048,576 tokens × 5.5 KB per token = 5.8 GB. only 11 of 45 layers store per-token KV (NoPE uses a 512-dimension latent, with no RoPE tail). The 34 KDA layers do not grow with context: they keep a fixed 71.3 MB state, derived from config.json. This is the mechanism behind Z.ai's stated 4.44× smaller KV cache versus GLM-5.3. At 1M context, dense-style memory would need much more capacity for the cache.

Per-token estimates use config.json dimensions (11 × 512-dimension latent). Lightning- indexer keys add about 45 KB per token after IndexPool compression; exact persistence is undocumented, so treat these totals as a floor. vLLM reports pools of 14.92M FP8 tokens versus 8.87M BF16 at TP=4.

At serving scale, cache capacity determines how many conversations can remain active at once. In the published serving recipe, tensor parallelism (TP) is set to 4 on a GB200 system. Its FP8 pool holds 14.92 million KV tokens, or about 114 concurrent 128,000-token conversations. That is 1.68× the BF16 capacity. At a 1-million-token context, the lower-bound sparse MLA cache is about 5.9 GB plus 71 MB of fixed KDA state. Model weights and indexer keys still consume memory, so this is a cache comparison rather than a full hardware fit calculation. The important serving consequence is not that every request becomes equally cheap. It is that a fixed pool can hold more active context before the server must reject, shorten, or queue another request.

One caveat applies to every derived total here. The per-token figures come fromconfig.json; Z.ai publishes only the aggregate 4.44× result, and the indexer's exact per-layer caching is not documented. The 5.5 KB and 5.9 GB figures are therefore a floor for the sparse MLA cache, not a final system total. Even with that uncertainty, the expected scale at 1 million tokens is tens of gigabytes rather than terabytes.

receipts, every claim in this chapter, checked 2026-08-31