DEEPSEEK-V4

Chapter 2 introduced compressed memory. A key-value cache is the saved attention state that lets a language model continue a long prompt without reprocessing every earlier token from scratch. DeepSeek-V4's sharpest long-context claim concerns that state: at a 1M-token context, V4 keeps 10% of DeepSeek-V3.2's KV cache, or 7% for Flash. The paper also compares the design with a BF16 Grouped-Query Attention baseline, where eight key-value heads serve many query heads, and reports roughly 2% of that cache. This is a reference comparison, not a claim that every deployed Grouped-Query Attention model has exactly the same footprint. It gives us a useful scale: at one million tokens, saving a few bytes per remembered token becomes a large server capacity decision.

That is why a 50× smaller reference cache can affect prices. Serving hardware has a fixed pool of fast memory shared by model weights and the conversations currently in flight. Use 1 TB as a round-number illustration, not a published V4 hardware specification. If a conversation's cache takes half as many bytes, the same memory pool can hold twice as many such conversations before cache capacity becomes the limit. Each request then consumes a smaller share of the machine that must remain allocated while it generates. Cache size is not a cosmetic implementation detail. It limits concurrency, prefill reuse, and eventually the portion of the invoice that comes from keeping a conversation warm.

Where the bytes go

The paper's accounting in §2.3.4 combines four savings rather than one magic ratio. Read the list as a chain: compression reduces the number of entries, lower precision reduces bytes in each entry, sparse selection reduces the entries read for a query, and a smaller indexer reduces the overhead of finding them. Drag the cache-size controls below after the list to see why each layer of savings matters more at a million tokens.

1. The CSA/HCA compression itself (chapter 2): Compressed Sparse Attention keeps one entry for each 1/4-sized CSA group, while Heavily Compressed Attention keeps one for each 1/128-sized HCA group. The layers interleave, so the combined schedule reaches roughly 2% of the classic reference cache. Fewer stored entries also leave fewer candidates for later attention work.
2. Mixed-precision storage: the 64 Rotary Positional Embedding dimensions, which carry position information, stay in BF16 because positional rounding is risky. The remaining dimensions use FP8, one byte instead of two. The representation therefore saves nearly another half of the storage without treating every part of the vector as equally safe to round.
3. A smaller top-k (512, vs V3.2's 2,048): top-k means retaining only the highest scoring cache entries for close attention. Flash reads 4× fewer selected entries per token, so the sparse path has a smaller working set as well as a smaller stored cache.
4. The indexer runs in FP4, so the small scout network that scores the cache uses four-bit values too. It can search the compressed history without turning the search step into a full-precision memory bill.

Why retain a cache on disk

The paper documents an additional tier in §3.5.2: compressed cache entries for a finished shared prefix can be persisted on an SSD. A prefix is the common beginning of requests, such as a long system prompt or a document that several users ask about. When a later request matches it, the server can read back the compressed CSA and HCA entries instead of computing that old portion again. The recent sliding-window entries remain harder to persist because they are uncompressed and ~8× the volume of the compressed entries. DeepSeek describes three choices for them: store everything, checkpoint every p tokens, or store nothing and recompute from the compressed entries. Each choice trades disk space against recomputation time, which is the same tradeoff a storage system makes.