DEEPSEEK-V4

Before this chapter makes sense, we need one fact about how a transformer continues text. When the model generates token number 50,000, it does not run the first 49,999 tokens through every attention layer again. It consults a per-token memory computed when each token arrived. Those saved summaries are the key-value cache, or KV cache. A key describes how a past token can be found by the current query; a value carries what the model can use after it finds that token. Every past token contributes cache state in every attention layer, so more history means more entries. This is the basic long-context cost: the cache grows linearly with the conversation even before the model decides which part of it matters.

DeepSeek-V3.2 already reduced the reading cost with DeepSeek Sparse Attention, or DSA. Its lightning indexer is a small scoring network: it estimates which past entries are relevant to the current token, then sends only the top few thousand entries to close attention. DSA makes selection cheaper, but it still begins with a detailed cache for every token. V4 asks a prior question. If later attention will inspect only a small fraction of history closely, can the rest be represented at lower resolution from the moment it enters the cache? The answer is a two-tier design: Compressed Sparse Attention, or CSA, for selective detail, and Heavily Compressed Attention, or HCA, for a broad low-cost view.

The compression ladder

How CSA works, step by step

CSA does two jobs in sequence. First, it compresses. At the reader's scale, everym = 4 tokens become one cache entry, so four nearby pieces of history occupy the space of one. The paper does not use a plain average. It learns a weight for each token's contribution, which lets a cache entry retain more of the useful part of a four-token span (paper Eqs. 9 to 12). The implementation uses overlapping windows, but after accounting for that overlap the net compression rate is still 1/4. A paragraph summary is a reasonable analogy: a good summary gives more space to its important sentence, then stops trying to preserve every word verbatim.

Second, CSA selects. The lightning indexer runs over these compressed entries rather than individual tokens. For each new query it produces relevance scores, keeps the top 512 groups in Flash, or the top 1,024 in DeepSeek-V4-Pro, and sends only those groups to close attention. This is sparse attention after compression. The model searches one quarter as many groups, then pays detailed attention to a bounded shortlist. Compression lowers the cost of representing history; selection lowers the cost of reading that representation.

Why add HCA

HCA, or Heavily Compressed Attention, makes a different tradeoff. It mergesm′ = 128 consecutive tokens into one entry, 32 times coarser than CSA, then attends to every compressed entry. There is no top-k selector and no indexer on this path. At one million, or 1M, input tokens, 128-way compression leaves about 8,000 entries, which is small enough to scan densely. HCA therefore preserves a cheap outline of the whole history. CSA supplies selected detail, while HCA makes it possible for the model to keep a global view without paying one-million-entry attention cost.

What the shipped schedule does

The paper's §4.2.1 settings and the shipped compress_ratios array show the layer-by-layer schedule. DeepSeek-V4-Flash gives layers 0 and 1 pure sliding-window attention: each token sees its 128 nearest neighbors at full resolution. DeepSeek-V4-Pro starts with two HCA layers instead. After that, CSA and HCA alternate through the stack. That alternation is a dial between local detail and broad coverage. Drag the controls in the lab below to compare the number of retained entries as the context grows.

• Shared-KV Multi-Query Attention (MQA): ordinary multi-head attention often keeps separate keys and values for each head. MQA keeps one shared copy that all heads read while retaining separate queries. V4 goes further because a compressed entry serves as both the key and the value, reducing what must be stored for each layer.
• Low-rank queries: a low-rank representation first projects a wide hidden state into a shorter latent vector. Flash uses 1,024 numbers and Pro uses 1,536. The attention heads and the indexer derive their queries from that shared latent, which avoids rebuilding a separate full-width query representation for every consumer (paper Eqs. 13 to 18).
• Grouped output projection: 64 heads × 512 dimensions is a wide attention result to return to the model. V4 divides the heads into 8 groups, or 16 in Pro, compresses each group to 1,024 numbers, then merges the groups. That two-stage route avoids one enormous projection while still returning a full model-width result (§2.3.1).