Sixty of the seventy layers in MiMo-V2.6-Pro read only the newest 128 tokens. The other ten read the entire prompt. If you are sizing a long-context deployment, this is the first thing to understand, because that split decides both what the model can attend to and how the key-value cache grows with your prompt.
Full attention lets every token look at every earlier token, and its cost grows with the square of the sequence while the cache grows with its length. Most of what a layer does with its neighbors is local work anyway: finishing a word, matching a variable to its use, reading the last tool result. Sliding window attention (SWA) covers exactly that. Global attention (GA) layers are where distant facts meet.
How the layers alternate
The backbone is a stack of hybrid blocks. The shipped config puts a global-attention layer at positions 0, 7, 15, 23, 31, 39, 47, 55, 62, and 69 of 70, which leaves runs of six or seven sliding-window layers between them. The first block is special: global attention plus a dense feed-forward network instead of experts, which the report says stabilizes early representation learning.
Two details in the config sharpen the design. The window is 128 tokens, which is small for a model trained on million-token sequences, so after your first 128 tokens only those ten layers can still reach the beginning of the conversation. And the two attention types are not identical hardware: both use 128 query heads over 8 key-value heads, but keys are 192 numbers wide and values 128, and the rotary position encoding covers about a third of each head. The sliding-window layers also carry an attention-sink bias that the global layers do not.
The window and the cache
Move the query, switch the layer type, then drag the context length.
With 224 tokens behind the query, an SWA layer reads 128 of 224 of them. A GA layer always reads the whole prompt. An SWA layer reads at most the newest 128 tokens and stops, however long the conversation gets.
growing part 26.8 GB, fixed window buffers 20 MB
every layer stores keys and values for every token
Growing cache per context token: 25.6 KB in FP8. The window makes the difference a factor of 7, because only 10 of the 70 layers keep a full history. The ring buffers for the other 60 layers cost the same whether the prompt is 200 tokens or a million.
What the window does to the cache
Start with one layer. Every layer keeps keys and values for the tokens it can read, and those numbers stay in memory for the whole conversation. With 8 key-value heads at 192 and 128 numbers per head, one layer stores 2,560 numbers per token.
Now the window pays off. A sliding-window layer never reads past 128 tokens, so its cache is a ring buffer that never grows: 60 layers times 128 tokens, about 39 MB in BF16 whether the prompt is a paragraph or a repository. Only the 10 global layers accumulate history, 25,600 numbers per token, which is 51.2 KB in BF16 and 25.6 KB in FP8. Fill the full 1,048,576-token context and those ten layers hold 53.7 GB in BF16, against 375.8 GB if all 70 layers worked the same way.
You may be asking why the ring buffers are so cheap. A sliding-window layer can never read keys or values older than the window, so the new token overwrites the oldest slot instead of appending. The memory is bounded by the window, not by how long the conversation runs.
What the window costs
A window this narrow moves the whole burden of long-range binding onto 10 layers in 70. Anything older than 128 tokens is invisible to the rest of the stack. The model was trained to live with that: pre-training ran at 32K and 256K context, mid-training took it to 1M, and the reinforcement-learning run in chapter 5 trained at 1M directly.
The honest gap is in the report itself. It states the design and the window size, but publishes no ablation comparing 128 tokens with a wider window, and it defers the fuller backbone description to the earlier MiMo-V2-Flash report. You can verify the cache arithmetic from the config, and you cannot verify from this paper alone how much quality a larger window would buy back.
The window is half of the memory story. The other half is where the 1.02T parameters live and which of them actually run, which is the next chapter.