QWEN3.8-FLASH-NEXT

A hybrid sequence model gives different layers different jobs. Full attention lets the current token compare itself directly with earlier tokens, which is useful when it must retrieve one specific detail from far back in the context. Gated DeltaNet (GDN), by contrast, carries a fixed-size running state that summarizes the past as it goes. Qwen3.8-Flash-Next uses three GDN layers for every one retrieval layer, then makes that retrieval layer Qwen Sparse Attention (QSA). The report describes this as replacing the full-attention slots with QSA at continued-pretraining time. The arrangement asks most layers to remember a compact history and reserves the costlier search for regular checkpoints.

QSA still needs a way to decide where to search. Its key choice is the unit it scores. Rather than assign a score to every previous token, it first groups nearby positions into micro-blocks, small contiguous sections of context. The lightweight indexer scores those sections, selects promising regions, and lets attention examine the selected material more closely. On the documented configuration, the indexer budget is 2,048 tokens, or 512 blocks, in each QSA layer. A fixed search budget matters most when the context becomes very long: one million tokens presents one million possible positions, not a small constant.

Why does the index need its own budget?

Sparse attention has two jobs. It must retrieve useful context, and it must choose that context cheaply enough to be worth using. The token-level indexers discussed around GLM's lightning indexer and DeepSeek's earlier sparse attention score every past token in a preliminary pass. Each score can be cheap, but the count grows directly with context length. At 128K tokens, that preliminary search can be a manageable line item. At one million tokens, deciding where to look can consume a meaningful share of the work saved by looking at fewer positions.

QSA changes the count before it starts scoring. If a micro-block stands for several nearby tokens, the indexer sees fewer candidates than the full sequence contains. It does not make long-context retrieval free. A larger block can contain the detail you need without saying exactly where it is, while a smaller block creates more entries to score. QSA chooses a middle path: compress the candidate list at the block level, then spend precise attention only on the regions that survive that first pass.

The savings matter in two different serving phases. Prefill is the first pass over the prompt, when the model builds its context state. Decode is the next-token loop that produces the response. At a one-million-token context, Qwen reports up to 7.6 times faster prefill and 4.9 times faster decode for the QSA attention kernel than for full attention. Those are vendor-reported kernel measurements, not an end-to-end promise for every application. They still explain the design pressure: a long prompt makes the initial scan expensive, and every generated token has to retrieve from that same long history again.

QSA also pays a deliberate cost for its hybrid setting. It builds its compressed index inside each sparse-attention layer instead of assuming that another layer's chosen blocks remain relevant. Rebuilding an index takes work, but a GDN layer and a QSA layer represent context differently. The consequence is an index that is tailored to the retrieval layer using it, rather than a reused shortcut whose relevance the model has not checked.

Drag the micro-block slider from 4 to 128 tokens. The 4K label in the toy means a 4,096-token context. Watch the number of compressed entries fall as blocks get larger, then compare that saved index work with the toy's deliberately softer recall label. This is a teaching model of the tradeoff, not a published QSA throughput measurement.

There is one more detail behind the word “Qwen” in QSA. The model compresses the sequence independently inside each QSA layer instead of reusing one layer's index in another. That repeated work costs something, but it avoids assuming that an index made beside a GDN layer will remain useful for the next attention layer. The consequence is a sparse retrieval path designed around this hybrid layout, rather than a sparse-attention component pasted onto a different kind of model.