QWEN3.8-FLASH-NEXT

For ten years, large language model (LLM) engineers largely followed one rule: model weights live next to the compute. A weight is one learned value used during a model calculation. If the processor must wait for a weight, it cannot produce the next token, so the model's speed is set by memory movement as much as by arithmetic. This is why a model's weight files are usually kept in high-bandwidth memory (HBM), the fast memory beside the graphics processing unit (GPU), rather than in ordinary host random-access memory (RAM).

The diagram is a memory hierarchy. Memory close to the processor is fast and scarce; memory farther away is slower and larger. The chapter's rough comparison gives each step down the hierarchy about 30 times more capacity and about 30 times more waiting. Exact ratios depend on the hardware, but the direction matters: capacity is cheaper farther from the GPU, while a request that arrives late can leave expensive compute idle.

Reading a frequently used weight from HBM is like reading a book already open on your desk. Reading it from host RAM requires a trip over the connection between the CPU and GPU. That trip is acceptable only when the system knows what it will need early enough to hide the delay behind work that is already happening. A normal model weight is a poor fit: every generated token needs many of those values at once, in a pattern determined by the computation itself.

Qwen3.8-Flash-Next makes a narrower exception. It moves the parameters with an access pattern that can survive that trip: 51 billion parameters of n-gram embedding tables, about a quarter of the model, held in ordinary host RAM. These table values are looked up by address. They are not another dense matrix that every token must multiply through. That gives the system a chance to fetch a small selected piece early, rather than keeping the whole table permanently in accelerator memory.

What does an n-gram table remember?

An n-gram is a sequence of n consecutive tokens. Tokens are pieces of text rather than necessarily whole words, but “the captain” makes a useful two-token example, or bigram, and “the captain logged” makes a three-token example, or trigram. Before neural networks, language models used large tables to count how often such sequences occurred. Flash-Next uses the same local-context idea in a learned form.

Its 20 million bigram and trigram entries lead to embeddings, which are learned lists of numbers that represent a phrase or local pattern. When the recent tokens include “the captain,” layer 2 can add the stored representation for that phrase. The useful distinction is mechanical: a lookup chooses a row by address, while a matrix multiply combines large arrays of values. Qwen can add the table's capacity without adding another matrix multiplication to every token's main compute path.

Press “step the pipeline” four times. Follow the same phrase from host RAM to the accelerator: recent tokens identify the likely n-gram, the host starts a direct memory access transfer, and layer 2 reads the fetched embedding when it arrives. The sketch is deliberately small, but its timing is the design's central constraint.

What changes when memory becomes part of the design?

In this architecture, the memory hierarchy is part of the model. The n-gram table uses host RAM for phrase-level capacity, while the main computation stays near the GPU. DeepSeek-V4 explores the other side of the same systems question by storing its key-value (KV) cache, the record attention keeps for prior tokens, on disk between chats. The two approaches do different jobs, but both make the placement of model state a design choice with consequences for cost, latency, and hardware requirements.