Today we are looking at Kimi Delta Attention (KDA), a form of linear attention designed to keep a running memory as a sequence grows. In late 2025, Moonshot published Kimi Linear with a specific research bet: a delta-rule update, which can replace a stale association instead of only adding another one, could preserve useful retrieval while cutting the key-value cache (KV cache) by ~75%. Eight months later, Moonshot put KDA into 69 of Kimi K3's 93 layers, the majority of the largest model ever trained. That is the useful test. A memory mechanism has to look good in a paper, then keep working through training, long sequences, and serving at model scale.
What does KDA store instead of a token list?
Start with the ordinary version of attention. Full attention keeps a key and value for every earlier token, then compares a new query with that growing collection. That gives it a direct route back to old details, but its cache grows with the conversation. KDA takes a different bargain. Each attention head keeps one fixed-size notebook, a 128×128 state, and updates that state for each new token. The delta rule first removes the stale memory associated with the current key, then decays each channel at its own learned rate, then writes the new value. A plain accumulator can only add, so conflicting facts pile up. KDA can revise an entry, so its fixed state can represent a newer association without carrying every old version forward. Press next event, then drag the channel-decay slider to see overwrite and fading as separate parts of the update.
One head's notebook, token by token
Click Step to feed tokens one at a time, then drag the sliders to change how the notebook remembers.
The notebook starts empty. Click Step to feed the first token.
- 1. decay: every cell fades by α (0.90)
- 2. erase: the old entry at this key is removed
- 3. write: the new value lands in its place
trace of one memory cell (row 3, column 4), the fact that keeps being revised:
total memory mass: 0.0 of a maximum of 144. A plain accumulator's mass only ever grows; the delta rule holds it near a fixed budget.
The toy above plays the update on one head's notebook. A real KDA layer runs 32 to 96 of these heads in parallel, each with its own keys, values, and learned decay rates, and K3 runs that across 69 layers. The fixed-size state is what makes the cost profile special: whether the conversation is 1,000 tokens or 1 million, the notebook is the same 128×128 grid, so KDA's memory bill never grows. The toy below shows the same rule on a stream of facts: click Next event, then drag the retention slider to separate overwriting from fading.
Watch the delta-rule memory
Click Next event, then drag retention to compare overwriting with accumulation.
plain accumulator, older linear attention
No entries yet.
KDA update rule
No entries yet.
What does that design buy?
The paper ties three returns to the architecture, data, and training recipe. First, it reports a ≈2.5× scaling-efficiency gain over K2. A floating-point operation (FLOP) is one unit of numerical compute, so scaling efficiency is the claim that a given training budget goes further than before. Second, K3 trains with a 1M-token context. One million tokens is far longer than a book, which is why the curriculum grows the window in stages instead of asking the model to handle that length from the first update. Third, serving can reuse work. KDA-aware prefix caching keeps a processed prefix available for a later request, and Moonshot reports a cache hit rate above 90% on coding workloads. That mechanism connects directly to the application programming interface (API) price of $0.30/M for cached input. When GLM-5.3-Flash adopted KDA for its linear layers, it was licensing this lineage. K3 shows that the lineage can run through most of a 93-layer model.