KIMI K3

A layer is one stage of a model's computation. It reads the representation passed from the stage below, changes it, and hands a new representation upward. More layers give a model more chances to transform a difficult signal, but they also create distance. Kimi K3 has 93 layers. If layer 12 finds a useful feature, layer 40 normally receives it only after it has passed through every layer in between. A residual connection helps by adding an earlier representation back into the current path. That creates a short route for information and gradients, which makes deep stacks easier to train. It still has one limitation: the current layer receives a blended result. It cannot directly ask whether the representation from layer 12, layer 24, or layer 38 is the most useful source for this stage of processing. Features can also change role as depth grows. An early layer may encode a local pattern, a middle layer may combine it with context, and a later layer may need one of those versions rather than the final blend alone. K3's answer isAttention Residuals (AttnRes), a depth-wise lookup mechanism that lets a later layer choose from earlier representations instead of receiving only their accumulated result.

Why give depth more than one skip lane?

Attention is a weighted lookup. A query expresses what information is needed, keys describe the available candidates, and values are the information returned. Sequence attention uses this process across token positions. AttnRes applies the same idea across model depth. Each receiving layer owns a learned pseudo-query. It is pseudo because it is a model parameter rather than a vector calculated from the current token. The pseudo-query scores the token embedding and earlier outputs, then gives more weight to the representations that fit its learned preference. Root mean square normalization (RMSNorm) puts those candidates on a comparable scale before the scores are made, so a layer with unusually large values does not win merely because of its magnitude. The plain-language effect is a direct retrieval lane through depth. A late layer can use a lower-layer feature without hoping that feature survived unchanged through the whole stack. The full version of this idea would keep every earlier layer output available. That is affordable in arithmetic for fewer than 100 layers, but keeping every output alive costs memory and communication. K3 therefore uses Block AttnRes. The config sets attn_res_block_size: 12: layers inside a block contribute to one running block representation, and later blocks attend to those summaries. The paper describes eight 12-layer blocks, with a partial final block and the embedding as an additional source. This changes the stored history from every layer to a smaller set of block summaries, which bounds the extra state while preserving a choice among earlier depth regions. Drag the slider to change how strongly layer 40 borrows from layer 12, then watch the skip lane widen. It shows the central tradeoff: wider access to earlier work costs more state, so the model groups that work before it passes it forward.