A Transformer carries a residual stream from layer to layer. A residual stream is the running representation of the text: each layer reads it, adds its own result, and passes the combined signal forward. In a conventional stack, that is one shared lane. Early features must survive many later additions in the same lane, so a useful long-range signal can be repeatedly mixed with newer information before a later layer gets a chance to use it.
Qwen's answer is Gated Residual (GR), a change to that information path rather than to the attention calculation itself. It gives the model four residual branches to carry different kinds of state, then asks a learned gate how much to read from each branch. The motivation is modest and practical: create more ways for a useful signal to remain available across depth, while giving the model a way to suppress values that would otherwise grow too large. A 100+ layer stack makes that accumulation pressure easier to see, because even a small unwanted contribution has many later layers through which to amplify.
How does Qwen widen the residual path?
GR widens the original stream to four branches. Its read gate is elementwise, which means the model can apply a different control to every feature dimension instead of making one decision for the whole layer. Its write gate is a scalar for each branch, so the model can also regulate how strongly a layer writes back into each lane. The configuration uses a bottleneck rank of 320 for this control path. Rank here describes the width of the smaller learned representation used to form the gates, not the number of residual branches.
The mechanism leads to two consequences. First, different branches can preserve different pieces of the running representation, which gives later layers more than one route back to earlier information. Second, elementwise rescaling can suppress activation outliers, values that become unusually large and destabilize the next computations. Qwen reports that this also lets the residual state use 8-bit floating-point storage, or FP8. The smaller storage representation means less memory traffic when the model moves residual information between operations.
A branch is not another expert or a separate model. It is another route for the same kind of hidden representation, the changing list of numerical features that describes the tokens so far. The model does not assign a permanent job such as “recent syntax” or “old facts” to a given branch. Instead, the gates can learn different uses as training proceeds. That keeps the wider path flexible, while the gates prevent four routes from becoming four uncontrolled copies of the same noisy signal.
Read the drawing as a path comparison, not as a literal map of fixed branch jobs. The left side shows one stream receiving each layer's update. The right side shows several routes feeding a gate before the next layer reads the result. Qwen's analysis reports that one branch naturally emerged as a long-range path from the first attention layer to many middle and later layers. That observation is useful precisely because it was learned, rather than hard-coded as a special lane for distant context.
Move the four sliders one at a time. Start by raising the “deep past” lane, then lower the “fresh token” lane and watch the combined bar change. The controls make a plain point: a gate can favor one source without deleting the others. In the model, those choices are learned from each token's representation and happen at feature-level granularity, far more finely than this four-slider illustration.
Set the gate weights
Drag each lane to change its share of the residual stream.
The lane names and percentages in the mixer are explanatory, not values published for a trained Qwen checkpoint. What the public configuration does establish is the four-branch structure and the gated read and write behavior. That is enough to see the mechanism: widening offers additional routes, and gating stops wider routes from becoming an uncontrolled accumulation of activations.