Efficiency only helps if a training run reaches the end. At 1.6T parameters, a model does not necessarily fail with a clean error message. A loss curve can drift upward, then spike after thousands of hours of otherwise useful work. An hour-9,000 failure can erase an eight-figure compute commitment. V4's §2.2 through §3.4 stability sections describe how DeepSeek tries to prevent that outcome: constrain how signals move through layers, shape how matrices are updated, and watch the routing decisions that make a Mixture-of-Experts model economical. We will take those three safeguards one at a time.
How mHC limits signal growth
Manifold-Constrained Hyper-Connections, shortened to mHC, begins with the residual stream. A residual stream is the running collection of features a transformer carries from layer to layer: each layer reads it, computes a refinement, and adds that refinement back. Earlier hyper-connections, an idea from a separate 2025 paper DeepSeek cites, widen one stream into n = 4 parallel streams. Each layer learns three small mappings, one that mixes the 4 streams into its input, a 4×4 mapping that carries the streams between layers, and one that writes the layer's result back. The layer itself still receives an ordinary model-width input. The extra streams give the residual path more ways to carry information without rewriting the interior of every transformer block.
The risk is repeated amplification. A free 4×4 mapping can stretch a signal. Apply a toy stretch of 1.2 through 40 layers and an activation can grow roughly 1,700-fold. mHC constrains the between-layer mapping to the Birkhoff polytope, the set of doubly stochastic matrices: every row sums to 1, every column sums to 1, and no entry is negative. Such a matrix mixes by weighted averaging rather than arbitrary amplification. DeepSeek obtains it with 20 Sinkhorn-Knopp iterations, repeatedly rescaling rows and columns until they meet those sums. The paper's constraint bounds the residual transformation's spectral norm at 1, so composing many of these mappings remains non-expansive. Input and output mappings use bounded, non-negative sigmoid forms as a second guard. The report says its fused implementation keeps mHC's wall-time overhead to 6.7% of an overlapped pipeline stage (§2.2).
How Muon shapes an update
An optimizer turns a gradient, the training signal that says how a weight should change, into an actual weight update. AdamW, the common baseline, rescales each value using its own running statistics. Muon treats a matrix update as one geometric object. It approximately orthogonalizes the update, bringing its singular values, the amounts it stretches along each independent direction, toward all 1s. The goal is to avoid a few directions receiving a much larger push than the rest. A full singular-value decomposition would be too slow in every training step, so Muon uses the Newton-Schulz iteration, a sequence of matrix multiplications that moves singular values toward 1. V4 uses a hybrid 10-step schedule in Algorithm 1. The first 8 steps use the more aggressive coefficients (3.4445, −4.7750, 2.0315) to converge quickly. The final 2 use (2, −1.5, 0.5) to settle the values precisely at 1. Muon handles most modules. Embeddings, the prediction head, root mean square normalization weights, and mHC's static biases and gates remain on AdamW. The attention design supplies one more safeguard. Query-key root mean square normalization, or QK RMSNorm, normalizes queries and KV entries before their dot products, which prevents attention logits from exploding. DeepSeek therefore reports that it can omit the QK-Clip patch used in other Muon settings (§2.4).
How the MoE avoids routing spikes
A Mixture-of-Experts router decides which experts receive each token. DeepSeek found that routing and expert activations could reinforce one another: a popular expert grows large, looks even more attractive to the router, then receives still more traffic. V4 uses two countermeasures. Anticipatory routing computes the routing choices from an earlier copy of the parameters, then applies them to the current backbone update. That breaks the synchronous feedback loop, but adds about 20% wall-clock work, so a detector turns it on for a spike and off after the run settles. SwiGLU clamping provides a local limit. It clamps the activation's linear branch to [−10, 10] and caps its gate at 10, preventing one extreme token from creating an unbounded expert activation. These are narrow engineering controls with a clear purpose: keep a rare routing outlier from becoming a run-wide failure (§4.2.3).