KIMI K3

A mixture-of-experts (MoE) layer is the part of a model that sends each token to a small selection of specialist feed-forward networks instead of running every specialist. Kimi K3 has896 routed specialists. For each token, 16 wake up, while 2 shared experts process every token. That is how the model can hold a very large set of possible transformations without paying for all of them at once. It also creates a routing problem. If a few experts receive most of the traffic, they become the slow part of the distributed computation while rarely selected experts receive less useful training signal. An auxiliary balancing loss is the traditional response: add a second training objective that pushes the router toward equal load. K3's Stable LatentMoE instead builds the balance work into the routed path.

Why does the router need three moves?

First comes latent routing. A latent representation is a compressed internal vector, and a dimension is one coordinate in that vector. The routed path projects a token into a3584-dim latent space, half the model width, before the selected specialists process it and map it back. Moving specialist work through that smaller space reduces the traffic and weight movement that would otherwise grow with the number of selected experts. Second comes root mean square normalization (RMSNorm), which keeps the combined routed signal at a controlled scale before it returns to the full-width path. Third comes Sigmoid Tanh Unit Gated Linear Unit (SiTU-GLU). A Gated Linear Unit (GLU) multiplies two learned branches, one of which decides how much of the other to pass. SiTU-GLU uses sigmoid and tanh soft caps so small values behave like the familiar Swish Gated Linear Unit (SwiGLU) activation while unusually large values do not make both factors grow without bound (hidden_act: situ). The last move is the balance rule itself.

A quantile is a cut point in a sorted list. The 0.9 quantile, for example, is the value below which 90% of entries fall. A conventional router scores each token-expert pair and uses top-k selection, meaning it takes the k highest scores for that token. K3's Quantile Balancing (QB) also uses token-level choices, but it sets an expert-specific bias from the score quantile that matches that expert's target workload across the batch. The bias changes which expert is admitted to a route, while the original router scores still determine the mixture weights after selection. That separates load control from the expert contribution itself. QB therefore aims for an equal target load before the next batch is dispatched, rather than adding a penalty after imbalance appears. No auxiliary loss is used, and there is no fixed balancing step size to tune. Click rebalance the queue and watch the toy move its specialist loads toward an even queue.