MiMo-V2.6-Pro: 8 of 384

Every token in MiMo-V2.6-Pro is routed to 8 experts out of 384. A small router scores all 384 and keeps the top 8, and during reinforcement learning that router is frozen in place.

If you are going to serve, fine-tune, or reason about the cost of this model, the router is where the interesting tradeoff lives. It decides how much hardware you need per token, and the report runs the experiment that forced the freeze, in both directions, with numbers.

Where the trillion parameters live

Start with the arithmetic from chapter 1 and finish it. One expert is three matrices of 6144 by 2048 numbers, 37.7M parameters. One layer holds 384 experts, 14.5B parameters. Sixty-nine of the 70 layers use experts, so the experts alone are 1,000B of the total. The attention weights add 18.7B, the first layer's dense feed-forward adds 0.3B, and the embedding table and output head add 1.9B.

per token, per layer:
  chosen     8 experts x 37.7M            = 302M
  idle     376 experts x 37.7M            = 14.2B

per model:
  experts    69 layers x 384 x 37.7M       = 1000.2B
  attention  70 layers x 267.4M            =   18.7B
  layer 0    dense feed-forward            =    0.3B
  embedding  152,576 x 6144, twice         =    1.9B
  backbone total                           = 1.021T
  shipped checkpoint, all modules          = 1.024T

active per token                           =   41.7B  (published as 42B)

One MoE layer drawn at true count: 384 expert slots, 8 of them working on one token. The panel on the right is the accounting that produces the headline numbers, all of it arithmetic on the shipped config.

How a token picks its eight

The router runs one sigmoid score per expert, keeps the top 8, and normalizes their weights, so the layer output is a weighted sum. There are no shared experts that every token passes through, which makes the choice starker: a token either reaches an expert or it does not.

The config answers a natural question here: how the load stays balanced without a penalty term pushing it there. Routing uses a method called noaux_tc, chosen during pre-training specifically so no auxiliary balance loss fights the task loss for control of the router.

Be the router

Pick 8 experts yourself, then watch what RL does to the picking.

384 slots, one click adds or removes an expert

8 of 384 chosen: 302M parameters run this token, 14.2B sit idle in this layer. The router scores all 384 with a sigmoid and keeps the top 8, with no auxiliary loss pushing it toward balance.

load CV

0.7

peak load

5.5x

cold experts

with the router frozen, all three statistics stay flat through the run

The report prints these statistics at the start and at step 20 of the run, plus the restore test. This bench shows those snapshots and does not draw a curve between them.

What happened when the router was left trainable

Reinforcement learning updates billions of parameters from a reward signal, and a router sitting inside that gradient flow drifts. The report tracked three load statistics at layer 9 across its 384 experts and ran two copies of the same RL run, differing only in whether the router was frozen.

In the trainable copy, load balance collapsed over the first 20 steps. The coefficient of variation of expert load rose from 0.78 to 2.0, peak load went from 6 times the mean to 16 times, and the share of cold experts below a tenth of mean load went from 0.5% to 22%. A fifth of the model was effectively standing idle while a few experts took on everything.

The diagnosis is the part worth keeping. They restored only the router weights of the step-20 checkpoint to their pre-RL values, left everything else untouched, and load balance came back with benchmark performance unchanged. The experts were fine. The router had drifted. The frozen run holds all three statistics flat, around 0.7 variation, 5.5 times peak load, and 1% cold experts, while scores rise normally.

This is not only a serving statistic. The failure log in section 5.5 records a training out-of-memory crash where one expert-parallel rank received over 30 times the mean token load of a layer inside a single micro-batch, despite a balanced full batch. Unbalanced routing turns into memory spikes, and memory spikes end runs.

What freezing gives up

A frozen router cannot adapt its expert assignments to the new tasks RL teaches. That is the real cost, and the report does not measure it. What it shows instead is that router drift destroys the load balance the training stack depends on, and that holding the router at its pre-RL state avoids the damage.

If you fine-tune this model yourself, the practical recommendation is the same one the report follows: watch expert load, and treat the router as fixed unless you have a reason to move it. The mid-training optimizer switch in chapter 5 pulls in the same direction, preparing the weights for large-batch updates before RL begins.

receipts, every claim in this chapter, checked 2026-09-27