GLM-5.3-FLASH

The most reliable way to make a language model more capable has been to give it more parameters. A parameter is one learned number in the network. In a conventionaldense model, every parameter participates in every generated token. Ask for a haiku and the same vast network that can reason about a codebase reports for work.

That arrangement makes a model's capacity and its serving cost the same number. It is simple to build, but it is a poor fit for a task where each token needs only a small part of what the model knows. GLM-5.3-Flash separates the two with a Mixture of Experts(MoE): a large collection of specialist feed-forward networks and a small router that chooses which specialists run for the next token.

The hospital comparison is useful here. A hospital keeps many specialists on staff, but a receptionist does not send a sprained ankle to every department. The router reads the incoming token, scores the experts, and pages only the relevant ones. The experts it does not page do not run slowly. They do not run at all.

What that changes on the bill

Serving compute scales with active parameters, not the parameters waiting on the shelf. We can estimate the difference in floating-point operations (FLOPs), the small arithmetic steps a chip performs. A rough transformer estimate uses two operations per active parameter for each token:

dense 320B:  2 × 320B ≈ 640 GFLOP per token
FLASH MoE:   2 ×  18B ≈  36 GFLOP per token
                      →  17.8× less compute

Seventeen-point-eight times less compute per token, for a model with the same footprint on disk. That sentence deserves a slow read, because it splits an assumption most people carry: that a model's size and a model's cost are the same number. In a dense model they are. In an MoE they split apart. The capacity to know things lives in all 320B parameters on the shelf; the cost to think a token lives in the 18B that wake up. You buy the knowledge once (that's the disk space and the memory) and you rent the thinking by the token (that's the 18B). Z.ai pushed the same logic down-market: GLM-4.5 had 32B active across 92 layers; Flash nearly halves both (18B active, 45 layers) while adding vision, a bigger corpus (30T tokens) and, as the next chapters show, an attention scheme built to waste less.

Who decides? The router.

The router is a tiny classifier, a fraction of a percent of the model, that scores every expert for every token and pages the top 8. It is the single most load-bearing piece of arithmetic in the model, and Flash treats it that way: the router runs in 32-bit floating point (float32), a higher-precision number format than the rest of the network, because if the router's scores wobble, the wrong experts wake and quality drops in ways that are hard to debug. Its scoring uses sigmoid activation with a scheme called noaux_tc, a top-k routing method (it keeps the k highest-scoring experts) that keeps expert traffic stable while the model trains, so experts specialize smoothly instead of thrashing.

Routers are learned, not programmed. Nobody sat down and labeled expert #143 “particle physics.” At the start of training every expert is identical, and specialization emerges on its own: the router notices it keeps sending certain kinds of tokens to certain experts, and those experts drift toward being good at exactly those kinds. The resulting specializations are messier and better than human categories, closer to “things that follow closing parentheses in long code blocks” than to “the code expert.”

Click reveal the real choice, then tap a few experts to assemble your own team. This toy shows eight named stand-ins, not the real model's 288 specialists. The real experts do not have tidy human labels. The point is to make the selection dynamic visible: every extra expert adds compute, and a weak choice spends it without improving the answer.

Two failure modes make routers interesting to study. The first is collapse: if the router sends everything to one popular expert, that expert overloads while the other 287 employees sit idle, and the model quietly becomes a much smaller model. Labs fight this with load-balancing losses during training, a gentle penalty every time the traffic skews too far toward any one expert. The second is misrouting: if the router picks the wrong specialists, the token still gets processed, no error is raised, and quality silently drops. There is no stack trace in a neural network. Every MoE layer runs this election independently (42 MoE layers, every token, millions of elections per second in production), and the first 3 layers of Flash stay dense (no router at all) to build a solid common foundation before specialization begins.

MoE is the first part of Flash's cost model. GLM-4.5 already used experts, so routing alone does not explain the price. The next chapter turns to attention, the other major serving cost, and shows how Flash avoids re-reading a long conversation for every token.