GLM-5.3-FLASH

Serving a model is physical work. For every output token, a machine must read the weights needed for that step, update its record of the conversation, and calculate the next token. Whether a chip can serve a model therefore depends on what limits that work, not only on its peak specification. The General Language Model (GLM) technical report includes a section that most artificial intelligence (AI) papers do not have: “Adapting GLM-5 to Chinese Chip Infrastructure.” It names seven domestic platforms, Huawei Ascend, Moore Threads, Hygon, Cambricon, Kunlunxin, MetaX, and Enflame, and describes ports from custom kernels to serving engines. The GLM-5.3-Flash launch makes the deployment claim concrete:

“Before release, we tested GLM-5.3-Flash anonymously as ox-alpha … It quickly became the most popular model of the week, with all of this traffic served on Chinese AI chips.”

Why memory bandwidth can be enough

Start with the three limits a serving machine faces. Memory capacity is how much of the model and its conversation state it can hold. Memory bandwidth is how many bytes it can move between memory and the processor each second. Compute is the rate at which it can perform arithmetic. A graphics processing unit (GPU) can be very strong at arithmetic and still wait on memory if each output token requires a large read. Z.ai describes its serving chips as primarily constrained by capacity and bandwidth. Flash shifts work toward that constraint: its Mixture-of-Experts (MoE) routing activates 18 billion of 320 billion parameters per token, 34 of 45 Kimi Delta Attention (KDA) layers keep fixed-size state, and sparse multi-head latent attention (MLA) reads 0.2% of a one-million-token context. Decoding, the token-by-token phase after a prompt has been read, therefore becomes more limited by moving the active weights than by doing arithmetic on them. An Ascend A3 card moves 3.2 terabytes per second, while the high-bandwidth NVIDIA H100 server module version moves 3.35 terabytes per second. Those cards are not identical, but a single decode stream limited mainly by memory bandwidth has nearly the same upper bound on both.

single-stream decode ceiling ≈ bandwidth ÷ bytes read for one token
                              (eight-bit floating point (FP8): 18 gigabytes of active weights per token)

Ascend A3 card       3.2 terabytes/second  →  ~178 tokens/second ceiling
NVIDIA H100 module   3.35 terabytes/second →  ~186 tokens/second ceiling
NVIDIA RTX 4090      1.0 terabytes/second  →  ~56 tokens/second
Mac M3 Ultra         0.8 terabytes/second  →  ~44 tokens/second

different levels of compute, nearly identical bandwidth ceilings

What the serving stack has to coordinate

Bandwidth establishes a limit. The serving stack must still keep chips busy without spending that bandwidth on unnecessary copies. Z.ai says it built its dedicated inference engine on Structured Generation Language (SGLang), a framework for running language models. Its listed techniques divide the work in several ways: intra-node tensor parallelism splits the linear-attention and language-model (LM) output-head calculation across chips in one machine; ReplaySSM replays state-space-model inputs instead of keeping every full state; and quantization stores numbers in fewer bits where that loss of precision is acceptable. The launch names W8A8, eight-bit weights and eight-bit activations, plus a cache that mixes eight-bit integers (INT8), eight-bit floating point (FP8), and bfloat16 (BF16), a 16-bit floating-point format. It also names Layer Split without publishing its mechanics. At cluster scale, Encode, Prefill, Decode (EPD) separates multimodal encoding, prompt prefill, and token-by-token decoding into worker pools that can be scheduled and scaled independently. Z.ai reports a 3× end-to-end serving improvement over its initial baseline on the same hardware, reaching per-token cost comparable to mainstream NVIDIA GPUs by its own measurement.

Why lower data movement broadens hardware support

Arithmetic intensity is the amount of floating-point work a program performs for each byte it moves. When that ratio is low, a processor spends more time waiting for data than calculating, so memory bandwidth matters more than peak arithmetic throughput. Flash has that shape during decoding: sparse routing reduces the weights read for each token, and its attention design keeps long-context state small. This does not make every device equivalent. The weights still have to fit in memory, and the serving software still has to support the chip. It does explain the published paths across NVIDIA through virtual large language model (vLLM) and SGLang, Advanced Micro Devices (AMD) through Radeon Open Compute (ROCm) recipes, Apple Silicon through a third-party MLX port, and central processing unit (CPU) offload rigs through KTransformers. Support for llama.cpp remains a pull request (PR) in progress rather than a released route. Before speed becomes the question, the model file must fit. The fitting table measures storage in gigabytes (GB), includes GPT-Generated Unified Format (GGUF) local-model files, and labels a machine's random-access memory (RAM). Use it: pick a quantization level, then compare it with each machine to see which ones can hold the weights.

Choose a city and trace one request