Model architectures, explained slowly
Dims & Sums
We read the technical reports, draw the mechanisms, and work through the arithmetic. Then you can move the controls and see what changes.
On the table today
A 1.02T-parameter model that reads text, images, video, and audio, then trains every skill in one mixed reinforcement-learning run
MIMO-V2.6-PRO
MiMo-V2.6-Pro interleaves 60 sliding-window layers with 10 global-attention layers, routes each token to 8 of 384 experts, and takes text, images, video, and audio in one sequence. Xiaomi released the weights under MIT together with the RL environments and training framework used to build it. Read the chapters in order, then drag the controls to see where the numbers change.
read chapter 1:Meet MiMo-V2.6-Pro
Other models on the table
each one makes a different tradeoff between capability, cost, and context
GLM-5.3-FLASH
GLM-5.3-Flash pairs 320B total parameters with 18B active parameters, mixes two cheaper attention paths, and ships native FP8 weights under the MIT license. Read the chapters in order, then use the controls to test the tradeoffs yourself.
start,Meet the Flash
KIMI K3
Kimi K3 scales Kimi Delta Attention, Attention Residuals, and LatentMoE to 2.8T total parameters. The technical report describes about 2.5 times more capability per unit of compute than K2; the chapters separate that claim into mechanisms, measurements, and open questions.
start,The 2.8T Statement
DEEPSEEK-V4.1-FLASH
DeepSeek-V4.1-Flash splits its 40 layers into a causal encoder and a decoder, projects the decoder global cache from the encoder hidden state, and shares one compressed cache across layers. The reported global key-value footprint is 890 bytes per token, about a quarter of DeepSeek-V4-Flash and an eighth of its persisted cache. These chapters follow the cache from the prefill pass to the SSD budget, then to the scoreboard.
start,The Input Bill
QWEN3.8-FLASH-NEXT
Qwen3.8-Flash-Next relocates 51B n-gram parameters to host memory, combines Gated DeltaNet with Qwen Sparse Attention, and evaluates each design change against quality, training cost, and stability. The reported result targets 397B-parameter flagship quality with roughly one ninth of its training compute.
start,The 1/9 Recipe