DEEPSEEK-V4

Neural-network weights are stored as floating-point numbers. A floating-point format assigns bits to a sign, an exponent that sets the number's rough scale, like the 10 in 10³, and a mantissa that records the finer detail within that scale. The format name is a budget. Bfloat16, written BF16, uses 16 bits: 8 for the exponent and 7 for the mantissa. FP8 uses 8 bits. In the E4M3 flavor DeepSeek uses for most compute, that means 4 exponent bits and 3 mantissa bits. FP4 uses 4 bits. In its E2M1 flavor, it has 2 exponent bits and 1 mantissa bit, so each number has one of just 16 possible values and neighboring representable values can be far apart. Fewer bits mean less memory and less data moving across a machine. The cost is that the model must learn to tolerate much rougher rounding.

Raw E2M1 would be too coarse if one scale factor had to cover every weight in a large matrix. DeepSeek instead uses microscaling FP4, or MXFP4. It divides weights into small blocks of 32 numbers and gives each block a shared scale factor stored at higher precision. A block with large values can use a large scale; a block with small values can use a small one. The 4-bit values then express their position within that local range. Scientific notation is the useful analogy: share the exponent for a small group, then spend the scarce bits on the digits that distinguish values inside the group.

DeepSeek's recent releases have made precision a design choice rather than a final deployment shortcut. V3 trained in FP8 when many training stacks still depended on 16-bit values. V4 goes further in two places: it stores and trains routed experts in FP4, and it uses an FP4 lightning indexer to scan context. The shipped configuration states the first choice directly asexpert_dtype: fp4. The distinction matters because these are not post-hoc weights squeezed after training. The model sees the low-precision representation while it learns how to use it.

Why FP4 works for experts specifically

A Mixture-of-Experts model routes each token through a small subset of its many feed-forward experts. DeepSeek-V4-Pro has 384 routed experts and Flash has 256; both select six routed experts for a token. These experts are wide matrix multiplications with no softmax inside them, so many small rounding errors can average across the computation. Attention makes a different kind of calculation. It turns similarity scores into probabilities with softmax, where a small numerical shift can redirect a large share of attention. V4 therefore spends FP4 on the large expert matrices, while keeping sensitive attention and routing paths at FP8 or BF16. DeepSeek notes that current GPUs run FP4-times-FP8 at the same speed as FP8-times-FP8. The immediate gain is memory traffic and storage; the forward-looking hardware case is another third of efficiency when future silicon makes the narrower format faster too.