The cache work is the spine of this model. Four smaller systems sit around it: vision input, Engram lookup, DSpark drafting, and a cheaper residual path.
Each one reduces traffic or moves work to a cheaper place. The details are different. The serving motive is shared.
Seeing: a vision encoder trained from scratch
DeepSeek-ViT is trained from scratch rather than added to a finished text checkpoint. It uses two-dimensional rotary positions, so it can accept different image sizes.
A 3×3 pixel-unshuffle folds each neighborhood into channels and cuts visual token count by nine. That keeps a code-editor screenshot from taking over a long context.
Image and text tokens receive separate load-balancing biases in the expert router. The pre-training mix is seven parts text to one part multimodal, and the long-context loader reads each image once.
Remembering: a table that is read, not multiplied
Engram separates memorization from the backbone's matrix work. It is a 196B-parameter set of embedding tables at layers 1 and 14.
For each token, it hashes every 2, 3 and 4-token group containing that token. Eight hash heads per order address tables of roughly 16 million rows, then the retrieved rows are gated back into the representation.
The address depends on the token sequence, not the current hidden state. The report says first-module background prefetch overlaps the first Transformer block. The model reads a small set of rows instead of multiplying all 196B Engram parameters for every token.
Engram rows and projections use FP8. DeepSeek uses a momentum update with Sinkhorn balancing for the tables, token embedding, and prediction head, avoiding Adam's two extra state tensors for a very large table.
Trace a phrase into Engram
Use a real phrase, form its token groups, then click one group and fetch the row it addresses.
Start with the phrase, then ask Engram to form every pair, triple and quadruple.
Writing faster: a drafter that knows when to stop
DSpark drafts ahead of the backbone. Three transformer blocks with a 128-token window propose logits for five positions in one pass.
A small Markov head links those five candidates, and a confidence head estimates how far they are likely to survive verification. The draft is a sequence, not five unrelated guesses.
The serving scheduler can verify one candidate under load or all five when the machine has room. DSpark is trained after the backbone, with the backbone frozen, and stays aligned during post-training without gradients flowing back into the model.
Verify a five-word DSpark draft
Pick a draft, then verify one candidate at a time. A rejection returns control to the backbone.
backbone has already written
DSpark proposes five next words in one draft
accepted draft prefix
0
words the backbone did not write alone
draft window
5
the released DSpark proposal length
verification path
checking
toy acceptance, not a released output
Keeping information: four notes, one trip
Think of the residual stream as a notepad passed between layers. Most models pass one page. This model passes four: for one token that could be the name, the date, the tone, and the plan, kept side by side.
The old way needed three trips over that notepad for every block: update the pages, write the blending recipe for this block, then blend the input. The blending step had to wait for the recipe, so nothing could fuse.
V4.1 shifts the recipe by one block: each block blends with the recipe the previous block already made. No waiting, so one kernel reads the four notes once, blends, predicts the next recipe, and writes once. The report puts the saving at about half the memory traffic, with no measured quality loss.