The usual fix is an auxiliary load-balancing loss added to the training objective, which penalizes the model when expert loads drift apart. The penalty’s gradient reaches the same weights as the language-modeling loss, though, so if it is set too large it hurts model performance, as the report says.

A bias term for each expert

DeepSeek-V3 moves most of the balancing out of the loss. Each expert gets a bias term that is added to its affinity score, and the top-K experts for a token are picked from those biased scores. The bias is used only for that choice: the gating value that weights the expert’s output still comes from the original score.

The bias changes once per training step. DeepSeek monitors the expert load over the whole batch of the step, and at the end of it lowers the bias of every overloaded expert by a fixed amount and raises the bias of every underloaded one by the same amount. An overloaded expert then needs a higher raw score to make a token’s top-K, and an underused one starts to win tokens that sat near the cutoff. The report sets that amount to 0.001 for the first 14.3T training tokens and to 0.0 for the remaining 500B.

Because the bias never enters the loss, it adds no gradient of its own. But it does not remove every balancing term. The report keeps a complementary sequence-wise balance loss to prevent extreme imbalance within any single sequence, with its weight set to an extremely small 0.0001. So “auxiliary-loss-free” in the README means the bias does the balancing work instead of a large loss term, and the README says this keeps down the performance degradation that comes from encouraging load balancing.

Load stayed even enough that DeepSeek-V3 did not drop any tokens during training. The report also tested the bias method against baselines balanced purely by auxiliary losses, at 15.7B and 228.7B total parameters, and it did better on most of the evaluation benchmarks. On the Pile test set, a 16B model trained with the bias showed greater expert specialization by domain than its auxiliary-loss counterpart.

Multi-token prediction

DeepSeek-V3 also trains with a Multi-Token Prediction (MTP) objective. A standard language model is trained to predict only the next token, at position $t+1$. DeepSeek-V3 adds one MTP module after the main model, so each position is trained to predict the next two tokens. The module is a single Transformer block that shares the embedding layer and the output head with the main model. It predicts sequentially: it takes the main model’s representation of the current token together with the embedding of the next one, which keeps the complete causal chain at each prediction depth.

The report gives two reasons this could help. The extra target densifies the training signal and may improve data efficiency, and the model may learn to pre-plan its representations for tokens further ahead. In its ablations, adding the module improved the main model on most of the evaluation benchmarks, at both 15.7B and 228.7B total parameters. The module ships with the weights, which is why the files on Hugging Face total 685B: 671B for the main model and 14B for the MTP module.

At inference the module is optional. The main model works normally without it, and the module can also be reused for speculative decoding, where it proposes the second token and the main model checks it. Speculative decoding usually pairs a large model with a separately trained small draft model; here the draft comes with the weights. The report measures the acceptance rate of the second token at 85% to 90% across generation topics, and puts the resulting decoding speed at 1.8 times the tokens per second. A runtime has to support the module to use it that way, and the README says MTP support is currently under active development within the community.

This post is put together from the DeepSeek-V3 README and technical report.

為了提升推論效率，官方說明他們採用了 Multi-head Latent Attention（MLA）架構（“To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2."）。這項設計的核心目標，在於緩解自回歸生成時持續膨脹的鍵值快取（KV cache）記憶體佔用。

KV 快取的記憶體瓶頸

大型語言模型在自回歸生成文字的過程中，每計算一個新 token，注意力機制都需要回頭查詢先前所有 token 的 Key 與 Value。為了避免重複運算，推論系統會把先前步驟產生的鍵值向量暫存在顯存當中，也就是常見的 KV 快取。

在標準的多頭注意力（Multi-Head Attention，MHA）當中，每個注意力頭都各自擁有一組 Key 與 Value 向量。當模型層數增加、上下文長度拉長，或是同時處理多個請求時，快取所消耗的顯存容量會迅速累積，變成限制批次大小和序列長度的主要瓶頸（來自 V2 論文）。常見的分組查詢注意力（GQA）透過讓多個 Query 頭共用同一組 Key 與 Value 來縮減體積，不過每一組存的仍然是完整的 Key 與 Value，這類做法雖然讓快取變小，效果卻比不上 MHA（來自 V2 論文）。

壓縮至低維潛在向量

但是 MLA 採取了不同的處理方式，不再只是減少 Key 與 Value 的組數，而是引入低秩壓縮機制。在計算注意力之前，模型先透過投影矩陣，將隱藏狀態壓縮成維度小得多的低維潛在向量（latent vector）。

到了自回歸生成階段，顯存中不必再為每個注意力頭保留完整的 Key 與 Value，而是只需要儲存這個緊湊的潛在向量，另外再加上一小段帶有位置資訊、所有頭共用的 Key。當新的 token 需要計算注意力分數時，直覺的做法是把潛在向量還原成每個頭的 Key 與 Value。不過 V2 的論文則說明，還原用的矩陣可以事先併進 Query 和輸出的矩陣裡，所以推論時其實不必把 Key 和 Value 重新算出來，也就省下了這段重複的計算。

實際推論時的影響

這種結構大幅減輕了長文本生成時的記憶體壓力。當每個 token 佔用的快取空間縮小，相同的顯存容量就能支撐更長的文件輸入，或是提高系統並行處理的吞吐量。

而且這是從架構層面縮小快取，並不和數值量化衝突，DeepSeek 在 V2 就兩個一起用（KV 快取平均量化到 6 bits）。至於效果，V3 的報告則說明能和標準的 MHA 維持相近的表現。最後說明，這篇是根據 DeepSeek-V3 公布的架構資料整理而成，提供給想了解大型模型推論優化方向的朋友參考。