DeepSeek-V3 has 671B parameters, 37B of which are activated for each token, and its full training took 2.788M H800 GPU hours. The gap between 671B and 37B comes from the Mixture-of-Experts (MoE) design: each token is routed to a small subset of feed-forward experts, so the parameter count can grow without a matching jump in compute. But the routing is also where MoE training goes wrong. If the gate keeps sending a large share of tokens to a few experts, the remaining parameters sit idle, and the GPUs that host the busy experts slow everything else down. The technical report calls the extreme case routing collapse.

The usual fix is an auxiliary load-balancing loss added to the training objective, which penalizes the model when expert loads drift apart. The penalty’s gradient reaches the same weights as the language-modeling loss, though, so if it is set too large it hurts model performance, as the report says.

A bias term for each expert

DeepSeek-V3 moves most of the balancing out of the loss. Each expert gets a bias term that is added to its affinity score, and the top-K experts for a token are picked from those biased scores. The bias is used only for that choice: the gating value that weights the expert’s output still comes from the original score.

The bias changes once per training step. DeepSeek monitors the expert load over the whole batch of the step, and at the end of it lowers the bias of every overloaded expert by a fixed amount and raises the bias of every underloaded one by the same amount. An overloaded expert then needs a higher raw score to make a token’s top-K, and an underused one starts to win tokens that sat near the cutoff. The report sets that amount to 0.001 for the first 14.3T training tokens and to 0.0 for the remaining 500B.

Because the bias never enters the loss, it adds no gradient of its own. But it does not remove every balancing term. The report keeps a complementary sequence-wise balance loss to prevent extreme imbalance within any single sequence, with its weight set to an extremely small 0.0001. So “auxiliary-loss-free” in the README means the bias does the balancing work instead of a large loss term, and the README says this keeps down the performance degradation that comes from encouraging load balancing.

Load stayed even enough that DeepSeek-V3 did not drop any tokens during training. The report also tested the bias method against baselines balanced purely by auxiliary losses, at 15.7B and 228.7B total parameters, and it did better on most of the evaluation benchmarks. On the Pile test set, a 16B model trained with the bias showed greater expert specialization by domain than its auxiliary-loss counterpart.

Multi-token prediction

DeepSeek-V3 also trains with a Multi-Token Prediction (MTP) objective. A standard language model is trained to predict only the next token, at position $t+1$. DeepSeek-V3 adds one MTP module after the main model, so each position is trained to predict the next two tokens. The module is a single Transformer block that shares the embedding layer and the output head with the main model. It predicts sequentially: it takes the main model’s representation of the current token together with the embedding of the next one, which keeps the complete causal chain at each prediction depth.

The report gives two reasons this could help. The extra target densifies the training signal and may improve data efficiency, and the model may learn to pre-plan its representations for tokens further ahead. In its ablations, adding the module improved the main model on most of the evaluation benchmarks, at both 15.7B and 228.7B total parameters. The module ships with the weights, which is why the files on Hugging Face total 685B: 671B for the main model and 14B for the MTP module.

At inference the module is optional. The main model works normally without it, and the module can also be reused for speculative decoding, where it proposes the second token and the main model checks it. Speculative decoding usually pairs a large model with a separately trained small draft model; here the draft comes with the weights. The report measures the acceptance rate of the second token at 85% to 90% across generation topics, and puts the resulting decoding speed at 1.8 times the tokens per second. A runtime has to support the module to use it that way, and the README says MTP support is currently under active development within the community.

This post is put together from the DeepSeek-V3 README and technical report.