Plain summary: temperature, top-k, and top-p all act on the one probability distribution the model produces for the next token, in two different ways. Temperature reshapes the whole distribution by scaling the logits before softmax, sharpening it at low values and flattening it at high ones, while top-k and top-p truncate it down to a slice you are allowed to sample from. At temperature 0 the model just takes the top token (greedy decoding), and top-k and top-p do nothing.
Suppose a model has just read “The weather today is” and has to choose the next token. The final layer hands back a raw score, a logit, for every token in the vocabulary. Narrow the view to five candidates and the scores might look like this: sunny 4.0, cloudy 2.0, cold 1.0, nice 0.5, great 0.0. Softmax converts that column of logits into probabilities that sum to 1, which for these numbers lands near sunny 0.81, cloudy 0.11, cold 0.04, nice 0.024, great 0.015. The exact figures here are illustrative; the operations on them are not. Every sampling parameter you set is an edit to this one distribution.
Temperature edits it before softmax runs. You pick a number T, and each logit is divided by T. At T below 1 the logits spread apart: divide the five scores by 0.5 and they become 8, 4, 2, 1, 0, so after softmax sunny climbs to about 0.98 and everything else is nearly gone. At T above 1 the logits pull together: divide by 2 and they become 2, 1, 0.5, 0.25, 0, which softmax turns into roughly sunny 0.53, cloudy 0.19, cold 0.12, nice 0.09, great 0.07. The vLLM docs describe temperature as controlling “the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling.” Temperature never adds or removes a candidate token; it only changes the spacing between the ones already there.
The change is one of shape. A low temperature sharpens the distribution into a peak, so the top token dominates and sampling almost always returns it. A high temperature flattens the distribution toward uniform, so the tail tokens get real probability and less likely words show up in the output.
Truncating the list: top-k and top-p
top-k and top-p do not rescale the logits the way temperature does. They shorten the candidate list before a token is drawn, deciding which tokens are eligible at all. Take top-k first. Set k and you keep the k highest-probability tokens and drop the rest. The Hugging Face transformers config defines top_k as “the number of highest probability vocabulary tokens to keep for top-k-filtering,” with a default of 50. On the peaked distribution above, top_k = 2 keeps sunny and cloudy and throws away cold, nice, and great; a token is then sampled from just those two after their probabilities are renormalized.
top-p keeps a slice defined by cumulative probability instead of a fixed count. You set p, sort the tokens from most to least probable, and walk down the list adding up probabilities until the running total reaches p. That smallest set is what you keep. transformers puts it as “the smallest set of most probable tokens with probabilities that add up to top_p or higher.” On the peaked distribution, top_p = 0.9 adds sunny (0.81) then cloudy (0.11) to reach 0.92 and stops, so it also keeps two tokens. vLLM describes the same knob as controlling “the cumulative probability of the top tokens to consider.”
A fixed count versus a floating cutoff
The difference between the two shows up when the model’s confidence changes. Run top-p on the flatter distribution from the high-temperature example, where sunny is only 0.53: reaching 0.9 now needs sunny, cloudy, cold, and nice, so top_p = 0.9 keeps four tokens instead of two. top_k = 2 would keep exactly two in both cases. top-p’s cutoff floats with how sure the model is, widening when the probability is spread out and narrowing when one token already carries most of the mass. top-k’s cutoff sits at k no matter how the probability is distributed.
The order they run in, and temperature 0
Order matters because the two kinds of edit compose. Temperature reshapes the distribution, and then top-k or top-p truncate whatever shape it produced, which is why running top-p after a high temperature swept in more tokens above. Set temperature to 0 and this stacking collapses. At T = 0 the model stops sampling and takes the single highest-probability token at every step, which is greedy decoding, the “zero means greedy sampling” case. With one token already fixed deterministically, there is no distribution left to slice, so top-k and top-p have nothing to do.
The library defaults line up with all of this. transformers ships top_k at 50 and top_p at 1.0, and a top_p of 1.0 keeps every token, so out of the box top-p is effectively off and top-k trims to the top 50. vLLM uses the same off-switches: top_k “Set to 0 (or -1) to consider all tokens,” and top_p “Set to 1 to consider all tokens.” Other truncation methods exist on the same distribution too, notably min-p and repetition penalties, though the three above are the ones you meet first.
In practice, temperature is the knob to reach for when you want to move the overall character of the output between deterministic and loose, since it changes every probability at once. top-k and top-p are the knobs for putting a floor under quality by cutting off the unlikely tail, with top-p the more common default because its cutoff tracks the model’s confidence. And if you have set temperature to 0 to get greedy, deterministic output, you can leave top-k and top-p alone, because at that point they change nothing.
