To predict the 1001st token in a sequence, a transformer runs attention over every token before it, all 1,000 of them. To predict the 1000th, it ran attention over the first 999. Between those two steps the first 999 tokens are the same tokens in the same positions. A decoder-only model never lets an earlier token attend to a later one, so their internal representations were fixed the moment each was first processed. A plain generation loop hands the whole growing sequence back to the model at every step anyway, and the model projects all of those past tokens again from scratch.
That repeated projection is the work the KV cache removes. The Hugging Face docs state the problem: “Each prediction depends on the previous tokens, which means the model performs the same computations each time” (Cache strategies). Nothing about that computation changes between steps; the model just runs it again for every token it generates.
What the cache stores
Inside each attention layer, every token is projected three ways: into a query, a key, and a value. At a decoding step the query comes from the newest token; the keys and values come from every token in the context so far. Attention multiplies that query against all the keys, then uses the scores to take a weighted sum of the values. The scores and the sum both depend on the newest query, so neither of them survives the step that produced it.
The KV cache stores the keys and the values, one pair per token per layer. It does not store the queries. Only the newest token’s query is ever needed to predict the next token, and a query is used in exactly one step and then never again, so keeping it around would buy nothing.
Why reuse is safe
In a decoder-only transformer, a cached key and a freshly computed one are the same numbers, and the causal mask is why. “For causal attention, the mask prevents the model from attending to future tokens” (Caching). Token 5 attends to tokens 1 through 5 and nothing after. Appending token 1001 to the sequence cannot change token 5’s key or value, because token 5 was never allowed to look past itself in the first place. Its key and value are final from the step that first produced them, so a cached copy is exact. Reading it back skips a computation and changes nothing else. A model that let earlier tokens see later ones would have to recompute everything on every step, because appending a token would change what the earlier ones represent.
One token at a time
The prompt goes through the model once, and every token’s keys and values are written into the cache. After that each step feeds the model a single token, the one just generated. The model computes that token’s query, key, and value, appends its key and value to the cache, and runs attention with the new query against the full set of cached keys and values.
Without a cache the model recomputes all the previous keys and values at every step, and the attention cost of a step grows quadratically with the sequence length. With a cache it computes only the current token’s key and value, and the same cost grows linearly (Caching).
In transformers, generate() keeps the cache enabled by default; you pass use_cache=False to turn it off, which the docs reserve for training, where caching can cause unexpected errors.
# caching is on by default
model.generate(**inputs, max_new_tokens=100)
# turn it off
model.generate(**inputs, max_new_tokens=100, use_cache=False)
What it costs
Every generated token adds a key and a value, in every layer and every attention head, to a store that only grows. The default DynamicCache “allows the cache size to grow dynamically in order to store an increasing number of keys and values as generation progresses” (Cache strategies), so its memory climbs with the length of the sequence.
Predicting token 1001 no longer reruns the projections for the first 1,000 tokens. It reads their keys and values out of the cache and computes only the new one. What it pays instead is the room to keep all 1,000 of those key-value pairs resident, per layer, per head, for as long as generation continues. The longer the sequence runs, the more arithmetic the cache skips and the more memory it holds. At short lengths that room is nothing. At long-context lengths the KV cache “can occupy a significant portion of memory and become a bottleneck” (Cache strategies) — the same cache that removed the recompute is the first thing to run a long generation out of memory.
