How FlashAttention computes softmax one block at a time
One query's attention scores, split into two blocks and worked through by hand, to see how FlashAttention gets the softmax without having all the scores at once.
Attention
DeepSeek-V3 的 Multi-head Latent Attention 如何壓縮 KV 快取
我們來看看 DeepSeek-V3 如何利用 MLA 的低維潛在向量壓縮 KV 快取，解決長序列推論時的記憶體瓶頸。
How the KV cache speeds up LLM generation
A transformer redoes the same attention projections for every past token at each decoding step. The KV cache stores those keys and values so they get reused instead of recomputed, and the one cost it adds is memory that grows with the sequence.