How FlashAttention computes softmax one block at a time
One query's attention scores, split into two blocks and worked through by hand, to see how FlashAttention gets the softmax without having all the scores at once.
One query's attention scores, split into two blocks and worked through by hand, to see how FlashAttention gets the softmax without having all the scores at once.