<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>NumPy on KbWen Blog</title>
    <link>https://www.kbwen.com/tags/numpy/</link>
    <description>Articles and notes on AI systems, machine learning, Python, data engineering, and software development.</description>
    <generator>Hugo</generator>
    <language>zh-tw</language>
    <image>
      <url>https://www.kbwen.com/images/og-default.png</url>
      <title>KbWen Blog</title>
      <link>https://www.kbwen.com/</link>
    </image>
    
    <lastBuildDate>Thu, 05 Nov 2026 09:18:57 +0800</lastBuildDate><atom:link href="https://www.kbwen.com/tags/numpy/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Why does attention divide by √d_k?</title>
      <link>https://www.kbwen.com/why-attention-scales-by-sqrt-dk/</link>
      <pubDate>Thu, 05 Nov 2026 09:18:57 +0800</pubDate><dc:creator>KbWen</dc:creator>
      <guid>https://www.kbwen.com/why-attention-scales-by-sqrt-dk/</guid>
      <description>Where the √d_k in the attention formula comes from, and what happens to softmax without it.</description>
      <content:encoded><![CDATA[<p>In the base Transformer from <a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a>, every attention score is divided by 8 before it goes into softmax. The 8 is √d_k, the square root of the dimension of the query and key vectors, which the paper sets to 64 for each attention head.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Attention(Q, K, V) = softmax(Q Kᵀ / √d_k) V
</span></span></code></pre></div><p>Why 8, and why divide at all? The reasoning sits in Section 3.2.1 of the <a href="https://ar5iv.labs.arxiv.org/html/1706.03762">paper</a>, and it begins from a result the paper cites from earlier work: without the scaling, additive attention did better than dot-product attention once d_k got large. The suspected cause is worded as just that, a suspicion: &ldquo;We suspect that for large values of d_k, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients.&rdquo; The footnote under that sentence works out why the dot products would grow, from a toy assumption: &ldquo;assume that the components of q and k are independent random variables with mean 0 and variance 1.&rdquo;</p>
<p>Under that assumption, each product q_i·k_i has mean 0 and variance 1, and the dot product q·k is the sum of d_k of those products. Variances of independent terms add, so the sum has variance d_k. Its standard deviation, the typical size of a score, is √d_k.</p>
<p>Random vectors drawn the way the footnote describes make this easy to check, for d_k = 4, 64 and 512:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">rng</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">default_rng</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">softmax</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">e</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="n">x</span> <span class="o">-</span> <span class="n">x</span><span class="o">.</span><span class="n">max</span><span class="p">())</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">e</span> <span class="o">/</span> <span class="n">e</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="s2">&#34;d_k   std(q.k)   std(q.k/sqrt(d_k))&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> <span class="n">d_k</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">512</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">q</span> <span class="o">=</span> <span class="n">rng</span><span class="o">.</span><span class="n">standard_normal</span><span class="p">((</span><span class="mi">100_000</span><span class="p">,</span> <span class="n">d_k</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">k</span> <span class="o">=</span> <span class="n">rng</span><span class="o">.</span><span class="n">standard_normal</span><span class="p">((</span><span class="mi">100_000</span><span class="p">,</span> <span class="n">d_k</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">dots</span> <span class="o">=</span> <span class="p">(</span><span class="n">q</span> <span class="o">*</span> <span class="n">k</span><span class="p">)</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">d_k</span><span class="si">:</span><span class="s2">&lt;5</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">dots</span><span class="o">.</span><span class="n">std</span><span class="p">()</span><span class="si">:</span><span class="s2">9.2f</span><span class="si">}</span><span class="s2">   </span><span class="si">{</span><span class="p">(</span><span class="n">dots</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">d_k</span><span class="p">))</span><span class="o">.</span><span class="n">std</span><span class="p">()</span><span class="si">:</span><span class="s2">9.2f</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="k">for</span> <span class="n">d_k</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">64</span><span class="p">,</span> <span class="mi">512</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">q</span> <span class="o">=</span> <span class="n">rng</span><span class="o">.</span><span class="n">standard_normal</span><span class="p">(</span><span class="n">d_k</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">K</span> <span class="o">=</span> <span class="n">rng</span><span class="o">.</span><span class="n">standard_normal</span><span class="p">((</span><span class="mi">8</span><span class="p">,</span> <span class="n">d_k</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="n">scores</span> <span class="o">=</span> <span class="n">K</span> <span class="o">@</span> <span class="n">q</span>
</span></span><span class="line"><span class="cl">    <span class="n">raw</span> <span class="o">=</span> <span class="n">softmax</span><span class="p">(</span><span class="n">scores</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="n">scaled</span> <span class="o">=</span> <span class="n">softmax</span><span class="p">(</span><span class="n">scores</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">d_k</span><span class="p">))</span>
</span></span><span class="line"><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;d_k=</span><span class="si">{</span><span class="n">d_k</span><span class="si">:</span><span class="s2">&lt;4</span><span class="si">}</span><span class="s2"> max weight  raw </span><span class="si">{</span><span class="n">raw</span><span class="o">.</span><span class="n">max</span><span class="p">()</span><span class="si">:</span><span class="s2">.3f</span><span class="si">}</span><span class="s2">   scaled </span><span class="si">{</span><span class="n">scaled</span><span class="o">.</span><span class="n">max</span><span class="p">()</span><span class="si">:</span><span class="s2">.3f</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">()</span>
</span></span><span class="line"><span class="cl"><span class="c1"># raw and scaled still hold the d_k = 512 row from the last pass of the loop</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="s2">&#34;d_k=512 raw   &#34;</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">round</span><span class="p">(</span><span class="n">raw</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
</span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="s2">&#34;d_k=512 scaled&#34;</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">round</span><span class="p">(</span><span class="n">scaled</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">d_k   std(q.k)   std(q.k/sqrt(d_k))
</span></span><span class="line"><span class="cl">4          2.01        1.00
</span></span><span class="line"><span class="cl">64         8.01        1.00
</span></span><span class="line"><span class="cl">512       22.68        1.00
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">d_k=4    max weight  raw 0.733   scaled 0.410
</span></span><span class="line"><span class="cl">d_k=64   max weight  raw 0.976   scaled 0.360
</span></span><span class="line"><span class="cl">d_k=512  max weight  raw 1.000   scaled 0.289
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">d_k=512 raw    [0. 0. 0. 0. 0. 0. 0. 1.]
</span></span><span class="line"><span class="cl">d_k=512 scaled [0.135 0.13  0.023 0.012 0.186 0.17  0.054 0.289]
</span></span></code></pre></div><p>(Python 3.14.3 and NumPy 2.4.2. The vectors come from a standard normal distribution, which is exactly the footnote&rsquo;s mean 0, variance 1 case.)</p>
<p>The middle column of the table is close to 2, 8 and 22.6, the square roots of 4, 64 and 512. So the scores do get bigger as d_k grows, but by the square root: going from 64 to 512 dimensions multiplies d_k by 8 and the spread of the scores by a bit under 3. The last column is the same dot products after the division, and it reads 1.00 on every line. Dividing a number by c divides its variance by c², so dividing by √d_k takes the variance from d_k back to 1. Dividing by d_k itself would cut the variance to 1/d_k, and at d_k = 512 the scores would then sit so close together that softmax would hand all eight keys nearly the same weight, no matter which key matched the query best.</p>
<p>Without the division, the largest weight climbs with d_k: 0.733 at d_k = 4, 0.976 at d_k = 64, the base model&rsquo;s size, and 1.000 at 512, where the row, rounded to three places, is seven zeros and a 1. Softmax exponentiates the scores, and in that row the top score sits far enough above the second that its exponential swamps all the others. With the division, the largest weight stays under a half on all three lines. The scaled 512 row comes from the same query and the same eight keys, and it puts 0.289 on that last key and between about 0.01 and 0.19 on each of the other seven.</p>
<p>Is that row just bad luck with the seed? Partly. Each row is one random draw, and with other seeds the unscaled 512 row doesn&rsquo;t always come out as a single 1: when the top two scores happen to land close together, they split the weight between them, although both still sit far above the other six. (The standard deviations in the first table hardly move from seed to seed, since each one comes from 100,000 pairs.) What a bigger spread does is make it more likely that one score ends up far enough ahead to take nearly everything.</p>
<p>That lopsided row is where the suspicion about gradients comes in. When one weight is already 1 and the rest are 0, nudging any of the scores a little leaves the weights almost exactly where they were, so very little gradient flows back into the queries and keys. Scaled to variance 1, the same scores stay in a range where softmax still responds to small changes, which is what the scaled 512 row looks like: no key at zero, none at one.</p>
<p>The footnote makes its assumption only to illustrate why the dot products get large, and the queries and keys inside a trained model need not behave like independent numbers with variance 1. The divisor doesn&rsquo;t adjust for that: it is computed from d_k alone, so how close a trained layer&rsquo;s scaled scores come to variance 1 is something you would find out by printing them from that layer.</p>
]]></content:encoded>
    </item>
    
  </channel>
</rss>
