DeepSeek-V3 的 Multi-head Latent Attention 如何壓縮 KV 快取
我們來看看 DeepSeek-V3 如何利用 MLA 的低維潛在向量壓縮 KV 快取，解決長序列推論時的記憶體瓶頸。
Inference
How constrained decoding works for JSON
Let's see how a JSON schema, or a plain JSON mode, changes what a model is able to write.
What the parameter count in a model's name means
The B in a name like gpt-oss-20b counts parameters, in billions. Multiply that count by the bytes each parameter takes and you get a floor for the memory the weights need. That is also why some models now ship with two numbers instead of one.
How the KV cache speeds up LLM generation
A transformer redoes the same attention projections for every past token at each decoding step. The KV cache stores those keys and values so they get reused instead of recomputed, and the one cost it adds is memory that grows with the sequence.
把 temperature 設成 0，AI 就會每次都一樣嗎？
網路上常說：要 AI 每次給一樣的答案，把 temperature 設成 0 就好。但有人拿同一個 prompt、temperature 0 連跑 1000 次，還是冒出 80 種不同輸出。原因除了浮點誤差，更關鍵的是它在 GPU 上跟多少別的請求湊成一批一起算。聊聊為什麼『最確定』不等於『可重現』。