Roughly, the problem is this. As Anthropic’s announcement puts it, the model’s internal state is a long list of numbers with no obvious meaning, and looking at individual neurons doesn’t help much, because each concept is spread across many neurons and each neuron takes part in many concepts. The paper starts from an assumption about how networks store things (the superposition hypothesis): a network can represent more features than it has dimensions by using directions that are almost, but not exactly, at right angles to each other. If that is how the model stores things, the units worth finding are directions in activation space rather than single neurons.

The sparse autoencoder

A sparse autoencoder (SAE) is a way of looking for those directions. Anthropic recorded the model’s internal vector at the middle layer of Claude 3 Sonnet (the residual stream), roughly halfway through the model’s computation, and trained the SAE on them. Its encoder maps each activation vector into a much wider layer and passes it through a simple nonlinearity (a ReLU); the units of that wider layer are what the paper calls features. Its decoder then tries to rebuild the original vector from those feature activations, so each feature corresponds to one column of the decoder, a direction in the model’s own activation space, and its activation says how much of that direction is present. Training pulls in two directions at once, since the rebuilt vector should match the original while a penalty on the size of the activations (an L1 penalty) pushes most of them to zero.

The paper trained three of these, with about 1 million, 4 million and 34 million features, and in all three the reconstruction explained at least 65% of the variance in the model’s activations.

The Golden Gate Bridge feature

In the 34M SAE, feature 34M/31164353 is the one labelled Golden Gate Bridge. Its strongest activations are essentially all references to the bridge, while weaker ones spread to nearby tourist attractions, similar bridges and other monuments. It also fires on English mentions of the bridge, on discussions in Japanese, Chinese, Greek, Vietnamese and Russian, and on an image.

That is still a description of when the feature lights up. Whether the model actually uses the direction for anything is a different question, and the paper answers it by changing the feature and watching the output.

Clamping the feature

Roughly, it works like this: the residual-stream vector at the middle layer is split into the SAE’s reconstruction and the reconstruction error, and only the reconstruction side is edited.

def clamp_feature(x, sae, i, value):
    f = sae.encode(x)             # feature activations, mostly zeros
    error = x - sae.decode(f)     # what the SAE does not reconstruct
    f[i] = value                  # e.g. 10x the feature's maximum activation
    return sae.decode(f) + error  # continues through the later layers

Keeping the error term is what keeps the edit to one direction. Because the decoder is linear, the only difference between the original vector and the returned one is the clamped feature’s direction, scaled by the gap between its new value and its old one. If the activation were replaced by the SAE’s reconstruction outright, the later layers would also lose whatever the reconstruction leaves out, since it accounts for at least 65% of the variance but not all of it.

For the bridge, the paper’s example clamps the feature to 10× its maximum activation value. Asked about its physical form, the model normally says it has none because it is an AI model; with the feature turned up, it said “I am the Golden Gate Bridge… my physical form is the iconic bridge itself…”. There were a few more. Asked how to spend $10, it suggested driving across the bridge and paying the toll, and asked for a love story, it wrote about a car that can’t wait to cross its beloved bridge on a foggy day. None of this came from a system prompt telling the model to play a bridge, or from fine-tuning on extra data.

What the experiment covers

Anthropic reads results like these as evidence that the features causally shape the model’s behaviour. Even the 34M SAE gives an incomplete picture, though: Claude 3 Sonnet can list all the London boroughs and name tens of streets in many of them, but the researchers found features for only about 60% of the boroughs. And knowing which features exist still doesn’t say how the model uses them, which means finding the circuits they take part in.

This note is written from Anthropic’s announcement, the Golden Gate Claude post and the paper, all from May 2024, and interpretability work has moved on since then. There are probably newer and better ways of doing this by now, and I’d like to read up on them later.

這個版本沒有在系統提示詞裡叫它扮演一座橋，也沒有拿額外的資料重新微調。他們是直接調整了模型內部活化值裡的一個「特徵」（feature），把它的強度調高。這個特徵是從同一週發表的可解釋性研究裡找出來的。

大概是這樣，模型在讀一段文字時，內部的狀態也就是神經元的活化值。通常使用者直接閱讀這些數字，很難看出模型正在處理什麼概念，因為一個概念通常分散在很多神經元上，而每個神經元也會同時參與很多概念（來自 Anthropic 的研究文章）。同時發表的論文用疊加（superposition）的假設來看待這件事：模型在有限的維度裡，表示出比維度數量更多的概念。

研究團隊的做法叫字典學習（dictionary learning）。研究文章把特徵比喻成字典裡的詞，模型任何一個時刻的內部狀態，都可以用其中少數幾個詞拼出來。實際上，他們另外訓練了稀疏自編碼器（sparse autoencoder，SAE），讀取 Claude 3 Sonnet 中間層殘差流（residual stream）上的活化值。SAE 先把活化值展開成數量多的特徵，再用這些特徵把原本的活化值重建回去；訓練時一邊要求越接近越好，一邊要求同時有數值的特徵越少越好，所以重建出來的結果只是近似，不會完全一樣。這組特徵的數量很多，最大的版本有大約 3,400 萬個特徵；原本那串看不出意思的數字，於是變成「現在有哪幾百個特徵是亮著的」，那其中的意義和觀測就不一樣了。

金門大橋就是其中一個特徵。英文文字提到這座橋的名字時，這個特徵會亮；日文、中文、希臘文、越南文和俄文的相關討論裡也會亮起來，拿金門大橋的照片給模型看，同一個特徵一樣有反應。這個特徵活化最強的例子幾乎都在講這座橋，不過活化變弱的時候，就開始出現其他觀光景點、相似的橋和紀念建築。研究文章也列出這個特徵附近的其他特徵，像是惡魔島、金州勇士隊、1906 年的舊金山大地震，還有在舊金山取景的希區考克電影《迷魂記》。

找到特徵之後，就可以反過來調整它。做法是在模型往下計算的過程中，把中間層的殘差流拆成兩個部分，一部分是 SAE 重建出來的結果，另一部分是誤差。接著把重建結果裡金門大橋這個特徵的數值，固定在指定的大小，誤差那部分則原封不動。論文把固定數值的這一步叫做鉗位（clamp），這個替換會套用在每一個輸入的每一個 token 位置上。在這篇論文的例子裡，金門大橋特徵被固定在它最大活化值的 10 倍。

調整之後的反應，研究文章裡有一段對話。被問到「你的實體形態是什麼？」時，Claude 平常會回答「我沒有實體形態，我是 AI 模型」；特徵被調大之後，回答變成「我就是金門大橋……我的實體形態就是那座標誌性的大橋本身……」（原文為英文）。

這個反應對研究團隊來說，也算是一種驗證。調高一個特徵，模型的行為就往某個方向偏移，研究文章認為，這說明特徵除了跟輸入裡的概念相關，也會實際影響模型怎麼回答。展示說明也提到，同樣的方法可以拿來調整和危險程式碼、犯罪行為或欺騙有關的特徵，他們認為這是讓模型更安全的方向之一。

不過目前找到的這些特徵還很不完整。研究文章說明，找到的特徵只是模型在訓練中學到的概念裡的一小部分，用現在的方法要把特徵找齊，需要的計算量會遠遠超過當初訓練模型本身；而且知道模型有哪些表徵，也還不知道它怎麼使用這些表徵。這篇內容整理自 Anthropic 在 2024 年 5 月發表的研究文章、論文和展示說明；金門大橋 Claude 本身目前已經下線，沒辦法再直接跟它聊天了。但這也是兩年前的資訊了，現今應該有不同的優化方式，以後再來好好研究吧。