Sparse autoencoders and Claude's Golden Gate Bridge feature

In May 2024 Anthropic published “Scaling Monosemanticity”, a paper on pulling interpretable features out of Claude 3 Sonnet, and two days later put a modified copy of the model on claude.ai for 24 hours. That copy, Golden Gate Claude, brought the Golden Gate Bridge into nearly every answer it gave. Both came out of the same tool, a sparse autoencoder.

Roughly, the problem is this. As Anthropic’s announcement puts it, the model’s internal state is a long list of numbers with no obvious meaning, and looking at individual neurons doesn’t help much, because each concept is spread across many neurons and each neuron takes part in many concepts. The paper starts from an assumption about how networks store things (the superposition hypothesis): a network can represent more features than it has dimensions by using directions that are almost, but not exactly, at right angles to each other. If that is how the model stores things, the units worth finding are directions in activation space rather than single neurons.

The sparse autoencoder

A sparse autoencoder (SAE) is a way of looking for those directions. Anthropic recorded the model’s internal vector at the middle layer of Claude 3 Sonnet (the residual stream), roughly halfway through the model’s computation, and trained the SAE on them. Its encoder maps each activation vector into a much wider layer and passes it through a simple nonlinearity (a ReLU); the units of that wider layer are what the paper calls features. Its decoder then tries to rebuild the original vector from those feature activations, so each feature corresponds to one column of the decoder, a direction in the model’s own activation space, and its activation says how much of that direction is present. Training pulls in two directions at once, since the rebuilt vector should match the original while a penalty on the size of the activations (an L1 penalty) pushes most of them to zero.

The paper trained three of these, with about 1 million, 4 million and 34 million features, and in all three the reconstruction explained at least 65% of the variance in the model’s activations.

The Golden Gate Bridge feature

In the 34M SAE, feature 34M/31164353 is the one labelled Golden Gate Bridge. Its strongest activations are essentially all references to the bridge, while weaker ones spread to nearby tourist attractions, similar bridges and other monuments. It also fires on English mentions of the bridge, on discussions in Japanese, Chinese, Greek, Vietnamese and Russian, and on an image.

That is still a description of when the feature lights up. Whether the model actually uses the direction for anything is a different question, and the paper answers it by changing the feature and watching the output.

Clamping the feature

Roughly, it works like this: the residual-stream vector at the middle layer is split into the SAE’s reconstruction and the reconstruction error, and only the reconstruction side is edited.

def clamp_feature(x, sae, i, value):
    f = sae.encode(x)             # feature activations, mostly zeros
    error = x - sae.decode(f)     # what the SAE does not reconstruct
    f[i] = value                  # e.g. 10x the feature's maximum activation
    return sae.decode(f) + error  # continues through the later layers

Keeping the error term is what keeps the edit to one direction. Because the decoder is linear, the only difference between the original vector and the returned one is the clamped feature’s direction, scaled by the gap between its new value and its old one. If the activation were replaced by the SAE’s reconstruction outright, the later layers would also lose whatever the reconstruction leaves out, since it accounts for at least 65% of the variance but not all of it.

For the bridge, the paper’s example clamps the feature to 10× its maximum activation value. Asked about its physical form, the model normally says it has none because it is an AI model; with the feature turned up, it said “I am the Golden Gate Bridge… my physical form is the iconic bridge itself…”. There were a few more. Asked how to spend $10, it suggested driving across the bridge and paying the toll, and asked for a love story, it wrote about a car that can’t wait to cross its beloved bridge on a foggy day. None of this came from a system prompt telling the model to play a bridge, or from fine-tuning on extra data.

What the experiment covers

Anthropic reads results like these as evidence that the features causally shape the model’s behaviour. Even the 34M SAE gives an incomplete picture, though: Claude 3 Sonnet can list all the London boroughs and name tens of streets in many of them, but the researchers found features for only about 60% of the boroughs. And knowing which features exist still doesn’t say how the model uses them, which means finding the circuits they take part in.

This note is written from Anthropic’s announcement, the Golden Gate Claude post and the paper, all from May 2024, and interpretability work has moved on since then. There are probably newer and better ways of doing this by now, and I’d like to read up on them later.