模型可解釋性
Adding Compatible Adjectives Still Causes SAEs to Lose 20% to 60% of Activated Features, Challenging the “Bag of Features” Intuition
A Heidelberg University study finds that although individual latents in sparse autoencoders may be interpretable, the full set of activated latents does not map reliably onto human concepts. Simply adding a compatible adjective before a noun can make 20% to 60% of the original activation set disappear, limiting the reliability of interpreting model states through set overlap.
Sparse autoencoders (SAEs) are often treated as “microscopes” for model internals: they decompose a dense residual stream into a small number of activated latents, which researchers then assign conceptual interpretations based on the text that triggers them. A new study asks a further question: even if individual latents appear interpretable, does the complete set of activated latents really combine consistently like a “bag of features”? The authors replace cosine similarity between dense vectors with the Jaccard overlap of activation sets. They first show that this metric can recover feature unions in a synthetic toy model and form semantically coherent neighborhoods across 30,000 text passages from The Pile.
The results weaken when applied to natural language. In the main experiments using Gemma 3 270M and a 16K big SAE, SAE activation sets were no better than the residual stream at recovering human category boundaries. For within-category rankings—such as “a robin is a more typical bird than a penguin”—Spearman correlations were near zero for most categories. The researchers also created 200 noun sequences, progressively adding as many as five adjectives describing color, size, or state. If the representation exhibited simple compositionality, “yellow shovel” should preserve the latents for “shovel” while adding those for “yellow.” In practice, 20% to 60% of the original noun latents were typically lost, with the loss rate increasing alongside the number of adjectives and, in some cases, model-layer depth.
This does not mean SAEs have no value. About 60% of the latents lost in certain middle layers had appeared in upstream layers, suggesting that the issue may partly involve where the probe is applied and the activation threshold, rather than the model completely forgetting the concept. The study also tested only short English noun phrases, manually selected adjective sets, and a limited range of Gemma/SAE configurations, and reproduction code has not yet been released. In engineering applications, a single activation, differences between latent sets, or automatically generated labels should not be treated directly as causal evidence. If SAEs are to support monitoring, steering, or safety decisions, the next steps should include cross-prompt stability testing, layer-by-layer tracking, and experiments involving actual interventions.