AI 可解釋性
SAEVerbalizer Turns Feature Directions Directly Into Text, but Its Best Global Agreement Rate Is Still Only 56.1%
SAEVerbalizer injects sparse autoencoder decoder directions into Gemma 3 and trains the model to generate feature explanations directly, eliminating the need to search for highly activating examples for each feature. The 27B version achieved 56.1% Reference Agreement on the stricter global test, but the reference answers were themselves model-generated and cannot yet be treated as mechanistic ground truth.
Traditional SAE interpretation pipelines first run a model over a large corpus to find the highest-activating passages for each latent, then ask another LLM to infer their shared concept. The newly proposed SAEVerbalizer instead works directly from internal representations: during prompt prefill, it injects an SAE decoder direction into the token representations at a specified Transformer layer and fine-tunes only the subsequent layers, enabling the model to directly output a natural-language description of that direction. For cross-model transfer, a lightweight adapter maps directions from the source model into the verbalizer’s representation space.
The study used Gemma 3 1B, 4B, and 27B, along with 262k-width SAEs from Gemma Scope 2. The best configuration—a 27B model at layer 16—was trained on 48,000 Neuronpedia feature–explanation pairs and achieved Reference Agreement scores of 52.3%, 80.5%, and 56.1% across three held-out feature tests. The same backbone without fine-tuning scored only 1.6%, 2.5%, and 1.2%. When evaluated at the same layer using a previously unseen 65k-width dictionary, it scored 64.4%, 56.5%, and 65.9%, indicating that the learned capability was not completely tied to a single SAE dictionary. Feeding 1B directions through an adapter into the 27B verbalizer also slightly outperformed the native 1B version, while transferring 4B directions caused performance to decline across all three tests.
This work is notable because it replaces the need to “collect new evidence for every feature” with a reusable representation decoder, potentially reducing the cost of analyzing large dictionaries while also supporting tests of direction composition and sign inversion. However, RA measures only whether the output agrees with curated Neuronpedia explanations. Those explanations are themselves inferred from activation examples rather than constituting verifiable ground truth. Each configuration was also run only once, so random variation was not estimated. The authors primarily tested Gemma Scope 2 and did not release a public implementation. Next steps should include replication across model families, validation through causal interventions, and measurement of actual compute savings relative to a complete activation-based pipeline.