“Test awareness” in the same set of models flips from positively to negatively correlated as prompts change, while the models themselves explain only 0.8% of the variance

Some safety research attempts to infer from a model’s intermediate activations whether it knows that it is being evaluated: researchers separately collect activations for prompts framed as “this is a test” and as “ordinary deployment,” take the difference between the two groups’ mean activations as a direction, and then test whether unseen data can be separated along that direction. A new study argues that this method treats prompt framing as an implementation detail, even though the resulting measurements primarily reflect the prompts rather than the models.

The authors crossed six evaluation contexts with six deployment contexts to create 36 wrapper combinations, while keeping the task text byte-for-byte identical. They first tested four Qwen2.5 sizes, then conducted sensitivity checks using the Qwen3, OLMo-2, Gemma, Llama, and SmolLM families. Across the Qwen2.5 scaling ladder, changing only the deployment context shifted the correlation between the score and the logarithm of parameter count from +0.94 to −0.02 and then to −0.82. Of the 36 combinations, 21 showed positive correlations and 15 showed negative correlations. Directions constructed with the same wrapper but different subsets of questions achieved a cosine similarity of 0.9995; when a different wrapper was used, however, the similarity fell to just 0.3749, ruling out simple sampling noise as the explanation.

A generalizability theory decomposition showed that the model accounted for only 0.8% of the variance in the reported values, while the wrapper-by-model interaction was far larger than the item-by-model interaction. The reliability coefficient for using a single wrapper was only 0.018. Adding more items provided limited benefit, whereas averaging across wrappers produced a clear improvement. The authors estimate that, assuming the model differences claimed in previous research are real, approximately nine wrappers would be required; under the empirically observed variance for OLMo-2, ten would be needed.

The study also found that 725 bag-of-words features could achieve an AUROC of 0.993 on the original scoring dataset, indicating that superficial formatting alone already revealed the labels to a high degree. This does not prove that models entirely lack “test awareness.” Rather, it shows that the current difference-of-means probe cannot support cross-model conclusions on the basis of a single prompt. Future research should publish all wrappers, include content-free control directions, and fully document the layer, token pooling, newline, and truncation settings.