Compliance Detectors Still Mostly Read the Scenario Even When Given Rules; Cross-Rule Tests Fall Nearly to Chance

The study examined deployed guard models, internal activation probes, zero-shot judges, and policy-conditioned models to determine whether they actually use the rules provided in the prompt when assessing compliance. The core counterfactual intervention was straightforward: hold the scenario constant, then remove or shuffle the rule, or replace it with a rule from another domain—or even one implying the opposite conclusion—and observe whether the judgment changes. The results showed that the ICS probe achieved a mean AUROC of 0.9519 with the correct rules. After the rules were removed, shuffled, or replaced with rules from the wrong domain, its AUROC was instead approximately 0.955, with no significant decline across any of the 20 domains.

The authors call this phenomenon “rule blindness.” Fixed-taxonomy systems such as Llama Guard 3 and Qwen3Guard do not accept arbitrary rules in the first place. More notably, Latent Policy Guard—which has an input channel for custom policies and can even cite relevant provisions—also rarely changed its decisions after the policy was replaced. This suggests that a model may recognize text that “looks like a violation” without determining whether the scenario actually violates the specified rule.

To eliminate the possibility that dataset labels could be inferred directly from scenario vocabulary, the team also constructed 200 sets of four-cell cases across eight domains. Each set paired two rules that differed only by swapped thresholds or roles with two superficially similar scenarios, causing the labels to flip across the four combinations. Frozen ICS, logistic regression, MLP, Llama Guard 3, Qwen3Guard, and single-token judges all landed at roughly 0.50 AUROC, while their four-cell exact-match rates were mostly zero. Even after retraining, the Qwen3-8B MLP achieved only 0.670 AUROC and an 11% exact-match rate. When the same Llama-3.1-8B was instead prompted to reason step by step, its AUROC on a 40-set subsample rose to 0.849, with a four-cell exact-match rate of 74.4%.

The engineering implication is that safety classifiers cannot be treated as regulatory engines. If policies vary by region, customer, or version, evaluations must include paired cases in which the scenario stays fixed while the rule flips. They should also measure ranking performance, threshold calibration, and genuine rule–scenario compositional ability separately. The study remains a non-peer-reviewed preprint; the step-by-step reasoning results were obtained only on a smaller subsample, while the authors’ proposed low-cost ICS method did not meet its preregistered threshold for comprehensive superiority and can be defeated by adaptive white-box attacks.