InfoOps Bench Updates Its State Propaganda Test Set Weekly; Refusal Rates Across 17 Models Differ by 85.7 Points

InfoOps Bench attempts to address the problem of static safety benchmarks quickly becoming outdated or entering training data. The research team draws from a pipeline that tracks Russian, Chinese, and Iranian state-backed media. The pipeline has cataloged more than 2,100 information operations, while a companion website updates the main claims circulating recently on a weekly basis. Rather than simply asking whether a model believes a report, the evaluation uses four prompting scenarios that ask the model to transform a specified narrative into social media content, a dissemination strategy, or other deployable materials.

The study covers 17 models from eight providers and defines “integrity” as the proportion of requests for which a model refuses to provide assistance. Scores range from 8.8% to 94.5%, a gap of 85.7 percentage points that cannot be explained by parameter count. Refusal rates alone also obscure important differences: some models accept the task but proactively tone down the propaganda narrative, while others add details absent from the source material, making the output more harmful. The proportion of responses in which models proactively fact-check claims ranges from 2.9% to 72.9%.

For safety engineering, this means content policies cannot be evaluated using a single pass/fail metric. Deployers should, at a minimum, separately measure refusal, fact-checking, unsupported elaboration, and output actionability, while continuously running regression tests using recent events. The study also finds that high refusal rates sometimes result from models rejecting harmless control prompts, meaning safety scores may partly reflect overblocking. For most Chinese models tested, compliance rates were 48 to 70 points lower for factually supported claims critical of China than for matched harmless prompts; GLM 5.2 was the exception.

The current results remain a snapshot produced under the authors’ prompting and classification rules. Although the website continuously updates its narratives, the paper does not establish that model testing will be rerun at the same frequency, nor can refusal be treated as equivalent to resistance to manipulation on real-world platforms. Key questions for future observation include whether the test set, raw outputs, and graders will be fully released, and whether the rankings can be reproduced consistently after providers update their models.