Harness-IF Separates “What Models Would Do Anyway” From Genuine Instruction Following, Finding Gaps of Up to 7.4 Points Across 12 Coding Agents

Passing a test does not mean a coding agent truly understood and followed a project’s rules; it may simply have happened to take the same approach. [Harness-IF](https://arxiv.org/abs/2608.11727), submitted on August 12, establishes a control for this distinction. Its 60 multi-turn coding tasks draw from a repository of 642 rules, with final judgments covering 256 rules placed across five interfaces: system prompts, project instruction files, user messages, tool descriptions, and skill descriptions.

The core metric, Against-Prior Accuracy (AP-Acc), considers only rules that conflict with an agent’s default behavior when no instruction is provided. The researchers first reran nine probe variants with the rules removed to determine which behaviors would have occurred anyway, then measured whether adding the rules genuinely changed the agent’s actions. Across 12 frontier models, overall accuracy ranged from 72.1% to 85.9%, while AP-Acc ranged from just 66.1% to 78.6%. Every model’s score fell by 3.6 to 7.4 points, with an average gap of 5.81 points. After controlling for tasks shared across models, the top-ranked model remained unchanged, but three pairs of adjacent models swapped places, suggesting that conventional aggregate scores may misrepresent models’ actual instruction-following ability.

This has practical implications for teams using [AGENTS.md](https://agents.md/) or similar project-level rules. OpenAI’s published overview of the [Codex agent loop](https://openai.com/index/unrolling-the-codex-agent-loop/) shows that agents combine sandbox policies, project files, skills, and environment information into different prompt layers. Harness-IF’s nine-variant conflict experiment, however, found that actual precedence does not follow prompt depth in a simple order: system prompts, project files, and user instructions generally outranked tool and skill descriptions.

From an engineering perspective, critical rules should be converted into observable tests, policy checks, or execution receipts; verifying only that the final patch is correct is insufficient. One limitation is that the main experiment included only 60 tasks, while the conflict experiment covered just nine variants. The paper also provides no public code or dataset link. The next questions are whether the results can be reproduced across agent frameworks and whether the same precedence ordering persists when rules contradict one another or after context compaction.