Semantics-Preserving Rewrites Raise Coding Agent Costs by Up to 22.9%, While Robustness Rankings Flip Across Frameworks

Coding agents can produce different success rates and token costs even when working with repositories that are functionally identical but written differently on the surface. A new study created a randomized variant sampler that applies 14 types of semantics-preserving transformations to Python projects, including control-flow-equivalent rewrites, identifier renaming, and dead-code insertion, before asking agents to fix the same original issues. Variants still had to pass baseline tests, while agent-submitted patches were evaluated against both FAIL_TO_PASS and PASS_TO_PASS tests.

The researchers selected 28 tasks from SWE-bench Verified and 26 from SWE-bench Pro. They tested mini-SWE agent and OpenCode with Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B. For each task, they ran the original version 20 times and generated 20 variants that were each run once; the transformations affected a median of 6.9% to 7.7% of code lines. This setup was designed to separate randomness in the models themselves from differences caused by code variants. The [paper](https://arxiv.org/abs/2608.18389) and [complete experimental code](https://github.com/CSU-TrustLab/jagged-frontier) are publicly available.

Solve rates declined in 13 of the 16 framework-model-dataset configurations, with the 95% interval excluding zero in six configurations. The largest average decline was 6.7 percentage points for mini-SWE agent with Opus on SWE-bench Pro. However, there was no consistent winner in robustness rankings: Qwen declined by just 0.2 points with mini-SWE agent on Verified, but fell by 5.5 points after switching to OpenCode, making it the least robust model in that group. Reverse ranking flips also appeared for MiniMax and Kimi.

Successful runs also incurred additional costs. On Verified, token costs increased across all eight configurations, ranging from 4.0% to 22.9%. Input tokens per step increased in 13 of the 16 configurations, indicating that agents were not merely taking more steps but were reading more context at each step. This suggests that procurement and deployment teams should treat baseline solve rates as measures of capability, not as evidence of robustness to real-world coding styles.

The study is limited to 54 sampled tasks and did not search over transformer hyperparameters. Bootstrap intervals calculated over a fixed task set also cannot represent the full SWE-bench population. Future work should repeat the experiments across programming languages, refactoring tools, and real enterprise codebases, while incorporating success rates, input costs, and framework versions into continuous-integration evaluations.