PACE-Bench Breaks Existing Agent Designs After Physics Rules Change; Best Full-Scale Configuration Reaches Just 35.9% Pass@2

Most benchmarks for “self-evolving agents” ask only whether a system can use experience to improve on the next task, while the execution rules themselves usually remain unchanged. The [PACE-Bench paper](https://arxiv.org/abs/2608.14441) instead evaluates a scenario closer to maintenance in real-world deployments: the objectives and APIs for a bridge, vehicle, or controller remain unchanged, but friction, gravity, material strength, or torque limits are modified, ensuring that a previously validated Python design will fail. The agent must identify the cause from diagnostic signals returned by the sandbox—such as forces, constraint violations, and time to failure—and then rewrite the mechanism or control logic.

The dataset spans six physics domains, 36 base tasks, and four progressively harder target environments per task, producing 144 source-to-target adaptation cases. Each run may submit up to 20 candidate solutions. Reference solutions confirm that the target environments are solvable while also verifying that source solutions cannot be reused unchanged. In addition to final pass/fail results, the scoring system preserves partial constraint-completion scores so that binary success alone does not obscure how close an agent came to a feasible solution.

The study compares Reflexion, Self-Refine, ACE, ExpeL, Tree of Thoughts, CodeEvolve, and several test-time training methods. In the full evaluation, Reflexion with Qwen3-14B achieves the highest Pass@2, but only at 35.9%; the unaugmented loop using the same model reaches 32.0%. More notably, memory-based methods such as ACE, ExpeL, and ReasoningBank often underperform the baseline on larger models. Trajectory analysis indicates that earlier designs become anchors, causing agents to keep making incremental adjustments to an incorrect architecture. Even when the authors explicitly reveal which physical parameters changed, performance does not surpass the existing ceiling. This suggests that the bottleneck is not identifying “what changed,” but redesigning “what should be done.”

The [open-source repository](https://github.com/thunlp/PACE-Bench) provides Box2D tasks, validators, ten methods, an OpenAI-compatible API, and local vLLM/Transformers execution paths. It also includes an isolated coding-agent mode that keeps the real validators and credentials outside the container. However, all tasks remain author-designed 2D simulations, and the main full-scale comparisons primarily use Qwen3 models ranging from 4B to 14B. GPT-5.5’s 66.7% result covers only a 24-case statics subset and therefore cannot be directly compared with the full leaderboard. Future work should examine whether other teams can reproduce the results and whether memory methods can avoid design fixation by incorporating forgetting, branching, or counterfactual reconstruction.