AI 評測與基礎設施
Φ-Bench Tests Coding Agents on 85 Real-World Infrastructure Tasks, but the Best Model Still Scores Just 36.53% Overall
Φ-Bench expands evaluation beyond individual GPU kernels to multi-file implementation and end-to-end system optimization. All eight frontier models occasionally find effective modifications, but larger reasoning budgets do not consistently translate into better engineering outcomes.
The newly released [Φ-Bench paper](https://arxiv.org/abs/2609.10226) attempts to answer a self-referential engineering question: Can large language models improve the software stack underpinning their own training and inference? Rather than limiting the evaluation to generating a Triton or CUDA kernel, the benchmark compiles 85 tasks from systems research and open-source libraries. They span nine infrastructure categories, including GPU kernels, distributed training, inference serving, quantization, communication, checkpointing, MoE routing, and attention.
The tasks are divided into three tiers: 55 Kernel Function Completion tasks specify the function and interface; 20 Long-Horizon Implementation tasks describe only the required functionality, leaving agents to locate where changes should be made; and 10 End-to-End Optimization tasks provide only system-level objectives and constraints. The public [project page](https://faibench.org/) states that each task runs in a Docker environment, long-horizon tasks permit multiple rounds of modifications, and scoring is based on correctness or performance relative to a reference implementation. This makes the evaluation more akin to “understand an existing system, locate bottlenecks, modify it, measure the results, and iterate” than one-shot code completion.
The paper evaluates eight frontier models. Claude Opus 5 achieves the highest weighted overall score, but only 36.53%. It scores 37.16% on KFC, 21.60% on LHI, and 62.94% on E2EO. Trajectory analysis indicates that stronger models tend to change one variable at a time, establish more reliable local validation, and use failed results to narrow the search space. Weaker models, by contrast, often repeatedly tune parameters within the noise range. Increasing reasoning effort does not produce monotonic gains, with GPT-5.6 Sol proving especially unstable at the medium setting.
The benchmark is valuable because it exposes coding agents’ abilities in exploration and experimental design for performance engineering, but its scores should not be interpreted directly as measures of general infrastructure engineering productivity. Performance anchors are calibrated to an NVIDIA H20 or the authors’ Sapphire Rapids CPU, so changing the hardware requires rebuilding the baselines. The agent frameworks used with the models are also not identical. Engineering teams should next watch whether third parties can reproduce the leaderboard and whether the models are merely adapting to public tests and specific hardware.