程式代理評測
ProgramDistill Evaluates Agents Rebuilding Software from Replayable Web Behaviors, with Full-Workflow Success Rates of Only About 50%
A Microsoft Research team had coding agents observe interactive reference websites and then recreate their functionality without access to the source code. ProgramDistill generated 4,063 tasks from 26 applications, showing that repairing individual features is far easier, while complete workflows dependent on multiple stages of state remain the primary bottleneck.
Submitted on September 16, ProgramDistill moves coding-agent evaluation beyond “fixing a described bug based on an issue” toward a scenario closer to reverse engineering. An agent can interact with a complete reference website only through a browser and cannot access its source code or a gold patch. It must infer state transitions and feature dependencies, then modify a separate version of the application with missing functionality. A verifier replays the same actions and checks the results, so the implementation does not need to match the original line for line; instead, the comparison focuses on observable behavior.
The team’s `mine-craft-patch` pipeline first has an agent explore an application and save traces containing browser actions, success signals, and preconditions. It then removes the corresponding implementation, confirms that prerequisite features still work and that the target behavior does fail, and uses a gold patch to verify recoverability. Across 26 websites, the pipeline identified 1,975 replayable behaviors and, without human intervention, assembled 2,862 single-feature tasks and 1,201 cumulative tasks. The latter remove multiple features along a dependency chain, allowing reconstruction depth to be controlled from one to eight levels.
Nine agents participated in a 300-task partial-reconstruction evaluation. GPT‑6 Astra achieved a 100% success rate at depth one, but fell to 64% at depth eight. At depth eight, every other agent retained less than half of its own depth-one success rate. When rebuilding complete applications from a minimal scaffold, Astra achieved 49.15% across 413 cumulative workflows, compared with 28.81% for Claude Opus 5 and 21.07% for GPT‑5.6 Sol. Of 977 failed behaviors, 59.2% had never been observed by the agents on the reference websites, indicating that exploration coverage becomes a bottleneck before raw code-generation ability does.
This approach allows engineering teams to measure separately whether an agent discovers a feature and whether it can faithfully reconstruct it. It also provides deterministic replay traces that can serve as reinforcement-learning rewards. However, the results were all produced using the paper team’s harness, budgets, and model configurations. Validation currently focuses on website behaviors that can be described through actions and success signals, and screenshot-level visual fidelity has not yet been evaluated. The official full dashboard is also still marked as coming soon, so the degree of third-party reproducibility remains to be tested.