代理評測
MobilePA-Bench Runs 1,705 Stateful Mobile Tasks, Yet the Best Planner Scores Just 75.52%
Alibaba’s MAI team tests central planning agents for mobile devices using mutable application databases, permission errors, and tool dependencies, moving beyond static function-call matching. The best model is stronger at direct tool use, but still lags significantly in memory application and cross-agent collaboration.
MobilePA-Bench advances mobile-agent evaluation from checking whether “JSON matches the answer” to executing state transitions. Its simulated environment covers 1,705 tasks, 212 tools, and 13 functional domains. Every tool call modifies application data, device state, or permissions, then returns outcomes such as missing parameters, `PermissionDenied`, or entity ambiguity to the planner. Depending on the task, the evaluator checks the exact tool and parameters, terminal states that may be reached through multiple valid paths, required call ordering, or—when delegating to a specialized sub-agent—the routing decision and handoff content.
The tasks comprise 1,040 basic tool-use tasks, 376 memory-use tasks, 200 Skill-use tasks, and 89 sub-agent collaboration tasks. GUI operations are treated as a downstream capability that can be delegated, so the benchmark primarily isolates whether the central planner can select APIs, apply personal preferences, load compound workflows, and handle runtime feedback. Unlike AndroidWorld, it does not directly measure visual localization or click accuracy.
Among the 13 models evaluated, Claude Opus 5 achieved the highest weighted overall score at 75.52%. Its basic tool-use and Skill scores were 83.85% and 78.00%, respectively, but its memory-use score was only 58.51%. No single model led every category: Qwen 3.8 Max scored 64.63% on memory use, while Gemini 3.1 Pro reached 77.53% on sub-agent collaboration. This shows that direct function calling, personalized grounding, and delegation capabilities are not interchangeable. The error analysis also found that, when models encounter permission issues or constraints, they often generate nonexistent or invalid tool calls prematurely instead of asking follow-up questions or replanning.
A significant gap remains in the project’s public release. The paper claims that the complete tasks, data, and high-throughput sandbox have been open-sourced. However, as of August 27, the Apache-2.0 GitHub repository contains only the website, README, and license files, while Hugging Face provides no linked dataset. The public pages show examples only; the actual queries and ground truth remain in a private evaluation service. Engineering teams can currently study the benchmark’s validation design and leaderboard, but they cannot fully reproduce the reported 75.52% result from the public artifacts. The next question is whether the sandbox, task versions, and reproducibility configuration will actually be released.