軟體代理與評測
ASIL Replaces Screen Clicking With Structured State, Raising Software Agent Success Rates From 6.6% to 81.6%
ASIL lets agents read software state as JSON, then execute semantic actions through file parsers, native scripts, or service APIs. It significantly outperformed screen control across 380 tasks, but the comparison changed both the information visible to the agent and the granularity of its actions, so the results cannot be interpreted simply as an improvement in model capability.
Researchers at Shanghai Jiao Tong University have introduced the Agent-Software Interaction Layer (ASIL), arguing that software agents should not be forced to imitate humans by looking at screens, translating targets into coordinates, and clicking. Instead, ASIL exposes document, object, and service state as structured JSON and lets models output verifiable semantic actions. The execution layer selects the deepest auditable access path available for each application, including parsing open document formats, invoking native scripting interfaces in software such as Blender or GIMP, and bridging to REST, WebSocket, desktop services, or the file system. Actions undergo schema validation before execution, while their results are verified by application-state evaluators.
The team built adapters for 15 applications, including LibreOffice, Blender, GIMP, JupyterLab, Gitea, Thunderbird, and OBS. Its benchmark comprises 300 single-application tasks and 80 cross-application tasks. Using the same tasks, initial files, and final-state evaluators, Kimi K2.5 achieved an overall score of 84.8. GPT-5.4 scored 81.6 with ASIL and a limit of 15 steps, while repaired screen control reached only 6.6 even with its limit increased to 50 steps. Sonnet 4.6 produced corresponding scores of 81.2 and 26.6. With ASIL, agents averaged fewer than five semantic actions per task. The structured trajectories can also support post-training: Qwen3.5-9B improved from 66.6 to 80.4 with supervised fine-tuning (SFT), then reached 82.2 through online reinforcement learning (RL) using evaluator feedback.
These figures primarily demonstrate the importance of interface contracts, rather than proving that ASIL outperforms mature native APIs in every situation. It surpassed the UNO API on LibreOffice tasks, but on draw.io it merely matched the existing MCP content interface. Structured state may also conceal screen layout, rendering errors, or what users can actually see. The code, adapters, tasks, and model weights have been released publicly, but the training implementation, per-task trajectories, and individual scores were not included in the repository. Future work should examine adapter maintenance costs, stability across software versions, least-privilege design, and whether agents can combine structured state with visual input to achieve both reliability and interface fidelity.