電腦操作代理
TMI Reconstructs Task Trees from Interleaved Computer Activity, Raising Step-Description Accuracy from 30.3% to 74.9%
Task Model Induction does not compress an entire screen and input log into a single workflow. Instead, it first disentangles noncontiguous tasks, then reconstructs goal hierarchies and loop-based control flow. The resulting agent skills improved held-out accuracy from the best baseline’s 14.29% to 18.57%, but the full pipeline depends on frontier models, while raw interaction traces introduce significant privacy risks.
A team from Stanford University and Carnegie Mellon University has introduced Task Model Induction (TMI), an approach that converts screenshots, mouse clicks, and keyboard events into auditable, reusable task models. Existing workflow-summarization methods typically assume that a recording corresponds to a single goal. TMI instead allows a user to switch among multiple applications and regroups noncontiguous activities belonging to the same task.
The system operates in three stages. In the first stage, a vision-language model compares the screens before and after each interaction, grounding low-level events such as coordinate-based clicks into semantically meaningful actions and segmenting them according to local goals. In the second stage, each activity is assigned to an existing latent task or used to create a new one. Identifiers such as files, URLs, and entity names help the system remain robust to changes in applications and terminology. Finally, a global pass merges tasks that were incorrectly split apart.
The third stage separately generates a recursive goal tree describing “why” actions are performed and a procedural tree describing “how” they are performed. The procedure uses only control-flow constructs observable from the trace: `sequence`, `for-each`, and `while`. A deterministic verifier requires every activity to have an assignment and every loop to include either observed instances or an exit condition. The two trees are then reconciled into a model whose nodes contain both goals and control flow. All induction stages use GPT-5.4, while GPT-5.5 and Claude Sonnet 5 are used separately for evaluation.
On interleaved traces synthesized from 38 HumanWork recordings, TMI achieved an Adjusted Rand Index of 0.974 for task clustering. When evaluated by GPT-5.5, its step-description accuracy reached 74.9%, compared with 30.3% for workflow summarization. Its control-flow operator accuracy was 88.5%, versus the baseline’s 52.7%. However, the interleaved test data was created by segmenting and manually mixing recordings that originally contained single tasks, so it does not fully represent a naturally multitasked workday.
The team also created Codex skills from a single successful demonstration and used the same GPT-5-mini model to run the SkillLearnBench held-out tasks. TMI skills achieved 18.57% accuracy, a 30% relative improvement over the best automated baseline’s 14.29%, although the absolute success rate remained low. Before deployment, personally identifiable information in screenshots and keyboard events must also be redacted. In addition, the library link cited in the paper is not currently publicly accessible, limiting immediate reproducibility.