AI 代理與評測
Echoverse Trains Computer Agents on Resettable, Stateful Websites, Boosting 9B Model Evaluation to 67.1%
Microsoft’s Echoverse moves away from large numbers of shallow website replicas, instead building synthetic applications with database state, causal consequences, and verifiable tasks. After training on 12 environments, a 9B model improved from 36.5% to 67.1% across 14 evaluation splits, although the public release includes only four of those environments.
Computer-use agents need to repeatedly click, write data, observe the consequences, and learn from failure, but real websites are often constrained by logins, privacy concerns, changing data, and irreversible actions. On July 30, Microsoft Research unveiled Echoverse, shifting the unit of training from “pages that look like websites” to resettable, stateful applications. Each environment includes a FastAPI backend, a Vite/React frontend, SQLite state, tasks, and validators that score performance based on database outcomes.
Echoverse distinguishes between two types of environments. Full-domain environments preserve long-range causal structures such as shared discussion threads, permissions, audit logs, and the consequences of write operations. Capability environments, meanwhile, repeatedly train individual controls—such as date pickers and nested filters—across a wide range of layouts. The latter also include unseen component families and layouts, making it possible to distinguish whether an agent has learned general interaction capabilities or merely memorized specific screens.
The system also places models and environments in a joint co-evolution loop. The scores from each rollout are used both to repair tasks, environments, and verifiers and to provide supervision or reinforcement-learning signals for the next round. According to the paper, after training on 12 environments, a 9B model improved from 36.5% to 67.1% across 14 evaluation splits. When rewards combined database validation with step-by-step judging, the held-out test score increased from 58.8% to 68.0%. Shallow environments even reduced accuracy on real websites from 80% to 75%, showing that the volume of synthetic data alone is not enough to guarantee transfer.
The public repository is licensed under the MIT License, but releases only four environments—EchoStay, EchoForge, a date picker, and a nested filter—with a combined six splits and 722 tasks. Not all 12 environments used in the full experiments have been made public. Write tasks are evaluated by comparing database differences, while text answers are still handled by an LLM judge compatible with the OpenAI API, meaning scoring is not fully deterministic. Engineering teams should next watch whether the unreleased environments, training trajectories, and model updates are made available, and whether external agents can reproduce the paper’s claimed transfer gains on real websites.