代理評測
ContextWeave Re-Evaluates Agent Memory with 1,000 Executable Office Tasks, Raising Preference Score from 41.50 to 70.60
ContextWeave goes beyond asking whether agents can retrieve old information, examining whether memory improves subsequent document and office work. Experiments show that retaining complete, actionable experiences outperforms short summaries, although incorrect recollections are also more likely to affect execution results.
Language-agent memory evaluations are often reduced to whether an agent can find an answer in conversation history, without testing whether the retrieved content actually improves the next task. Submitted on August 5, ContextWeave instead evaluates this through longitudinal workflows. The research team reconstructed privacy-processed work records from 14 participants spanning several months into 1,005 executable tasks, 568 of which make up the core benchmark. Each task includes instructions, a containerized environment, an action trajectory, and task-specific grading rules, requiring agents to continue existing projects rather than answer isolated questions.
The evaluation is divided into Workspace Score and Preference Score. The former checks whether the resulting workspace completes the task, while the latter measures whether the outcome aligns with preferences previously demonstrated by a specific user. It also includes diagnostics for relevance, continuity, solvability, and robustness against misleading memories. With the base model held constant across six memory components, the best configuration raised Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60. Conversely, when the memory component was fixed, all five tested models benefited from the recall mechanism, although the magnitude of improvement varied.
Of greater interest to engineering teams, “experiential memory”—which includes operational details, reasons behind decisions, and previous results—was more effective than compressed summaries at reducing repeated exploration and maintaining continuity. The trade-off is that it was also more likely to introduce incorrect or irrelevant content into execution. This means production-grade agents cannot optimize solely for vector retrieval hit rate; they must also validate a memory’s recency, provenance, and compatibility with the task before using it. For now, the results are still based on data from 14 participants and grading rules designed by the authors, and the paper’s page does not link to a public repository. Future scrutiny should focus on whether the complete dataset, containers, and judge configurations are released, and whether other teams can reproduce the scores.