AI 程式開發
CAPA Tests Cross-Session Memory in Coding Assistants: Same-User History Raises First-Try Success by 15.6 Points on Average
CAPA embeds implementation preferences that users repeatedly omit into 600 executable coding tasks, testing whether assistants can infer new requirements from past conversations. Across 12 models, adding same-user history increased final success rates by only 6.8 points on average, but raised first-try success by 15.6 points—suggesting that memory mainly reduces clarification rounds rather than eliminating failures.
The newly proposed CAPA (Cross-Session Adaptation to Personalized Ambiguity) reframes the long-term memory problem for coding assistants from “remembering what a user said” into an executable test: when the same person repeatedly omits the same information, can a model infer their actual intent from previously resolved sessions? Using HumanEval tasks as a foundation, the researchers designed six personalized ambiguity mechanisms and created 600 conversations spanning 60 user–ambiguity configurations, with 300 conversations reserved for evaluation.
Twelve models were each evaluated under two conditions: with no history and with five previous sessions from the same user. Adding history increased the average executable success rate by 6.8 percentage points, raised the first-response success rate by 15.6 points, and reduced the number of turns required for completion by 0.81. The best-performing model, Claude Opus 4.8, saw its first-try success rate rise from 24.3% to 60.3%, yet nearly 40% of tasks still required follow-up questions or failed on the first attempt. ChatGPT-5.6-Sol’s final success rate even fell slightly, by 0.3 points, showing that additional context does not deliver consistent gains.
The study also used histories from other users as a control. Although mismatched histories still supplied general coding and conversation examples, correctly matched histories raised first-try success by an additional 2 to 12 points across the three models tested. General-purpose memory systems Mem0 and A-mem did not consistently outperform directly including the raw history. The researchers’ parameter-free history gate first determines whether past conversations contain consistent evidence linking an ambiguity to its resolution, then decides whether to answer directly or request clarification. It improved first-try success by as much as another 13.33 points.
The key engineering lesson is not to permanently pack every conversation into the context window, but to preserve structured evidence of how ambiguous requirements were clarified—and to verify identity, relevance, and consistency before using it. CAPA still consists of synthetic tasks derived from HumanEval and does not cover large repositories, changing requirements, or accounts shared by multiple people. The next questions are whether the data and code will be released, and whether the results can be reproduced using real-world IDE telemetry under privacy constraints.