代理記憶與評測
MemTrapBench: Adding Relevant Memories Causes Five Agent Memory Strategies to Lose at Least 10.66 Points
MemTrapBench does not test whether memories can be retrieved. Instead, it examines whether correct, relevant past experiences can lock a model into the wrong strategy or belief. Across two models, all five memory methods underperformed the no-memory baseline, while a simple prompt-based guardrail recovered only part of the loss.
Existing agent memory benchmarks typically divide the process into extraction, updating, and retrieval, treating the retrieval of relevant information as success. MemTrapBench, announced on August 20, asks a different deployment question: Even when a memory is factually correct and the retrieved result is relevant to the current problem, can it still become a reasoning anchor that causes the model to misapply an old strategy or mistake a locally valid condition for a universal fact?
The dataset contains 1,050 conversations of 18 to 40 turns each, divided into four categories: task boundaries, cognitive biases, strategy avoidance triggered by negative feedback, and distorted safety beliefs. Researchers first designed seed examples, then used GPT-5.4 to expand them and insert irrelevant turns. Automated filtering and human review were subsequently used to ensure that each final question could be answered independently. The key issue is not that the memories are outdated, but that models remain influenced by previously successful methods, criticism, or sandbox assumptions even after the relevant conditions have changed.
The study compared full-history prompting, LightMem, MemOS, SimpleMem, and EverMemOS. Gemini-3-Flash-Preview scored an average of 85.16% without memory, while the best memory strategy, EverMemOS, reached only 71.17%. Qwen3-30B-A3B-Instruct-2507 scored 81.83% without memory, while the best-performing memory method, LightMem, achieved 70.13%. The effect was particularly pronounced on safety questions: after memory was added, the best score for either model remained below 70%. In a control experiment, scores recovered when the deliberately planted traps were removed while the rest of the conversation history was retained, indicating that the decline was not caused solely by longer context.
The team also introduced AdaptiveMem, which uses a prompt to require the model to reassess the scope in which a memory applies. Across a 200-question sample, it improved Gemini's scores by 11.3 to 14.9 points when paired with three memory frameworks, while Qwen improved by 2.5 to 4.2 points. No regression was observed on LongMemEval. However, this remains a manually synthesized stress test, the primary evaluator was GPT-5.2, and the GitHub repository currently contains only an initial README. Engineering teams should treat whether to use a memory as a decision that requires calibration. The next questions are whether the full dataset will be released, whether human evaluation will confirm the results, and whether similar failures can be reproduced in real customer-service interactions or long-running agent logs.