Back Home

AI 研究

How Much Agent Memory Should Models Read? New Study Reveals Differences and Curation Risks

Tests by an IBM team show that a complete memory store may not outperform a small set of selected guides. Preserving experience, deciding how much to retrieve each time, and tracing the basis for curated guidance need to be designed separately.

zh-Hant

How much experience should an AI agent read each time it is connected to a long-term memory store? A new study from an IBM team reports that DeepSeek-V3.2 performed best with the full guide set, while gpt-oss-120b achieved a higher task completion rate with a small set of selected guides. For deployers, this means the amount of information stored in memory and the amount given to the model on each run should be configured separately. The results are not enough to establish a general rule for all models. Study abstract

The paper was submitted on November 19, 2026. It documents the team's experience building and deploying altk-evolve; it is not a newly launched memory feature. This open-source tool turns agents’ past execution traces into reusable text guides without updating model weights. The project also provides installation instructions and a memory retrieval interface. Paper record, project documentation

When can a small, selected set beat reading everything?

In the AppWorld simulated environment, the authors had agents write and execute Python code to interact with application APIs. The main comparison used 168 test_normal tasks; a task counted as complete only if it passed every check for that task. Each model’s guides were generated from its own training and development set traces, rather than from a shared memory store. Evaluation setup

DeepSeek-V3.2’s completion rate rose from 79.8% with no memory to 89.3% when it read all 191 guides, a gain of 9.5 percentage points. gpt-oss-120b rose from 39.9% to 51.2% with all 199 guides; using about 29 selected guides, it averaged 55.4% across two runs. Table 2

These are the authors’ task evaluation results and have not been independently replicated. Configurations were compared and selected directly on the test set, and most results were run only once. The numbers support the idea that retrieval volume is worth tuning; they do not establish an optimal setting, nor are they a local inference speed test. Measurement limitations

Memory curation can also discard the basis for a decision

altk-evolve’s public interface lets users adjust retrieval volume and returns the source task IDs for guides, making it possible to trace where a recommendation came from. That is a capability provided by the project; it does not mean the guide’s content has been verified as correct. Interface documentation

The paper records that an early curation process dropped source fields, such as the number of supporting instances, while rebuilding metadata. Another case was more direct: a frequently repeated pagination guide led an agent to assume it had retrieved all the data, causing it to miss later pages. The authors ultimately corrected the guide manually. “Support count” simply means how many experiences were merged into the same guide; it is not a vote tally for correctness. Curation and pagination cases

In deployment, evaluate storage and retrieval separately

The project currently provides tools to retrieve guides by task and configure a fixed core set alongside an additional retrieval volume. Developers can therefore retain a broader record of experience while controlling what is sent to the model each time. Public tools

A practical approach suggested by these cases is to check not only task scores but also whether guides are traceable, whether source fields survive curation, and whether new guides steer tasks that previously succeeded in the wrong direction. This is an editorial deployment recommendation drawn from the cases, not a reliability metric validated by the study.

Sources

  1. How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents
  2. How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents
  3. GitHub - AgentToolkit/altk-evolve: Self improving agents through iterations