代理系統／記憶工程
Agents May “Carry Their Memory but Forget the Facts” After a Model Swap: Fixed Structures Transfer Better Than Free-Text Summaries
A controlled study shows that retaining an agent’s existing memory store does not guarantee that an upgraded model can still use it correctly. Fixed-schema knowledge graphs were almost unaffected by changes to the writing model, while natural-language summaries and indexes mixing old and new embeddings could silently degrade.
Model upgrades require more than rerunning capability evaluations: external memory should also be treated as a version-dependent runtime component. A study added to arXiv’s new submissions list on September 7 stored the same histories in four formats—full raw text (LC-RAW), chunked retrieval (RAG), model-compressed natural-language notes (NOTES), and fixed-schema subject–relation–object knowledge graphs (KG-fixed)—and then replaced the models responsible for reading or writing the memories.
The experiment used 48 synthetic histories, each containing 160 questions with exactly verifiable answers. Answer codes were randomized to prevent models from guessing based on pretraining data. The models were fixed to Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct-1M. After the model writing the notes was replaced, KG-fixed accuracy changed by only `+0.0004 ± 0.0020`. NOTES, however, changed by `+9.91` or `-13.28` percentage points depending on the migration direction. This suggests that summaries’ content and phrasing become coupled to the model that generated them—and that testing migration in only one direction is insufficient to predict performance in the reverse direction.
RAG presents risks at a different layer. The study upgraded BGE Large English v1.0 to v1.5, which also produces 1,024-dimensional vectors. Although embeddings from both versions can be stored in the same index, they do not belong to the same representation space. A mixed index in which only half the data was recomputed delivered an improvement of just 4.96 percentage points, compared with 11.90 points after a complete rebuild. The system raises no dimensionality error, even though retrieval quality has already degraded.
The recovery results have even clearer operational implications. Rewriting memory using only the existing NOTES failed, across all 48 test sets, to recover at least 90% of the performance achieved when the new model built its own memory. When the original histories had been retained, 34 test sets reached that threshold in one migration direction. Engineering teams should therefore preserve traceable source text, isolate different embedding versions, and include writer, reader, and indexer versions in their migration matrices. However, this remains a preprint involving two sub-10B models and synthetic histories. The advantage of fixed schemas also stems partly from knowing the task fields in advance, so the findings cannot yet be directly generalized to open-ended personal assistants.