MemUse: Models Achieve a 78.8% Memory Rate in Direct QA, but Just 7.9% in Natural Conversation

A Kyoto University research team has released MemUse, challenging the direct question-answering approach commonly used to evaluate long-term conversational memory. The study was based on a four-month randomized deployment in which 40 users completed 1,872 sessions, with the system rotating among seven memory configurations. Accuracy on conventional direct QA ranged from 19.7% to 70.1% across configurations, but compared with a summary-only baseline, differences in user satisfaction were all below 0.06 within-user standard deviations.

The team therefore extracted 72 moments from real conversations in which users spontaneously referred back to past events, creating MemUse, a benchmark containing 316 factual questions. Its primary metric, Natural Integration, does not separately ask, “What do you remember?” Instead, it checks whether the model naturally incorporates relevant past information when responding to the original conversational prompt. Direct QA tests whether the model can produce the answer when explicitly asked, while Reference measures whether the natural response actually cites the same fact.

The gap was striking: using the same GPT-4.1-mini model with full history and reconstructed context, Direct QA reached 78.8%, while natural responses referenced the relevant facts only 7.9% of the time. The per-question correlation between the two was -0.009, indicating that successful retrieval was almost entirely unable to predict whether the information would appear in the response. In sessions where users proactively invoked a memory, Natural Integration was positively correlated with satisfaction (ρ=0.29), whereas Direct QA was nearly unrelated (ρ=0.03). Even when the correct details were extracted before response generation, the model still failed to use the extracted information in 37 of 48 cases, suggesting that the bottleneck lies beyond the retriever alone.

From an engineering perspective, conversational memory systems should not report only recall or QA accuracy. Evaluations should also replay natural trigger utterances and measure whether retrieved content meaningfully affects the next generated response. The dataset, grading prompts, and runner have been released publicly, but the data is licensed under CC BY-NC 4.0 and is restricted to noncommercial research. The conclusions should also be interpreted cautiously: the benchmark contains only 72 positive examples; Natural Integration is judged by GPT-5.4-nano; and its relationship with satisfaction is observational. Moreover, every deployment condition already included conversation summaries. The study therefore challenges the claim that additional memory capacity necessarily improves the user experience—not the value of memory itself.