FACTWASH Intercepts Tone Drift Before Agents Write to Memory, Achieving 0.91 F1 in Cross-Domain Negation Detection

Agent memory systems often compress conversations, emails, or documents into shorter records, but summaries may preserve the substance of a claim while dropping qualifiers such as “allegedly,” “possibly,” or “valid at the time.” Released on August 4, FACTWASH calls this phenomenon “factwashing” and places its checks at the point of writing. The tool first identifies the source sentence corresponding to a sentence awaiting storage, then checks for changes involving negation, hedging, speaker attribution, temporal scope, and numbers or entities. Content that cannot be corrected is rejected; correctable tone drift can be rewritten; and when no reliable source alignment can be found, the content is marked as unverifiable rather than automatically allowed through.

The implementation is an Apache 2.0-licensed Python package with zero runtime dependencies. It also provides an interface that wraps the existing mem0 storage layer. A key design judgment in the research is that “closed-class” cues such as explicit negation are close to enumerable: a lexicon still achieved an F1 score of 0.91 on untuned text. Hedging and attribution patterns are “open-class,” however, and lexicon-only recall plateaued at roughly 50%. For those two categories, the researchers added an LLM witness responsible only for identifying cues, increasing recall by 17 and 15 percentage points, respectively. This component can only lower—and never raise—the final safety determination, preventing the model from independently endorsing a suspicious memory write.

The authors evaluated cue detection using more than 105,000 externally annotated sentences. On an unmodified mem0 2.0.7 deployment, FACTWASH intercepted five of eight attempted memory writes containing hedged rumors. However, the real-world memory-write evaluation used only one annotator, and agreement in a second blinded annotation pass reached a κ of just 0.47. The tool currently supports only English, and its substring rules can be bypassed through paraphrasing. Engineering teams should treat it as a narrow, auditable gate rather than a general-purpose fact verifier. The next areas to watch are multilingual support, cross-sentence source alignment, and third-party measurements of false-blocking rates in real-world memory pipelines.