開源代理工具
OKF Agent Memory Stores Agent Memory in Git, but Independent Testing Finds Search Latency and Ranking Stability Issues
The new open-source tool uses Markdown, YAML, MCP, and local text retrieval to make coding agents’ long-term memory reviewable through Git. Its lightweight architecture works, but independent testing found that the “under 300 microseconds” claim applies only to extremely small datasets and that the same query may return results in different orders.
OKF Agent Memory 0.1 stores architectural decisions, debugging conclusions, and operational knowledge in a repository’s `knowledge/` directory. It uses Markdown with YAML frontmatter to record sources, `generated`/`verified` trust levels, and expiration times. The pure Go CLI can create, validate, and search concepts, while exposing tools including `search`, `show`, `validate`, and `create` through stdio MCP. This allows Claude Code, Cursor, or Codex to read and write cross-session memory without connecting to a vector database.
The project advocates progressive disclosure instead of stuffing the entire knowledge base into the prompt, and claims search latency below 300 microseconds and memory usage below 15 MB. An independent audit confirmed that the software could be compiled, pass its tests, and complete an MCP handshake, while measuring 11.9 MB RSS. Its example’s reduction from 3,034 tokens to 603 tokens—about 80%—also checks out mathematically. However, this mainly reflects loading only one relevant document, rather than compression produced by the retrieval algorithm itself.
Performance problems emerge as the system scales. Using the same function on synthetic knowledge bases, the auditor measured median search latency of 35–43 milliseconds for 500 concepts, 141–148 milliseconds for 2,000 concepts, and 337–370 milliseconds for 5,000 concepts. The reason is that the current implementation has no inverted index: every query re-tokenizes every document and scans the full contents for each term using `strings.Contains`. The so-called BM25 implementation also lacks `k1`, document-length normalization, and term-frequency saturation, making it closer to TF-IDF with field weighting and BM25 IDF.
Another correctness issue stems from Go map iteration’s randomized order and an unstable sort. When scores were tied, the auditor obtained 15 different top-three orderings across 15 runs. This undermines the project’s core promise of “reproducible memory.” The good news is that graph validation still takes only about 1.3 milliseconds with 5,000 concepts, while the format, provenance tracking, and Git review design are practically useful. For now, deployers should limit the tool to small, manually curated knowledge bases and wait for an inverted index, a stable tie-breaker, and independent retrieval-quality testing.