推論系統
MoNe Writes Long Context into Test-Time Neural Memory, Reducing Total FLOPs for 128K Inference by 81%
Qualcomm AI Research adds per-layer fast weights to a frozen Transformer, absorbing context in 512-token segments so queries no longer need to read the source text. In 128K experiments, Qwen2.5-0.5B maintained fixed memory usage and outperformed ICL and RAG, though the evidence is currently limited to three synthetic retrieval tasks.
MoNe, proposed by Qualcomm AI Research, divides long-context processing into two stages: “memory writing” and “query answering.” The system splits documents into 512-token segments and feeds them sequentially into a frozen Transformer. Each layer is augmented with a SwiGLU fast-weight network that updates memory using the key-value associations and local gradients produced at that layer, without requiring backpropagation through the entire model. After writing is complete, query tokens use the fast weights to generate a fixed number of memory tokens, which then serve as keys and values for self-attention; the original context does not need to re-enter the KV cache.
This design makes preprocessing cost scale as O(N) with context length, while the cost of each query no longer depends on N. The stored fast weights can also be reused across multiple queries or incrementally updated with new data. The paper evaluates a frozen Qwen2.5-0.5B-Instruct model: at 128K tokens, the full write-plus-inference process used 1.41GB of peak GPU memory and 149.61T FLOPs, compared with 7.07GB and 786.33T for direct ICL—reductions of roughly 80% and 81%, respectively. Although the adapter was trained only on contexts up to 4K tokens, at 128K it scored 0.96, 0.94, and 0.96 on RULER’s single-needle retrieval, multi-key retrieval, and frequent-word extraction tasks, respectively; direct ICL scored 0.28, 0, and 0.23.
The engineering value is that existing Transformers can retain their backbone weights, avoiding the need to retrain an entire architecture for linear-scaling long-context processing. However, the additional 6.4% of parameters still require offline training, and test-time memory writing also involves gradient updates. The paper reports FLOPs and peak GPU memory rather than end-to-end latency across different hardware. Its evaluation covers only one 0.5B model and three highly structured tasks, so it has not yet demonstrated that detailed semantics can be preserved in long-document question answering, codebase analysis, or multi-turn agent memory. The next steps should examine larger models, real-world workloads, memory contamination, and the stability of long-term incremental updates.