AI 研究與推論
A Million Tokens Does Not Equal Usable Memory: Tests Identify a Small Number of Retrieval Heads as the Long-Context Bottleneck
An inference-time experiment on an unreleased 27B hybrid model found that it retrieved only 21.4% of injected information at roughly 1 million tokens, far below the 92.9% achieved at 33,000 tokens. Position scaling, attention temperature, and a higher-precision KV cache failed to significantly fix the problem, while retrieval capability was highly concentrated in 18 attention heads.
A model’s claim to support a million tokens does not mean it can reliably use the entire context. Developer David Davidich conducted inference-time tests on a 27B hybrid model being prepared for public release. Its architecture includes 48 gated delta recurrent layers and 16 full-attention layers using GQA. The model’s original training length was 262,144 tokens, while support for 1 million tokens was added through a training-free extension.
The test inserted a unique access code into a long Russian-language text. At roughly 33,000 tokens with 14 similar records, the model achieved a 92.9% retrieval rate. This fell to 64.3% at 250,000 tokens and just 21.4% at 1,005,000 tokens. The author explains this as extreme-value competition: the target token must outperform a growing number of distracting positions as the context expands, amplifying even very small changes in logit margin.
Four intuitive approaches did not work. YaRN scored 0/14 in the million-token test, while compressing or freezing portions of the RoPE positions produced only sporadic hits. Restoring the long-context attention temperature did not improve the results. Replacing the 3-bit KV cache with FP8 increased performance from 60% to 70% across 40 tests, but the paired test yielded only p=0.34—insufficient evidence that quantization noise was the primary cause.
The more revealing result came from internal circuit analysis: of 384 attention heads, 18 accounted for half of the retrieval mass, and roughly three-quarters of retrieval was concentrated in a single GQA KV group. This aligns with prior research describing retrieval heads as fewer than 5% of all attention heads. It also narrows the next step from increasing cache precision across the board to training adapters for a small number of projections.
Engineering teams should note that this remains a single unreleased model evaluated with a small number of synthetic needle tests, with no code, weights, or independent reproduction available. The findings therefore cannot be directly generalized to every million-token-context model. Once the model is released, the most important follow-up evidence will come from RULER, long-document reasoning, and real-world codebase evaluations, as well as whether targeted adapters can restore retrieval without overfitting to the evaluation format.