模型訓練與長上下文
Long-Context Training Shows an Inverted-U Curve: Performance Weakens Once the Window Exceeds Task Length by About 16–32×
New research suggests that longer training windows may cause models to rely on context rather than encode knowledge in their parameters. Both pretraining experiments with models up to 750M parameters and Qwen3 fine-tuning experiments show that, as supporting information increases, robustness in no-context and conflicting-evidence settings declines.
Long context is usually viewed as a one-way expansion of capability, but a Johns Hopkins University team proposes an “information abundance paradox”: when training sequences repeatedly provide enough evidence to solve a task directly, a model can use attention to retrieve the relevant information from the current context, reducing the pressure to internalize patterns in its weights. This is not merely the lost-in-the-middle problem at inference time; the training-window length changes which solution strategy the model learns to use.
The team first varied the pretraining window for models with up to 750M parameters, then evaluated them on language modeling, SuperGLUE, and 11 closed-book multiple-choice question-answering datasets. All three metric categories followed an inverted-U curve: increasing the window helped initially, but performance declined after passing an intermediate optimum. The optimal windows for Penn Treebank, LAMBADA, and WikiSPAN were 512, 2,048, and 8,192 tokens, respectively; MCQA and SuperGLUE both began to weaken at 2,048 tokens. Across five dataset groups, the turning point was roughly 16–32 times the mean sample length. However, this is an empirical pattern observed under discrete experimental settings and should not be treated as a universal configuration formula.
In another set of experiments, the researchers used LoRA to fine-tune Qwen3 models ranging from 0.6B to 14B parameters. Each question was paired with eight documents, while the number drawn from the target domain was varied among zero, four, and eight. As the number of target documents increased, models became more accurate when given supportive context but more fragile when no context was provided or when documents were deliberately written to support an incorrect answer. Module-level gradient analysis showed that information-rich training shifted the relative update pressure from feed-forward networks toward self-attention. Experiments that restricted updates to only one module type, along with interventions on token attention, also supported the interpretation that models were shifting from parametric memory to contextual retrieval.
For engineering teams, long-window data recipes should not be evaluated solely through long-context perplexity or needle-in-a-haystack tests. Evaluations should also include closed-book tests, incomplete-retrieval scenarios, and conflicting-evidence tests. The study’s main limitation is that its pretraining experiments covered models only up to 750M parameters. Comparisons between the short- and long-context variants of Phi-3 and OLMo 3 were included only as motivating observations and could not control for every training difference. The authors have released the synthetic tasks, tokenizer, and small-model artifacts on Hugging Face. The next step is to reproduce the location of the turning point using larger models, equal token budgets, and genuinely long documents.