模型訓練研究
Auxiliary Views Experiment: Recasting the Same Knowledge as Textbooks, Q&A, and Articles Helps Models Learn Better Than Repeating the Original Text
Under a fixed knowledge-token budget, University of Pennsylvania researchers replaced some repetitions of original texts with multiple conceptual representations, improving factual recall and reasoning. The gains appeared mainly in models with 7B parameters or more, but the experiments involved only short-run continual pre-training and cannot be directly extrapolated to the full pre-training of frontier models.
“Data diversity” usually describes whether data comes from a sufficiently broad range of sources, but it does not address how the same piece of knowledge should be presented. A new study, [Knowledge Acquisition During Pre-training?](https://arxiv.org/abs/2609.04180), separates two kinds of variation: paraphrases change only the wording, while auxiliary views reorganize the same document into textbook chapters, Stack Exchange-style Q&A, and blog posts, adding explanations, analogies, and different contexts.
The team selected 36 documents that were not present in the base models’ existing training corpora, spanning computer science papers, U.S. federal appellate court decisions, and medical cases. It then conducted 100 knowledge-injection steps on the 1B, 7B, 13B, and 32B variants of OLMo-2. The total number of knowledge-related tokens seen under each condition was held constant, so adding auxiliary views meant reducing direct repetition of the original text rather than simply increasing compute. The evaluation included 6,435 factual fill-in-the-blank questions, 430 reasoning fill-in-the-blank questions, and multiple-choice versions of both. The data and stable IDs are available on [Hugging Face](https://huggingface.co/datasets/jiosephlee/auxiliary-views-knowledge-acquisition).
The results show a consistent performance ranking: repeated original text, nine paraphrases, and paraphrases plus auxiliary views, in ascending order. In an OLMo-2 7B experiment designed to more closely resemble original pre-training, factual multiple-choice accuracy reached 0.403 with auxiliary views, compared with 0.372 for the original-text condition; reasoning accuracy was 0.492, compared with 0.421. The same ordering also appeared with Qwen-2.5-7B. The researchers additionally used 11 sets of generators to create auxiliary texts. They found no positive correlation between generator size or the generator’s own question-answering accuracy and downstream results. Instead, the amount of generated text showed a moderate correlation with performance. This makes the technique look more like data augmentation than distillation of answers from a strong teacher into a student model.
The limitations are equally important. The 1B model gained almost no benefit, and the advantage of paraphrasing alone disappeared as batch size increased. The so-called “pre-training-faithful” control merely resumed training from step 925,000 for another 100 steps; it was not training from scratch. The test using human-written auxiliary views covered only two documents. Engineering teams could begin by comparing “document repetition” with “conceptual restatement” in domain-specific continual pre-training, but the next step is still to test long-tail specialized knowledge, non-English data, contamination controls, and whether the gains persist in models with tens of billions of parameters or more and across full training cycles.