模型訓練
IAR Splits Document Internalization Into Injection, Question-Answer Alignment, and Weight Merging, Preserving More General Capabilities on Average
A Beijing Academy of Artificial Intelligence research team has proposed a three-stage post-training method that enables models to answer questions about a fixed document set without RAG. IAR outperformed standard SFT across all metrics in seven of eight experiments, although the results still depend on synthetic question-answer pairs, model judges, and model-specific hyperparameter tuning.
The Beijing Academy of Artificial Intelligence has introduced IAR (Inject, Align, Recover), an attempt to solve a problem distinct from RAG: documents appear only during training, and at inference time the model receives only a question, requiring it to retrieve the relevant knowledge from its weights. The [paper](https://arxiv.org/abs/2608.20281) argues that SFT performed directly on document-derived question-answer pairs is limited by the coverage of the question generator. Continued pretraining can expose the model to the full text, but does not necessarily turn that knowledge into usable question-answering behavior. Both approaches may also cause catastrophic forgetting.
IAR therefore divides the process into three stages. Inject transforms documents into three types of training objectives: prefix continuation, full-text reconstruction from a summary or outline, and instruction-conditioned reconstruction. Align then establishes a question-answering interface using supervised loss calculated only on answer tokens. Finally, Recover merges the domain model with the original instruction checkpoint. The researchers test established methods including SLERP, task arithmetic, TIES, and DARE, then select checkpoints based on domain accuracy and guardrails measured by IFEval, MMLU, and MSBench.
The team tested Llama-3.2-3B, Phi-4-mini, Qwen3-4B, and SmolLM3-3B on Common Corpus and the Chinese CCI corpus. Compared with standard SFT, IAR simultaneously improved all four metrics in seven of the eight dataset-model combinations. On average, domain question-answering accuracy rose by 3.6 percentage points, while the mean score across the three general-purpose evaluations increased by 12.1 points. After controlling for training tokens, IAR beat BudgetMatch SFT in 14 of 16 comparisons. However, CC Llama still traded 3.9 points of domain accuracy for an 11-point increase in its average general-purpose score, showing that “recovery” is not a free gain.
The engineering implication is that when a document collection is stable and retrieval latency or privacy costs are too high, knowledge exposure, answering ability, and general capabilities can be optimized separately instead of merely increasing the number of SFT epochs. The [Hugging Face paper page](https://huggingface.co/papers/2608.20281) also shows that the best Inject recipe varies by model and corpus. At present, there is no publicly available implementation that can be independently verified. The test sets are generated from the documents, some answers are evaluated for correctness by an LLM judge, and parametric knowledge updates, source attribution, and frequently changing corpora may remain more difficult to handle than with RAG. The next questions are whether the code and data splits will be released, and whether third parties can reproduce the results on larger models and real-world enterprise documents.