模型訓練與資料
The Chapter Organization of Synthetic Textbooks Can Itself Improve Mid-Training, Raising the Average Score by 1.02 Points Even With the Text Held Constant
A new study generated 686,000 synthetic textbooks totaling 32 billion tokens and isolated the effect of document structure through controlled experiments using identical content. Organizing related chapters into complete books outperformed splitting them into separate passages or concatenating them randomly, but the gains still come from the research team’s custom training pipeline, and neither the data nor the code has been released.
Discussions of synthetic training data typically focus on text quality, rewriting prompts, and teacher models. A new study instead treats “how documents are organized” as an independent training variable. The team first retrieved source material from a pretraining corpus, clustered it by topic, planned hierarchical tables of contents, and assembled source-grounded chapters. The process ultimately produced 686,000 synthetic textbooks comprising roughly 32 billion tokens across more than 15,000 disciplines.
The key result is not a simple comparison between synthetic and natural text. The study created a Split control that preserved exactly the same generated text and tokens but separated each chapter into an independent document. The complete-book version still achieved an average downstream score 1.02 points higher, indicating that the gain cannot be attributed solely to content. RandomConcat joined chapters from different textbooks into documents of similar length, yet still underperformed the complete-book version, ruling out the simple explanation that longer documents are responsible. Compared with a Rephrase pipeline that used the same retrieval pool but rewrote each document individually, the full clustering, table-of-contents planning, and book-assembly pipeline led by an average of 1.17 points. When it replaced natural books in the mid-training recipe, the average improvement was 1.09 points.
The findings suggest that the ordering, repetition, and dependencies of concepts across chapters may affect how models learn long-range knowledge structures. For data engineering, document boundaries may no longer be merely a packaging and I/O detail; they may also represent a quality dimension that can be tuned alongside the data mixture and sequence length.
However, the reported scores are average differences across multiple evaluations and may not reflect consistent gains for every capability or language. The generation pipeline could also propagate a teacher model’s biases and errors across entire textbooks. Public corpora, generation code, and external reproductions are currently unavailable. Engineering teams should wait for per-benchmark results, contamination checks, and scaling results across different model sizes before deciding whether the added clustering and generation costs are worthwhile.