模型訓練與評估
LittleLearner Limits Pretraining Knowledge to a Fifth-Grade Level, With Few-Shot Prompting on Advanced Math Still Stuck at Around 6%
Researchers filtered 88 billion tokens of K–5 material from FineWeb‑Edu and trained 0.6B, 1.3B, and 5B models from scratch alongside compute-matched control models. Experiments show that scaling the models, applying SFT＋GRPO, and providing in-context examples primarily elicit existing capabilities rather than clearly pushing beyond the pretraining knowledge boundary.
Whether language models genuinely “learn” new capabilities has long been obscured by the opacity of pretraining data: higher post-training scores may simply reflect the elicitation of previously latent knowledge. LittleLearner takes the opposite approach by controlling the source data. It constructs LittleCurriculum, an 88-billion-token dataset derived from FineWeb‑Edu that retains only content at the U.S. kindergarten through fifth-grade level. The pipeline combines age-of-acquisition vocabulary data, an LLM classifier, curriculum rules, and symbol filtering. A scan for n-grams representing 126 groups of concepts above the fifth-grade level found matches in just 0.09% of retained passages.
Using the Qwen3 architecture, the team trained models with 0.6B, 1.3B, and 5B parameters from scratch. For each size, it also created a control model with the same architecture, token count, and training recipe but trained on unfiltered data. Training the 5B model took about 100 hours on eight B200 GPUs. MathCAMPS evaluations showed that increasing the parameter count improved performance on K–5 problems and nearby sixth- and seventh-grade material, while eighth-grade performance remained near the floor. After natural-language examples were added to the 5B chat model’s context, K–5 accuracy rose from 34.0% to 36.8%, whereas accuracy above the fifth-grade level edged down from 6.0% to 5.9%. Examples using concise algebraic notation or question-and-answer pairs alone performed even worse.
Results from SFT followed by GRPO were similarly limited: even when the restricted model underwent post-training on out-of-scope math problems, it failed to narrow the gap with the unfiltered control model. This makes LittleLearner a cleaner sandbox for studying whether RL can genuinely create capabilities, interference in continual learning, and model knowledge calibration. The model and control weights are listed on the [project page](https://littlelearner-ll.github.io/), but the findings currently cover only models up to 5B parameters, specific math tasks, and a limited post-training budget. Moreover, “outside the curriculum” does not mean the corpus is entirely free of leakage, so the results cannot be directly extrapolated to frontier models.