Even a Frozen Qwen3-8B Shows “Capability Expansion,” Prompting Researchers to Call for Empirical Null Baselines in Self-Improvement Evaluations

Determining whether a model has “self-improved” requires more than comparing its average accuracy before and after training. Recent studies often go further by tracking whether each item changes from incorrect to correct or from correct to incorrect. But these per-item transitions are actually differences between two noisy estimates. Researchers from University College Dublin and other institutions reran the same pipeline using Qwen3-8B, three rounds of rank-32 LoRA training, and a fully frozen control model, identifying seven measurement failures capable of reversing the conclusions.

The most immediate problem is that temperature 0 does not guarantee bitwise-deterministic batched inference. When the frozen model was compared with itself, a single greedy-decoding run still produced six items that appeared to be “newly learned” and nine that appeared to be “degraded.” Even after serializing the requests, roughly 2% of classifications still flipped. Another commonly used metric, the expansion rate, treats an item as a capability expansion when the base model fails all 128 samples but the trained model succeeds at least once. This metric gave the untrained model a score of 0.280. Raising the success threshold did not solve the problem: across 110 frozen-model comparisons, the measured null value remained 0.058, with a 95% interval of 0.038 to 0.078.

The team instead used an exact test for each item, comparing the trained model against a baseline pooled from multiple independent evaluations, and then controlled the false discovery rate. This method detected no changes in any held-out frozen replica. When applied to three forms of self-training and distillation from an external teacher, the teacher improved 8 to 11 of the 22 items that the base model answered correctly only rarely, while self-training improved just 0 to 2. For the ten items that the base model never answered correctly during sampling, the statistical evidence remained insufficient.

This does not mean self-training has no value. Across all seven training arms, every item that later reached pass@1 had already been answered correctly by the base model at least once in 128 samples, showing that these methods can “sharpen” low-probability correct answers into reliable outputs. But they can also damage previously demonstrated capabilities, while truncation rates, batch configurations, and training seeds can masquerade as capability changes. Future evaluations should retain multiple checkpoint 0 replicas, repeatedly sample each item, and report a separately measured noise floor for every transition statistic.