AI research
One-Shot OPD Captures Most Full-Data Distillation Gains With a Single Query, Shifting the Bottleneck to Teacher-Signal Absorption
Researchers found that repeatedly sampling from a single query while giving the student model token-level teacher supervision can, after 300 steps, cover 71.5% of the state reached by full-data training. Adding 16 semantically diverse queries raises coverage to 98.9%, though the findings remain limited to the specified small models, teachers, and benchmarks.