AI 評測與教育應用
TutorMoments Replays Real Tutoring Decisions, Finds Seven LLMs Default to Over-Scaffolding
Ai2 built a replayable evaluation from 462 real math tutoring transcripts, asking models to choose between providing scaffolding and requiring deeper student reasoning. Explicitly prompting for this tradeoff improves the performance of all seven models, but simulated students and model-based grading still cannot demonstrate real learning outcomes.
Ai2 has released a preview of TutorMoments, shifting AI tutor evaluation from whether a model produces the correct answer to whether it intervenes at the right time. The dataset contains 462 de-identified transcripts of one-on-one math tutoring sessions with U.S. students in grades 2–7. Twenty-seven experienced teachers identified more than 1,500 critical moments and judged whether the tutor should provide scaffolding, increase cognitive demand, or avoid giving too much help. The executable evaluation is frozen as a balanced set of 520 scenarios: 260 focused on scaffolding and 260 on rigorous reasoning.
During evaluation, the system truncates a real conversation at a critical moment and lets the model under test take over as the tutor for five turns, while another language model plays the student. A model-based classifier calibrated against teacher annotations then calculates three metrics: whether the tutor provides appropriate scaffolding when help is needed, whether it promotes deeper reasoning when a challenge is appropriate, and whether it avoids oversimplifying the task. Initial tests covered seven LLMs. When prompts merely instructed them to “teach well,” the models generally provided too much assistance and rarely required students to reason independently. Scores improved across all models when prompts explicitly described the tradeoffs among scaffolding, over-scaffolding, and rigor, but their strategies remained less varied than those of human tutors, often repeatedly asking students to explain their answers.
For developers, TutorMoments is valuable because it turns the policy choices made by tutoring agents into replayable tests that can be pinned to specific dataset revisions. The open-source runner records hashes for the model, prompt, and dataset contents, and outputs per-scenario conversations, scores, and summaries, making it easier to compare behavioral drift caused by prompt or model updates. However, it measures simulated decisions rather than learning outcomes: the student is simulated by an LLM, and grading also depends on a model-based classifier. The human baseline is deliberately concentrated on excerpts that the original teachers may have handled poorly, so it cannot support claims that AI has outperformed teachers. The dataset also currently covers only K–12 math contexts in a single country.