代理訓練
TurnSight Reviews Each Turn Using Tool Execution Results, Raising Qwen3-8B’s Three-Benchmark Average to 42.02
TurnSight no longer distributes final success or failure evenly across an entire tool-use trajectory. Instead, it uses subsequent tool responses to reassess decisions at each turn. The authors have released 4B and 8B checkpoints, data, and training code, although the additional teacher branches increase training costs.
Reinforcement learning for tool-using agents often provides a single reward only after a task ends, then assigns similar advantages across the entire trajectory. This obscures critical differences: an agent may select the correct tool initially but pass incorrect parameters on the next turn; final failure does not mean every step was equally wrong. TurnSight changes the unit of credit assignment to the complete interaction turn, including reasoning text, tool selection, and JSON parameters.
During training, the student model first generates trajectories as usual. A frozen reference model then receives the tool responses actually encountered by the student and recalculates the probabilities of existing tokens from three “hindsight” perspectives that look ahead by one, two, and three turns, respectively. The system aggregates token-level differences into turn-level scores. It first uses a directional majority vote across the three perspectives to filter out conflicting signals, then selects the strongest evidence aligned with the majority direction. The score is normalized across multiple rollouts for the same prompt, and a bounded tanh weight adjusts the magnitude of the GRPO advantage without reversing its original update direction.
The authors trained Qwen3-4B and Qwen3-8B on approximately 2,000 verifiable tool-use tasks from FTRL and evaluated them on FTRL, BFCL, and ToolHop. The 8B model’s average across the benchmark tables increased from 31.60 for the untrained baseline and 30.93 with GRPO to 42.02. The 4B model reached 37.51, outperforming MatchTIR’s 34.76. The repository includes a VeRL extension, Parquet data, evaluation scripts, and checkpoints for both model sizes.
One limitation is that all reported figures still come from a single experimental pipeline run by the authors. The submetric scales of the three benchmarks are also not fully consistent, so the cross-table average should be treated only as a summary. Multiple look-ahead teachers also increase training-time compute requirements. A key question for future validation is whether execution results can serve reliably as hindsight teachers in production agents involving network latency, nondeterministic APIs, and non-replayable tools—and whether the improvements extend to longer, more open-ended tasks.