代理訓練
SMRC-SD Verifies Agent State Before Distilling Successful Trajectories, Improving Qwen3-1.7B Task Success Rate by About 12 Percentage Points
Successful demonstrations may not apply to states reached after an agent takes a detour, and unconditionally providing the full trajectory can even reduce the probability of the correct action. SMRC-SD adds a self-distillation signal only when the state is compatible with a demonstration step, outperforming the full-path baseline on both ALFWorld and WebShop.
Every action taken by a multi-turn agent changes the subsequent environment. If the student completes subgoals in a different order, the “next step” in a successful demonstration may not be executable in the current state. SMRC-SD calls this issue state–reference mismatch and introduces a deterministic state router before privileged on-policy distillation.
The system first reconstructs state signatures from action prefixes in demonstration trajectories. It then compares them with the student’s current location, inventory, object attributes, or shopping-page progress and verifies that the candidate action remains in the set of executable actions. When a compatible position is found, the teacher receives the full path, a summary of the current state, and the grounded candidate next step. If none is found, the episode still receives the terminal GRPO reward, but no path-conditioned distillation loss is added. This additional information is used only during training; after deployment, the model does not require the state router.
In tests with Qwen3-1.7B, ALFWorld Average@4 increased from 0.746 with unconditional FullPath-SD to 0.865, while WebShop binary success rose from 0.574 to 0.693. Qwen2.5-3B also improved, although its WebShop success rate edged up only from 0.734 to 0.736. The evaluation covered a fixed set of 128 tasks in each environment, with four rollouts per task. The released codebase includes training scripts, reference paths, and fixed anchor probes.
An engineering limitation is that the state signatures and compatibility rules are still manually written for each environment; they are not a general-purpose state estimator that can be transferred directly to arbitrary browser-based or enterprise agents. The WebShop experiment also uses a text-only subset containing just 1,000 products. Future work should examine whether reliable matching can be maintained under semantic ambiguity, partial observability, and real-world website states, as well as whether routing errors might filter out supervision that would otherwise have been effective.