代理系統與推論
Same Model Doubles as Tool-Call Predictor, Boosting Qwen 4B Next-Step Hit Rate by More Than 17 Points
Researchers enabled agents to predict and pre-execute their own next tool calls during inference, replacing an external draft model with shared weights and a shared KV cache. Joint reinforcement learning raised average Hit@1 to 61.2 and 66.3 for two Qwen 4B models, respectively, though the paper has yet to quantify end-to-end speedups in real-world parallel deployments.
Agent latency does not come solely from model generation; search, databases, code executors, and sub-agents often leave entire workflows waiting on network or service responses. A team from the University of California, Santa Barbara and LinkedIn has proposed “self-speculative agents,” which attempt to predict and execute the next structured tool call before the model finishes its current reasoning. If the tool name and arguments exactly match the call the agent later produces, the system can immediately use the cached result.
Existing approaches typically deploy a separate small draft model, but experiments show that smaller models from the same family do not necessarily understand the target agent’s decision distribution. On MuSiQue with Qwen3-4B, 0.6B and 1.7B draft models achieved Hit@1 scores of only 4.3 and 14.7, respectively, while having the 4B model predict its own next step reached 25.3. External models also require additional weights and a second KV cache: in tests using SGLang on a single H100, the paper reports that the Qwen3-0.6B draft setup consumed 10.71 GB, compared with 8.70 GB for self-speculation.
During training, the same model is updated alternately in agent and speculation modes. Speculation targets are drawn from rollouts just generated by the current policy rather than from stale trajectories. After adding stabilization measures—including supervised warm-up and resetting the optimizer when switching modes—Qwen3-4B’s average next-call Hit@1 rose from 44.1 to 61.2, while Qwen3.5-4B improved from 48.9 to 66.3. Task success rates on search-based question answering and τ-bench remained broadly stable.
The engineering value of this design is that it eliminates the need to keep a second model resident and allows execution to branch from a shared prefix while reusing the KV cache. However, Hit@1 does not directly translate into lower real-world latency. Incorrect predictions can waste external API calls and compute, or even cause irreversible side effects. The paper also analyzes off-the-shelf predictors using only 50 queries per domain and does not yet report results from a parallel serving system with cancellation, idempotency, and cost controls. The next things to watch are whether the code and model weights are released, and whether the approach delivers net benefits with slow, expensive, or stateful tools.