代理基礎設施／推論排程
Delaying the Submission of Ready Turns in Agentic Workflow Scheduling Cuts P95 Completion Time by Up to 71.4% Under Congestion
New research separates a turn becoming ready from its submission to the inference engine, using CVaR tail risk and a dynamic workload budget to determine submission order. SWE-bench trace replays achieved up to a 3.5× P95 speedup, but the results cover only fixed agent trajectories, a single randomized arrival seed, and an A100/vLLM environment.
A [new paper](https://arxiv.org/abs/2609.10964) submitted on September 10 argues that latency in agentic workflows arises not only inside inference engines such as vLLM, but also before requests enter the engine. A typical agent executor submits the next model turn as soon as a tool finishes and the turn becomes ready. This is reasonable under light load, but during congestion, large numbers of “submitted but not yet completed” turns accumulate inside the engine, preventing the workflow scheduler from reordering them.
The researchers propose a scheduler that deliberately separates readiness from release. Its objective function combines mean completion time with Conditional Value at Risk (CVaR): once a workflow’s age crosses an online estimate of the 95th-percentile threshold, its priority increases. This latency weight is then divided by the estimated workload of the current turn. Workload is calculated from prompt tokens and an output-length estimate derived solely from completed turns, without looking ahead at the current turn’s actual output.
A second control loop manages a “committed work” budget. The scheduler monitors mean queueing delay, P95 queueing delay, and the number of waiting requests. As congestion rises, it reduces the token-equivalent workload that may be admitted to the engine; when capacity becomes available, it relaxes the limit. If a turn waits longer than 180 seconds, an anti-starvation rule gives it priority. The approach does not modify vLLM’s internal scheduler. Instead, it adds an admission-control layer between the agent executor and the shared inference service.
The experiments used vLLM 0.20.2 to replay mini-swe-agent trajectories from SWE-bench and SWE-Gym, with 100 workflows and approximately 1,600 model turns from each dataset. Qwen3-8B, Qwen3-32B, and Llama-3.3-70B ran on one, two, and four 80GB A100 GPUs, respectively. At the lowest load, the new method’s P95 ratio relative to immediate submission ranged from 0.99 to 1.00. In the strongest result, it reduced P95 completion time for Llama-3.3-70B from 1,986.3 seconds to 568 seconds—a 71.4% reduction. Ablation studies indicate that tail-risk-based ordering produced most of the gains, while the dynamic budget contributed an additional improvement of roughly 10.7% to 11.6%.
These findings do not yet support broad conclusions about production deployments. Each data point used only one fixed arrival seed, while P95 was determined by a small number of samples among just 100 workflows. The replayed tool delays and turn contents also did not change in response to scheduling outcomes as they might in real agents. The next step is to validate the method in online environments where multi-tenancy, model routing, prefix caching, and failure retries coexist, while examining whether delayed submission compromises service levels for individual short tasks or interactive requests.