AI 基礎設施
MISA-T Schedules Mixed RL Rollouts by KV Residency Time, Boosting Step3.7 Throughput by 53.3%
Rather than selecting inference nodes based solely on prefix hits, MISA-T also limits how many sessions from different workloads can occupy the KV cache concurrently. The authors measured rollout throughput gains of 43.6% to 53.3% on Step3.7 and Qwen3.6-35B-A3B, but the implementation has not yet been released.
Reinforcement learning post-training for large models is increasingly sending RLVR, RLHF, and multi-turn agent trajectories that invoke tools through the same asynchronous inference service. Existing prefix-aware routers typically dispatch a request to the node with the longest matching prefix already cached, but do not control how many new sessions can enter the limited KV cache. Under high concurrency, long-resident agent trajectories may evict other prefixes, leading to repeated prefills, longer queueing times, and more cache evictions.
A team from Nanjing University and StepFun has proposed MISA-T, which separates “where to place a request” from “when to admit it.” The routing layer dynamically adjusts the cap on new sessions according to cache pressure, then allocates protected capacity based on the respective KV footprints of RLVR, RLHF, and agent workloads. Compute demand is estimated using not only token counts, but also KV blocks multiplied by residency time. When an agent is waiting for an external tool response and the model is temporarily idle, that period still counts toward its occupancy cost. Continuations of existing sessions are preferentially retained, while new work can be deferred and reevaluated.
Compared with vLLM Router after a sweep over concurrency parameters, MISA-T increased the completed-sample rate in rollout-only tests by 53.3% for Step3.7 and 43.6% for Qwen3.6-35B-A3B, while maintaining prefix hit rates of 97.8% and 95.3%, respectively. Across 50 rounds of end-to-end training with Step3.7, rollout throughput rose by 35.6% and average iteration time fell by 22.8%. The deviation between the completed workload mix and the trainer’s target also decreased from 4.14 to 2.71 percentage points, while task scores remained comparable.
These results highlight that the bottleneck in mixed rollouts may not be individual inference requests, but rather the implicit commitment each session makes against future KV capacity. However, the figures come from only two models and the authors’ own deployment environment. MISA-T also requires requests to carry reliable workload-class labels and depends on timely snapshots of cache state; delayed snapshots or sudden shifts in workload distribution could temporarily make the admission cap inaccurate. The paper has not yet provided code that can be integrated directly with vLLM Router. The next things to watch are a public implementation, cross-cluster reproductions, and whether the system can continue operating without retuning when models are updated or tool latencies change.