推論系統
ParaTempo Asynchronously Prunes Reasoning Branches Using Temporal Confidence, Cutting Qwen3.5 Latency by 21.8%
ParaTempo periodically probes the provisional answer distributions of individual reasoning paths, then independently prunes, retires early, or forks them. On four benchmarks, Qwen3.5-35B-A3B averaged 1.1 percentage points lower accuracy than fixed 16-path self-consistency, while reducing latency by 21.8% and generated tokens by 30.3%.
Parallel sampling of multiple chain-of-thought paths can improve reasoning reliability, but typically gives every path the same token budget—even when some branches have already converged or continue to fluctuate. The [ParaTempo paper](https://arxiv.org/abs/2608.16425) reframes this as an online resource-allocation problem: after every 500 generated tokens, it appends a suffix requesting a direct answer to the current prefix, obtaining up to 20 candidate answers and their probabilities.
Rather than making decisions from a single top-1 confidence score, the system averages the answer distributions from the seven most recent probes and calculates “temporal confidence” as the exponential of negative entropy. A value near 1 indicates that recent probability mass has remained concentrated on the same answer, while a diffuse distribution suggests that the path has not yet stabilized. During warm-up, the pruning threshold is set from the confidence distribution for the current problem, avoiding a fixed threshold shared across different models.
After warm-up, low-confidence branches are removed. A branch is retired early if it maintains a probability of at least 0.9 for the same dominant answer across nine consecutive probes, although its answer remains in the voting pool. Freed execution slots are refilled by forking the highest-confidence active prefixes and continuing exploration with different random seeds. Once the weighted global vote for an answer reaches the threshold, all decoding can stop. Because paths are not required to reach the same depth, the process reduces both total computation and the latency of the slowest path.
The authors evaluated Qwen3.5-35B-A3B and GPT-OSS-20B with 16 branches using vLLM on a single 80GB A100, covering AIME 2026, two HMMT benchmarks, and GPQA. Qwen achieved an average accuracy of 71.1%, compared with 72.2% for fixed self-consistency, while using 21.8% less latency and 30.3% fewer tokens. Compared with synchronous Parallel-Probe, it improved accuracy by 3.9 percentage points and reduced latency by 10.6%.
The [public code](https://github.com/ScottZhang812/ParaTempo) includes runners and default parameters for all four datasets. However, the results cover only two models, reasoning tasks with clearly defined answer formats, and a single-GPU environment. Probing also requires the model to produce answer probabilities that can be normalized. The next engineering step is to determine whether asynchronous control delivers similar gains for free-form text, tool calling, and multi-GPU continuous batching.