推論系統
Pandora’s Router Factors Model-Evaluation Costs Into Routing, Cutting EmbedLLM’s Combined Loss to 0.386
Google DeepMind formulates the decision of whether to run costly model evaluations as a “Pandora’s box” search problem, deciding sequentially whether an additional assessment is worthwhile based on its value of information. It achieved the lowest average routing regret plus evaluation cost across all three tests, although its effectiveness depends on calibrated signal distributions and cost settings.
A Google DeepMind research team has introduced Pandora’s Router, which addresses an expense often overlooked by existing model routers: predicting each candidate model’s answer quality can itself incur inference, retrieval, or tool-use costs. The system first uses the average score of the three nearest neighbors in prompt-embedding space as an inexpensive estimate, `f`. It then uses a small fine-tuned model, retrieval results, or the first 20 tokens of a reasoning trace as a more accurate but costlier estimate, `g`.
The core method treats each candidate model as an unopened box. Under a Gaussian signal assumption, the researchers calculate a reservation value from the calibrated relationship between `f` and `g`; a box is opened only when the expected value of information from the expensive evaluation exceeds its cost. A non-obligatory inspection variant can also retain one candidate without running `g` at all, preventing the router from paying any additional evaluation cost for prompts that are easy to assess.
The experiments covered 16,512 mathematics problems, Wikipedia/PubMed RAG experts, and EmbedLLM, which includes more than 100 open-weight models. Measured by routing regret plus evaluation cost, Pandora scored 0.105, 0.118, and 0.386 across the three test sets, respectively. A margin-based heuristic with the same evaluation budget scored 0.105, 0.128, and 0.389. Under the Gaussian approximation, the mathematics and RAG experiments required averages of only 0.58 and 1.40 expensive-estimate queries per problem, fewer than the variant required to inspect at least one candidate.
This work advances model routing from “comparing predicted scores” to “deciding whether obtaining a better score is worthwhile,” making it relevant to model gateways, tiered RAG systems, and services with variable reasoning budgets. However, the mathematics setting had only two candidates, and Pandora and the margin heuristic achieved the same combined score there. The prices were also taken from specific APIs and did not represent full end-to-end latency. The next engineering steps should examine distribution drift, heterogeneous GPU queueing times, and whether non-Gaussian signals or correlated candidates undermine calibration.