AI 評測與開發工具
optstop Dynamically Stops Evaluations Using Bayesian Credible Intervals, Saving Up to 97.3% of Trials Across Nine Test Sets
The UK AI Security Institute has open-sourced optstop, which uses posterior uncertainty at the item and task levels to decide when to stop repeated evaluations. Across nine shadow tests, it saved an average of 81.1% of trials, though performance varied with score type, item ordering, and the exchangeability assumption.
Running every item the same number of times wastes API calls and GPU resources on items that have already converged. optstop reframes this as a sequential measurement problem. After each new result, it updates a hierarchical Bayesian model. When the width of the posterior credible interval for an individual item or a model–task group falls below a threshold, it stops the corresponding trials. The defaults are a 97% credible interval and a width of 0.05, effectively requiring the estimate to fall within roughly ±2.5 percentage points. If the interval repeatedly fails to meet the target, optstop also uses the recent slope of interval width to determine whether information gain has plateaued.
The tool handles binary, ordinal, and bounded continuous scores separately. Binary outcomes use a hierarchical logit-normal model; ordinal scales combine ordered logistic regression, modal intervals, and entropy convergence; and continuous scores use item-level summaries for hierarchical inference. It can prune existing data offline and implements the `inspect_ai` `EarlyStopping` protocol. To avoid missing rare successes when scores are near zero, the framework samples more conservatively instead of stopping immediately after a run of failures.
The paper evaluates the method through nine shadow experiments with 200 items and ten originally planned runs per item. All requests were still executed in full, after which the early-stopping points were replayed, eliminating stochastic differences between separate model runs. The share of omitted trials ranged from 57.2% to 97.3%, averaging 81.1%; the mean absolute difference between truncated and full estimates was 0.006. However, this was not a direct cost experiment under production traffic, and the models—GPT-3.5 Turbo, GPT-4o, and Claude Sonnet 4.5—were called in March 2026. Engineering teams must also randomize item order and verify within-group exchangeability. If items are ordered by difficulty, early stopping may introduce systematic bias. The next step is to determine whether optstop can maintain coverage for rare hazardous capabilities, long-horizon agentic tasks, and drift in model-based judges.