Orcetra Reveals Distorted AutoML Benchmark: Win Rate on the Same Subset Falls From 59.4% to 34.3%

The Orcetra team has published a rare self-audit. The 1,661-line AutoML search system originally achieved a 57.1% win rate across 513 OpenML tabular datasets, compared with 21.6% for AutoGluon and 10.9% for FLAML. However, its search loop evaluated dozens of candidate models on the same test split and then reported the highest score. Competing systems selected models using the training data and accessed the test set only once at the end. Orcetra was therefore not simply evaluating generalization performance; it was adaptively selecting the most favorable test-set noise.

The second issue was that the nominal 60-second limit was not a hard deadline. The code checked the time only before launching the next candidate, allowing models that had already started to run to completion. As a result, Orcetra’s median runtime per dataset reached 120 seconds, 2.24 times that of AutoGluon. Four worker processes also forced models to contend for resources on the same 20-core ARM64 host. FLAML and AutoGluon, which respected the time limit, ran fewer trials under this contention, while Orcetra preserved its search volume by extending wall-clock time. If results from a separate rerun covering only regression tasks were incorrectly merged, the apparent win rate could even rise to 61.2%.

The authors changed the protocol to select models on a validation set, enforce deadlines through an external process, and allocate a fixed five cores to each framework. Across 143 rerun datasets, the old protocol gave Orcetra a 59.4% win rate; after the corrections, it fell to 34.3%, with no significant pairwise difference against either FLAML or AutoGluon. Paired analysis showed that selecting models directly on the test set contributed about 4.8 percentage points, while compute allocation accounted for more of the distortion.

The findings carry direct warnings for short-budget model search, proxy evaluation, and inference routing: budgets must be enforced by an external executor, with actual wall-clock time, core allocation, and candidate counts recorded. The corrected benchmark has limitations: it reran only 143 datasets, used just one split per dataset, and omitted AutoGluon’s optional fastai model. The percentages therefore should not be extrapolated to other hardware. Orcetra’s website and README, however, still display the old 57.1% result. Engineering teams should watch whether its public claims and default evaluation pipeline are updated accordingly.