評測與可重現性
ERA’s Reported 11% Edge in Retrospective Epidemic Forecasting Is Reassessed, With Data-Revision Leakage Explaining Nearly All of the Gain
A new analysis finds that Google’s ERA was backtested using season-finalized data, effectively giving it access to revisions that were unavailable to CDC models when their forecasts were made. The two achieved similar Weighted Interval Scores when the data underwent little or no revision, showing that AI forecasting benchmarks must preserve the version of the data available at each point in time.
Johannes Bracher and Sebastian Funk reanalyzed the COVID-19 hospitalization forecasting case study of Google’s ERA published in Nature. They found that nearly all of its reported 11% improvement in Weighted Interval Score (WIS) over the CDC Forecast Hub ensemble model can be explained by retrospective data leakage. The issue was not that test labels were directly included in the training set, but rather the version of the data used. CDC models submitted forecasts in real time each week during the 2024/25 season and could see only the hospitalization figures available at the time, which were still incomplete and sometimes substantially revised later. ERA’s backtest instead used season-finalized data, effectively giving it access to information that would not become available until the future.
Across 52 U.S. states and jurisdictions and forecast horizons of zero to three weeks, the original study reported that the mean WIS fell from 29.1 for the CDC ensemble to 25.9 for ERA. The reanalysis divided 1,430 state-week combinations into bins according to the size of the revision to the most recent observation. When the relative revision was no more than 1%, ERA’s WIS relative to the CDC model was 1.00. When the absolute revision was no more than one, the relative WIS was 1.01. Neither result indicates a meaningful advantage. By contrast, the relative WIS fell to 0.69 for cases with revisions greater than 25%. Just nine of the most heavily revised cases accounted for 20% of the total WIS gap, with mean WIS values of 88 for ERA and 189 for the CDC model.
The result does not show that ERA’s LLM-plus-tree-search method is ineffective. The authors specifically note that the updated Google SAI Ensemble used genuine real-time submissions during the 2025/26 season and ranked first on the leaderboards for three diseases. The two experiments also used different model-selection and ensemble strategies. What the reanalysis overturns is the attribution that the original retrospective result demonstrated superiority over the contemporaneous CDC ensemble.
The implications extend beyond epidemic forecasting. Demand, financial, and macroeconomic data are also subject to delayed reporting, reclassification, and revision. When building benchmarks for AI forecasting or automated data science, engineers should preserve timestamped snapshots of datasets, features, and external API responses; restrict models to the versions available before the forecast cutoff; and prioritize prospective, blinded evaluations. Simply rerunning historical forecasts using today’s data can produce a leaderboard that is reproducible but unfair.