Applied AI research
Hugging Face and Earthmover Connect Aurora and ERA5, Shifting the AI Weather Forecasting Bottleneck to the Data Pipeline
The newly released executable workflow streams ERA5 initial conditions directly, enabling Aurora to generate and backtest forecasts locally or on per-minute-billed compute. It does not introduce a new model; its technical value lies in connecting weights, meteorological arrays, static variables, and validation steps into a reproducible pipeline.
On September 8, Hugging Face and Earthmover released an [AI weather forecasting execution and validation workflow](https://huggingface.co/blog/hugging-science/earthmover-hf), demonstrating how to build inputs for Microsoft Aurora from analysis-ready ERA5 data instead of first downloading complete meteorological archives. The problem it addresses is not model inference itself, but the data engineering that precedes it: initializing a single run requires about 1 GB of data, while backtesting an entire year may consume roughly 360 GB in inputs alone. When the model itself runs in seconds, the GPU often ends up waiting for downloads and data conversion.
The example uses `aurora-0.25-pretrained`. This checkpoint accepts ERA5-formatted grids at 0.25-degree resolution, with each forward pass predicting the global atmospheric state six hours ahead. Four autoregressive rollout steps produce a 24-hour forecast, while 28 steps produce a seven-day forecast. Using a read-only Arraylake session and `xarray.open_zarr`, the code selects surface and pressure-level variables, then assembles 2-meter temperature, 10-meter winds, mean sea-level pressure, temperature, winds, humidity, and geopotential height into an Aurora `Batch`. Static variables such as soil type—which are absent from ERA5 but must match the training configuration—are downloaded separately from the Hugging Face Hub.
The pipeline also includes the validation stage most likely to be omitted from demonstrations: it reconstructs the output as a coordinate-aware `xarray.Dataset`, retrieves the ERA5 state for the forecast valid time, and calculates global RMSE using cosine-of-latitude weighting. The official demo claims that an end-to-end 24-hour forecast can be completed in about 30 seconds. On a local CPU, each six-hour step takes approximately two to three minutes, compared with a few seconds on a GPU. When suitable hardware is unavailable, the job can be submitted through Hugging Face Jobs; the example uses an A100 billed by the minute.
Engineering teams should still treat this as an onboarding and reproducibility template rather than an operational weather service. The [Aurora documentation](https://microsoft.github.io/aurora/models.html) lists checkpoints with different resolutions and fine-tuning configurations. Choosing the wrong model, pressure levels, variable names, or static fields can produce outputs that appear plausible but are not comparable. ERA5 is also a reanalysis dataset, not real-time observations; the data available to the example currently lags the present by several months. It therefore does not yet address real-time data assimilation, data latency, long-term drift, or calibration for extreme events. The next areas to watch are a unified input schema across models, the cost of large-scale backtesting, and whether the same data pipeline can provide fair comparisons with models such as AIFS and FourCastNet.